{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 7 — Combining datasets\n", "\n", "**Theme:** joining data along time, and joining it across size.\n", "\n", "Two different operations are easy to confuse:\n", "\n", "- **`combine_measurements`** joins runs of the *same* instrument along the time\n", " axis — three days of monitoring into one record.\n", "- **`combine_size_ranges`** joins *two different* instruments covering\n", " neighbouring size ranges into one wider distribution, over the same period.\n", "\n", "Both differ from `load_data_from_folder`, which is about reading many files of\n", "one instrument at load time — see [1 — Loading data](01-loading-data.ipynb)." ], "id": "cell-00" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import aerosoltools as at" ], "id": "cell-01" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Joining runs along time\n", "\n", "A Ranger with interchangeable heads returns one object per measured component,\n", "so pick the component you want from each file first." ], "id": "cell-02" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def chlorine(path):\n", " loaded = at.load_ranger_file(path)\n", " objects = loaded if isinstance(loaded, list) else [loaded]\n", " return next(o for o in objects if o.dtype == \"Cl₂\")\n", "\n", "\n", "runs = [\n", " chlorine(\"../../tests/data/Ranger_2605-1000088-A_20260612-0603.csv\"),\n", " chlorine(\"../../tests/data/Ranger_2605-1000088-A_20260617-0712.csv\"),\n", " chlorine(\"../../tests/data/Ranger_2605-1000088-A_20260701-0832.csv\"),\n", "]\n", "\n", "for run in runs:\n", " print(f\"{len(run.data):5d} samples {run.time[0]} -> {run.time[-1]}\")" ], "id": "cell-03" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "campaign = at.combine_measurements(runs)\n", "\n", "print(f\"combined: {len(campaign.data)} samples\")\n", "print(f\" {campaign.time[0]} -> {campaign.time[-1]}\")\n", "print(f\" {campaign.dtype} in {campaign.unit}\")" ], "id": "cell-04" }, { "cell_type": "markdown", "metadata": {}, "source": [ "The result is one object of the same class, sorted by time and de-duplicated.\n", "Gaps between runs are preserved — nothing is interpolated across them, so a\n", "weekend with no measurement stays a hole rather than becoming a straight line.\n", "\n", "Note the combined length is smaller than the sum of the inputs: overlapping\n", "timestamps are collapsed rather than duplicated.\n", "\n", "By default all inputs must share a serial number, which is what stops two\n", "different instruments being silently concatenated. Pass\n", "`require_same_serial=False` when the metadata is missing and you know the\n", "inputs belong together." ], "id": "cell-05" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig, ax = campaign.plot_total_conc()\n", "ax.set_title(\"Three runs combined into one record\")" ], "id": "cell-06" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Joining across size\n", "\n", "No single instrument covers the whole size range. A NanoScan measures roughly\n", "10–400 nm; an OPS roughly 300 nm–10 µm. `combine_size_ranges` stitches them\n", "into one distribution." ], "id": "cell-07" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ns = at.load_ns_file(\"../../tests/data/Combine_example_NS.csv\")\n", "ops = at.load_data_from_folder(\n", " \"../../tests/data\", at.load_ops_file, search_word=\"Combine_example\"\n", ")\n", "\n", "print(f\"NanoScan: {ns.bin_mids.min():.0f} - {ns.bin_mids.max():.0f} nm\")\n", "print(f\"OPS : {ops.bin_mids.min():.0f} - {ops.bin_mids.max():.0f} nm\")" ], "id": "cell-08" }, { "cell_type": "markdown", "metadata": {}, "source": [ "The two instruments log on different clocks, so they must be aligned in time\n", "first. `match=\"nearest\"` pairs each NanoScan sample with the closest OPS sample\n", "within `tolerance`; `match=\"rebin\"` puts both onto a common grid instead." ], "id": "cell-09" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "combined = at.combine_size_ranges(ns, ops, match=\"nearest\", tolerance=\"30s\")\n", "\n", "print(f\"combined: {combined.bin_mids.min():.0f} - {combined.bin_mids.max():.0f} nm\")\n", "print(f\" {len(combined.bin_mids)} bins, {len(combined.data)} samples\")" ], "id": "cell-10" }, { "cell_type": "markdown", "metadata": {}, "source": [ "Where the ranges overlap, one instrument has to win. The crossover diameter\n", "decides: bins below it come from the lower instrument, bins above from the\n", "upper. Left as `None` it is chosen automatically; set it explicitly when you\n", "have a reason to trust one instrument over the other in the overlap.\n", "\n", "The crossover has to lie *inside* the shared overlap — asking for one outside\n", "it raises rather than silently discarding a range." ], "id": "cell-11" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(f\"shared overlap: {ops.bin_mids.min():.0f} - {ns.bin_mids.max():.0f} nm\")\n", "\n", "low = at.combine_size_ranges(ns, ops, crossover=320, match=\"nearest\", tolerance=\"30s\")\n", "high = at.combine_size_ranges(ns, ops, crossover=400, match=\"nearest\", tolerance=\"30s\")\n", "\n", "print(f\"crossover 320 nm -> {len(low.bin_mids)} bins\")\n", "print(f\"crossover 400 nm -> {len(high.bin_mids)} bins\")" ], "id": "cell-12" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "try:\n", " at.combine_size_ranges(ns, ops, crossover=700, match=\"nearest\", tolerance=\"30s\")\n", "except ValueError as err:\n", " print(\"ValueError:\", err)" ], "id": "cell-13" }, { "cell_type": "markdown", "metadata": {}, "source": [ "The result is an ordinary size-resolved dataset, so everything from the earlier\n", "notebooks applies to it." ], "id": "cell-14" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig, axs = combined.plot_timeseries(y_3d=(1, 0))" ], "id": "cell-15" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig, ax = combined.plot_psd()\n", "ax.set_title(\"Stitched distribution, 10 nm to 10 um\")" ], "id": "cell-16" }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "**Next:** [8 — PSD fitting](08-psd-fitting.ipynb)." ], "id": "cell-17" } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.11" } }, "nbformat": 4, "nbformat_minor": 5 }