7 — Combining datasets

Theme: joining data along time, and joining it across size.

Two different operations are easy to confuse:

  • ``combine_measurements`` joins runs of the same instrument along the time axis — three days of monitoring into one record.

  • ``combine_size_ranges`` joins two different instruments covering neighbouring size ranges into one wider distribution, over the same period.

Both differ from load_data_from_folder, which is about reading many files of one instrument at load time — see 1 — Loading data.

[1]:
import aerosoltools as at
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

Joining runs along time

A Ranger with interchangeable heads returns one object per measured component, so pick the component you want from each file first.

[2]:
def chlorine(path):
    loaded = at.load_ranger_file(path)
    objects = loaded if isinstance(loaded, list) else [loaded]
    return next(o for o in objects if o.dtype == "Cl₂")


runs = [
    chlorine("../../tests/data/Ranger_2605-1000088-A_20260612-0603.csv"),
    chlorine("../../tests/data/Ranger_2605-1000088-A_20260617-0712.csv"),
    chlorine("../../tests/data/Ranger_2605-1000088-A_20260701-0832.csv"),
]

for run in runs:
    print(f"{len(run.data):5d} samples   {run.time[0]}  ->  {run.time[-1]}")
 1578 samples   2026-05-19 09:44:00  ->  2026-06-11 12:10:00
 1020 samples   2026-06-16 11:24:00  ->  2026-06-17 04:23:00
 2033 samples   2026-06-16 11:24:00  ->  2026-07-01 04:17:00
[3]:
campaign = at.combine_measurements(runs)

print(f"combined: {len(campaign.data)} samples")
print(f"          {campaign.time[0]}  ->  {campaign.time[-1]}")
print(f"          {campaign.dtype} in {campaign.unit}")
combined: 3611 samples
          2026-05-19 09:44:00  ->  2026-07-01 04:17:00
          Cl₂ in ppm

The result is one object of the same class, sorted by time and de-duplicated. Gaps between runs are preserved — nothing is interpolated across them, so a weekend with no measurement stays a hole rather than becoming a straight line.

Note the combined length is smaller than the sum of the inputs: overlapping timestamps are collapsed rather than duplicated.

By default all inputs must share a serial number, which is what stops two different instruments being silently concatenated. Pass require_same_serial=False when the metadata is missing and you know the inputs belong together.

[4]:
fig, ax = campaign.plot_total_conc()
ax.set_title("Three runs combined into one record")
[4]:
Text(0.5, 1.0, 'Three runs combined into one record')
../_images/examples_07-combining-datasets_6_1.png

Joining across size

No single instrument covers the whole size range. A NanoScan measures roughly 10–400 nm; an OPS roughly 300 nm–10 µm. combine_size_ranges stitches them into one distribution.

[5]:
ns = at.load_ns_file("../../tests/data/Combine_example_NS.csv")
ops = at.load_data_from_folder(
    "../../tests/data", at.load_ops_file, search_word="Combine_example"
)

print(f"NanoScan: {ns.bin_mids.min():.0f} - {ns.bin_mids.max():.0f} nm")
print(f"OPS     : {ops.bin_mids.min():.0f} - {ops.bin_mids.max():.0f} nm")
Loading files: 100%|██████████| 5/5 [00:00<00:00, 130.91file/s]

File load summary:

+----------------------------------+---------+------------------------------------------------------------------------------------+
|               File               | Status  |                                       Reason                                       |
+----------------------------------+---------+------------------------------------------------------------------------------------+
| Combine_example_OPS - file 1.csv | Loaded  |                                         -                                          |
| Combine_example_OPS - file 2.csv | Loaded  |                                         -                                          |
| Combine_example_OPS - file 3.csv | Loaded  |                                         -                                          |
| Combine_example_OPS - file 4.csv | Loaded  |                                         -                                          |
|      Combine_example_NS.csv      | Skipped | IndexError: too many indices for array: array is 0-dimensional, but 1 were indexed |
+----------------------------------+---------+------------------------------------------------------------------------------------+
NanoScan: 12 - 365 nm
OPS     : 337 - 9016 nm

The two instruments log on different clocks, so they must be aligned in time first. match="nearest" pairs each NanoScan sample with the closest OPS sample within tolerance; match="rebin" puts both onto a common grid instead.

[6]:
combined = at.combine_size_ranges(ns, ops, match="nearest", tolerance="30s")

print(f"combined: {combined.bin_mids.min():.0f} - {combined.bin_mids.max():.0f} nm")
print(f"          {len(combined.bin_mids)} bins, {len(combined.data)} samples")
combined: 12 - 9016 nm
          28 bins, 9582 samples

Where the ranges overlap, one instrument has to win. The crossover diameter decides: bins below it come from the lower instrument, bins above from the upper. Left as None it is chosen automatically; set it explicitly when you have a reason to trust one instrument over the other in the overlap.

The crossover has to lie inside the shared overlap — asking for one outside it raises rather than silently discarding a range.

[7]:
print(f"shared overlap: {ops.bin_mids.min():.0f} - {ns.bin_mids.max():.0f} nm")

low = at.combine_size_ranges(ns, ops, crossover=320, match="nearest", tolerance="30s")
high = at.combine_size_ranges(ns, ops, crossover=400, match="nearest", tolerance="30s")

print(f"crossover 320 nm -> {len(low.bin_mids)} bins")
print(f"crossover 400 nm -> {len(high.bin_mids)} bins")
shared overlap: 337 - 365 nm
crossover 320 nm -> 28 bins
crossover 400 nm -> 27 bins
[8]:
try:
    at.combine_size_ranges(ns, ops, crossover=700, match="nearest", tolerance="30s")
except ValueError as err:
    print("ValueError:", err)
ValueError: crossover 700.0 nm is outside the instruments' shared overlap (300.0-420.0 nm).

The result is an ordinary size-resolved dataset, so everything from the earlier notebooks applies to it.

[9]:
fig, axs = combined.plot_timeseries(y_3d=(1, 0))
../_images/examples_07-combining-datasets_15_0.png
[10]:
fig, ax = combined.plot_psd()
ax.set_title("Stitched distribution, 10 nm to 10 um")
[10]:
Text(0.5, 1.0, 'Stitched distribution, 10 nm to 10 um')
../_images/examples_07-combining-datasets_16_1.png

Next: 8 — PSD fitting.