1 — Loading data¶
Theme: getting a file off disk and understanding what you get back.
This notebook covers loading a single file, a whole folder, and letting the package identify the instrument for you — then what a loaded object actually contains. Plotting is deliberately left out; see 5 — Plotting.
[1]:
import aerosoltools as at
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
Loading a single file¶
Every instrument has a loader named load_<instrument>_file. They all take a path and return an object of the appropriate class.
[2]:
elpi = at.load_elpi_file("../../tests/data/Sample_ELPI.txt")
elpi
/home/runner/work/aerosoltools/aerosoltools/src/aerosoltools/loaders/elpi.py:525: RuntimeWarning: ELPI density is not 1.0 g/cm3; bin edges were estimated from geometric means of CalculatedDi values.
return load_elpi_file_txt(
[2]:
<aerosoltools.elpi.ELPI at 0x7f23d4895890>
A warning is shown because the particle density recorded in this file is not 1.0 g/cm³. The value is read and stored, but you are told about it because it affects any mass-based quantity — see 6 — Dtypes, density and corrections.
Letting the package identify the instrument¶
If you do not know (or do not want to hard-code) which instrument produced a file, load_file inspects the contents and dispatches to the right loader.
[3]:
data = at.load_file("../../tests/data/Sample_OPS.csv")
type(data).__name__
[3]:
'Aerosol2D'
detect_instrument performs the same identification without loading, which is useful when triaging a folder of mixed exports.
[4]:
for name in ["Sample_OPS.csv", "Sample_SMPS.txt", "Sample_DustTrak.csv",
"Sample_Partector.txt", "Sample_ACSM.csv"]:
print(f"{name:25s} -> {at.detect_instrument('../../tests/data/' + name)}")
Sample_OPS.csv -> OPS
Sample_SMPS.txt -> SMPS
Sample_DustTrak.csv -> DustTrak
Sample_Partector.txt -> Partector
Sample_ACSM.csv -> ACSM
Identification works from the file contents first and falls back to the filename, so renaming a file does not break it. The instruments it knows about are listed in INSTRUMENT_LOADERS, which maps each name to its loader.
[5]:
list(at.INSTRUMENT_LOADERS)
[5]:
['CPC',
'DiSCmini',
'DiSCmini (raw)',
'ELPI',
'FMPS',
'Fourtec',
'Grimm',
'NanoScan (NS)',
'OPC-N3',
'OPS',
'Partector',
'SMPS',
'Aethalometer',
'DustTrak',
'Ranger',
'APS',
'Tiger',
'ACSM']
Loading a whole folder¶
load_data_from_folder applies one loader to every matching file in a folder and concatenates the results. Use search_word to restrict which files are picked up.
[6]:
ops = at.load_data_from_folder(
"../../tests/data/OPS_data",
at.load_ops_file,
search_word="OPS",
)
Loading files: 100%|██████████| 4/4 [00:00<00:00, 119.62file/s]
File load summary:
+---------------+--------+--------+
| File | Status | Reason |
+---------------+--------+--------+
| OPS_data1.csv | Loaded | - |
| OPS_data2.csv | Loaded | - |
| OPS_data3.csv | Loaded | - |
| OPS_data4.csv | Loaded | - |
+---------------+--------+--------+
The progress bar and the summary table tell you which files were loaded, which were skipped, and why. Files are sorted by time, so they do not need to be passed in order. Only files from the same instrument are combined — the serial number in the metadata is checked.
To join separate runs of one instrument, or to stitch two instruments covering different size ranges, see 7 — Combining datasets.
What a loaded object contains¶
The measurements live in .data, indexed by time.
[7]:
elpi.data.head()
[7]:
| Total_conc | 22.6 | 88.8 | 128.7 | 190.4 | 287.0 | 458.3 | 740.9 | 1280.9 | 2087.3 | 3388.4 | 5493.9 | 8152.8 | 11955.5 | 19593.6 | All data | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Datetime | ||||||||||||||||
| 2023-09-07 09:06:38 | 192.59404 | 0.000 | 26.76 | 21.14 | 31.25 | 36.22 | 31.48 | 32.71 | 10.95 | 1.332 | 0.3153 | 0.1676 | 0.1027 | 0.11440 | 0.05204 | True |
| 2023-09-07 09:06:39 | 198.47531 | 10.790 | 23.07 | 21.32 | 30.06 | 35.63 | 32.91 | 31.78 | 10.99 | 1.212 | 0.3076 | 0.1822 | 0.1060 | 0.07515 | 0.04236 | True |
| 2023-09-07 09:06:40 | 192.15342 | 5.988 | 23.08 | 19.76 | 30.23 | 35.89 | 31.86 | 32.30 | 11.14 | 1.169 | 0.3561 | 0.1572 | 0.1192 | 0.06589 | 0.03803 | True |
| 2023-09-07 09:06:41 | 231.09214 | 48.460 | 18.86 | 20.95 | 29.67 | 36.08 | 31.03 | 32.91 | 11.18 | 1.197 | 0.3101 | 0.1770 | 0.1820 | 0.05388 | 0.03216 | True |
| 2023-09-07 09:06:42 | 268.55528 | 89.410 | 17.39 | 19.68 | 29.91 | 35.99 | 29.17 | 33.94 | 11.12 | 1.211 | 0.3037 | 0.1558 | 0.1779 | 0.05711 | 0.03977 | True |
Anything the loader found that is not a measurement — flows, temperatures, status flags — goes to .extra_data, keeping .data clean.
[8]:
list(elpi.extra_data.columns)
[8]:
['Status',
'Channel1',
'Channel2',
'Channel3',
'Channel4',
'Channel5',
'Channel6',
'Channel7',
'Channel8',
'Channel9',
'Channel10',
'Channel11',
'Channel12',
'Channel13',
'Channel14',
'MISC',
'Charger current',
'Charger voltage',
'Trap voltage',
'Low pressure',
'Pressure',
'Aux channel1',
'Aux channel2',
'Aux channel3',
'Aux channel4',
'Aux channel5',
'Aux channel6',
'Temperature1',
'Temperature2',
'Temperature3',
'Temperature4',
'Reserved',
'Concentration value',
'COM',
'User comment',
'PM10',
'PM2.5',
'PM1',
'PN10',
'PN2.5',
'PN1',
'CHARMEAS',
'Stage12',
'MEDIAN',
'Number Median',
'Number GSD',
'Channel from',
'Channel to',
'Unnamed_78']
.metadata holds everything the loader learned about the instrument.
[9]:
elpi.metadata
[9]:
{'Date': '07/09/2023',
'Time': '08:22:18',
'Location': '',
'Description': 'Default',
'Operator/run': '',
'Cleaned': '06/09/2023 09:07',
'Sampled': 'other',
'UnitNo': 'A',
'FlowRate': 10.0,
'D50values(um)': [0.006,
0.013,
0.0181,
0.0326,
0.0497,
0.097,
0.169,
0.315,
0.59,
0.91,
1.63,
2.47,
3.65,
5.37,
9.89],
'Pressure(kPa)': [4.0,
4.56,
9.77,
22.29,
39.24,
68.56,
88.99,
97.24,
99.64,
100.51,
101.01,
101.19,
101.24,
101.3,
101.32],
'ResTime': [0.001,
0.0024,
0.0056,
0.0127,
0.0222,
0.0391,
0.0505,
0.0552,
0.0585,
0.0622,
0.0571,
0.0578,
0.0736,
0.1029,
0.0],
'DiffLossFit': [9.8e-08,
-2.73,
0.05085,
9.8e-08,
-2.73,
0.05085,
9.8e-08,
-2.73,
0.05085,
3.63e-08,
-3.06,
0.03083,
1.58e-06,
-2.26,
0.02342,
4.83e-06,
-2.06,
0.02183,
1.02e-05,
-1.9,
0.02097,
6.22e-05,
-1.49,
0.01158,
4.03e-05,
-1.56,
0.00804,
7.31e-05,
-1.4,
0.00671,
0.000101,
-1.31,
0.00475,
8.22e-05,
-1.35,
0.00288,
0.000109,
-1.31,
0.00129,
9.07e-05,
-1.36,
0.00186],
'FilterLow(um)': 0.006,
'FilterCal': 1.0,
'ChargerSetup': '10 lpm',
'ChargerI(uA)': 1.0,
'ChargerU(kV)': 3.0,
'TrapU(V)': 20.0,
'Efficiency(Dp/mult/exp)': [1.035,
1.83,
1.225,
4.282,
1.8114,
1.515,
3.3868,
1.085],
'VersionNo': 'ELPI+VI 2.1 rev. 932',
'DelimiterChar': '',
'RawData': 'Current (fA)',
'StokesDp': 'FALSE',
'Correction': 'TRUE',
'Dilution': 1.0,
'DataOrder': '',
'density': 0.14,
'bin_edges': array([1.88000e+01, 4.48000e+01, 1.06900e+02, 1.56500e+02, 2.33800e+02,
3.62700e+02, 5.82700e+02, 9.74200e+02, 1.63510e+03, 2.65940e+03,
4.31460e+03, 6.69260e+03, 9.87270e+03, 1.53053e+04, 2.37271e+04]),
'bin_mids': array([ 22.6, 88.8, 128.7, 190.4, 287. , 458.3, 740.9,
1280.9, 2087.3, 3388.4, 5493.9, 8152.8, 11955.5, 19593.6]),
'instrument': 'ELPI',
'serial_number': 'HR-E+26255',
'dtype': 'dN',
'unit': 'cm⁻³',
'source_quantity': 'converted_export',
'column_units': {}}
The most-used entries also have their own properties.
[10]:
print("instrument :", elpi.instrument)
print("serial number:", elpi.serial_number)
print("unit :", elpi.unit)
print("dtype :", elpi.dtype)
print("measurement :", elpi.measurement)
instrument : ELPI
serial number: HR-E+26255
unit : cm⁻³
dtype : dN
measurement : None
unit and dtype describe the primary measurement. Instruments that record several different quantities report them per column instead, which is what column_units gives you.
[11]:
dust = at.load_dusttrak_file("../../tests/data/Sample_DustTrak.csv")
dust.column_units
[11]:
{}
Originals and copies¶
Most operations modify an object in place. .original_data always keeps the data as it was loaded, so you can see what changed.
[12]:
before = len(elpi.data)
elpi.timecrop(start="2023-09-07 09:07:00", end="2023-09-07 09:09:00")
print(f"data : {before} -> {len(elpi.data)} rows")
print(f"original_data : {len(elpi.original_data)} rows (unchanged)")
data : 256 -> 121 rows
original_data : 256 rows (unchanged)
When you want to branch an analysis without disturbing the original object, copy_self returns an independent deep copy.
[13]:
branch = elpi.copy_self()
branch.timecrop(start="2023-09-07 09:07:30", end="2023-09-07 09:08:00")
print(f"branch : {len(branch.data)} rows")
print(f"elpi : {len(elpi.data)} rows (untouched)")
branch : 31 rows
elpi : 121 rows (untouched)
Next: 2 — Time adjustments puts two instruments onto a common time base.