{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 1 — Loading data\n", "\n", "**Theme:** getting a file off disk and understanding what you get back.\n", "\n", "This notebook covers loading a single file, a whole folder, and letting the\n", "package identify the instrument for you — then what a loaded object actually\n", "contains. Plotting is deliberately left out; see\n", "[5 — Plotting](05-plotting.ipynb)." ], "id": "cell-00" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import aerosoltools as at" ], "id": "cell-01" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Loading a single file\n", "\n", "Every instrument has a loader named `load__file`. They all take a\n", "path and return an object of the appropriate class." ], "id": "cell-02" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "elpi = at.load_elpi_file(\"../../tests/data/Sample_ELPI.txt\")\n", "elpi" ], "id": "cell-03" }, { "cell_type": "markdown", "metadata": {}, "source": [ "A warning is shown because the particle density recorded in this file is not\n", "1.0 g/cm³. The value is read and stored, but you are told about it because it\n", "affects any mass-based quantity — see\n", "[6 — Dtypes, density and corrections](06-dtypes-density-corrections.ipynb).\n", "\n", "## Letting the package identify the instrument\n", "\n", "If you do not know (or do not want to hard-code) which instrument produced a\n", "file, `load_file` inspects the contents and dispatches to the right loader." ], "id": "cell-04" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data = at.load_file(\"../../tests/data/Sample_OPS.csv\")\n", "type(data).__name__" ], "id": "cell-05" }, { "cell_type": "markdown", "metadata": {}, "source": [ "`detect_instrument` performs the same identification without loading, which is\n", "useful when triaging a folder of mixed exports." ], "id": "cell-06" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "for name in [\"Sample_OPS.csv\", \"Sample_SMPS.txt\", \"Sample_DustTrak.csv\",\n", " \"Sample_Partector.txt\", \"Sample_ACSM.csv\"]:\n", " print(f\"{name:25s} -> {at.detect_instrument('../../tests/data/' + name)}\")" ], "id": "cell-07" }, { "cell_type": "markdown", "metadata": {}, "source": [ "Identification works from the file *contents* first and falls back to the\n", "filename, so renaming a file does not break it. The instruments it knows about\n", "are listed in `INSTRUMENT_LOADERS`, which maps each name to its loader." ], "id": "cell-08" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "list(at.INSTRUMENT_LOADERS)" ], "id": "cell-09" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Loading a whole folder\n", "\n", "`load_data_from_folder` applies one loader to every matching file in a folder\n", "and concatenates the results. Use `search_word` to restrict which files are\n", "picked up." ], "id": "cell-10" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ops = at.load_data_from_folder(\n", " \"../../tests/data/OPS_data\",\n", " at.load_ops_file,\n", " search_word=\"OPS\",\n", ")" ], "id": "cell-11" }, { "cell_type": "markdown", "metadata": {}, "source": [ "The progress bar and the summary table tell you which files were loaded, which\n", "were skipped, and why. Files are sorted by time, so they do not need to be\n", "passed in order. Only files from the same instrument are combined — the serial\n", "number in the metadata is checked.\n", "\n", "To join separate *runs* of one instrument, or to stitch two instruments\n", "covering different size ranges, see\n", "[7 — Combining datasets](07-combining-datasets.ipynb).\n", "\n", "## What a loaded object contains\n", "\n", "The measurements live in `.data`, indexed by time." ], "id": "cell-12" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "elpi.data.head()" ], "id": "cell-13" }, { "cell_type": "markdown", "metadata": {}, "source": [ "Anything the loader found that is not a measurement — flows, temperatures,\n", "status flags — goes to `.extra_data`, keeping `.data` clean." ], "id": "cell-14" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "list(elpi.extra_data.columns)" ], "id": "cell-15" }, { "cell_type": "markdown", "metadata": {}, "source": [ "`.metadata` holds everything the loader learned about the instrument." ], "id": "cell-16" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "elpi.metadata" ], "id": "cell-17" }, { "cell_type": "markdown", "metadata": {}, "source": [ "The most-used entries also have their own properties." ], "id": "cell-18" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(\"instrument :\", elpi.instrument)\n", "print(\"serial number:\", elpi.serial_number)\n", "print(\"unit :\", elpi.unit)\n", "print(\"dtype :\", elpi.dtype)\n", "print(\"measurement :\", elpi.measurement)" ], "id": "cell-19" }, { "cell_type": "markdown", "metadata": {}, "source": [ "`unit` and `dtype` describe the primary measurement. Instruments that record\n", "several different quantities report them per column instead, which is what\n", "`column_units` gives you." ], "id": "cell-20" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "dust = at.load_dusttrak_file(\"../../tests/data/Sample_DustTrak.csv\")\n", "dust.column_units" ], "id": "cell-21" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Originals and copies\n", "\n", "Most operations modify an object in place. `.original_data` always keeps the\n", "data as it was loaded, so you can see what changed." ], "id": "cell-22" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "before = len(elpi.data)\n", "elpi.timecrop(start=\"2023-09-07 09:07:00\", end=\"2023-09-07 09:09:00\")\n", "\n", "print(f\"data : {before} -> {len(elpi.data)} rows\")\n", "print(f\"original_data : {len(elpi.original_data)} rows (unchanged)\")" ], "id": "cell-23" }, { "cell_type": "markdown", "metadata": {}, "source": [ "When you want to branch an analysis without disturbing the original object,\n", "`copy_self` returns an independent deep copy." ], "id": "cell-24" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "branch = elpi.copy_self()\n", "branch.timecrop(start=\"2023-09-07 09:07:30\", end=\"2023-09-07 09:08:00\")\n", "\n", "print(f\"branch : {len(branch.data)} rows\")\n", "print(f\"elpi : {len(elpi.data)} rows (untouched)\")" ], "id": "cell-25" }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "**Next:** [2 — Time adjustments](02-time-adjustments.ipynb) puts two\n", "instruments onto a common time base." ], "id": "cell-26" } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.11" } }, "nbformat": 4, "nbformat_minor": 5 }