4 — Statistics and exposure

Theme: turning a marked record into numbers — task statistics, and the metrics used for occupational exposure assessment.

Everything here builds on the activities from 3 — Activities.

[1]:
import aerosoltools as at

elpi = at.load_elpi_file("../../tests/data/Sample_ELPI.txt")
elpi.mark_activities({
    "Background": [("2023-09-07 09:06:50", "2023-09-07 09:07:50")],
    "Emission":   [("2023-09-07 09:07:55", "2023-09-07 09:08:30")],
    "Decay":      [("2023-09-07 09:09:00", "2023-09-07 09:10:50")],
})
elpi.activities
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
/home/runner/work/aerosoltools/aerosoltools/src/aerosoltools/loaders/elpi.py:525: RuntimeWarning: ELPI density is not 1.0 g/cm3; bin edges were estimated from geometric means of CalculatedDi values.
  return load_elpi_file_txt(
[1]:
['All data', 'Background', 'Emission', 'Decay']

What can this dataset be summarised on?

Which quantities are available depends on the instrument. A size-resolved instrument can produce mass and number fractions at any cut diameter; a single-channel one cannot. available_metrics reports what this particular dataset supports.

[2]:
for metric in elpi.available_metrics():
    print(f"{metric.key:12s} {metric.label:28s} [{metric.unit}]")
PNC          Number concentration         [cm⁻³]
MASS         Mass concentration           [µg/m³]
PM1          PM1                          [µg/m³]
PM2.5        PM2.5                        [µg/m³]
PM4          PM4                          [µg/m³]
PM10         PM10                         [µg/m³]

One channel at a time

summarize gives descriptive statistics for a single channel, split by activity.

[3]:
elpi.summarize()

Summary of Total_conc:

+---+------------+---------+---------+---------+--------+--------------+
|   |  Segment   |   Min   |   Max   |  Mean   |  Std   | N datapoints |
+---+------------+---------+---------+---------+--------+--------------+
| 0 |  All data  | 118.566 |  442.5  | 197.436 | 48.901 |     256      |
| 1 | Background | 137.253 |  442.5  | 222.08  | 61.048 |      62      |
| 2 |  Emission  | 138.416 | 314.828 | 202.937 | 42.281 |      37      |
| 3 |   Decay    | 121.176 | 309.717 | 179.173 | 39.193 |     112      |
+---+------------+---------+---------+---------+--------+--------------+
[3]:
Segment Min Max Mean Std N datapoints
0 All data 118.566 442.500 197.436 48.901 256
1 Background 137.253 442.500 222.080 61.048 62
2 Emission 138.416 314.828 202.937 42.281 37
3 Decay 121.176 309.717 179.173 39.193 112

Every activity at once

summarize_activities is the usual starting point: descriptive statistics per activity, including duration and — for size-resolved data — size metrics such as mode and geometric mean diameter.

[4]:
elpi.summarize_activities()

Summary of aerosol properties (transposed):

+------------------------+----------+------------+----------+--------+
|                        | All data | Background | Emission | Decay  |
+------------------------+----------+------------+----------+--------+
|    Duration (HH:MM)    |  00:04   |   00:01    |  00:00   | 00:01  |
|       PNC [cm⁻³]       |  197.44  |   222.08   |  202.94  | 179.17 |
|     PNC [cm⁻³] std     |   48.9   |   61.05    |  42.28   | 39.19  |
|      PM1 [µg/m³]       |   1.29   |    1.4     |   1.31   |  1.2   |
|    PM1 [µg/m³] std     |   0.11   |    0.08    |   0.02   |  0.04  |
|     PM2.5 [µg/m³]      |   3.41   |    3.54    |   3.53   |  3.26  |
|   PM2.5 [µg/m³] std    |   0.18   |    0.15    |   0.1    |  0.1   |
|      PM4 [µg/m³]       |   5.77   |    5.63    |   6.22   |  5.7   |
|    PM4 [µg/m³] std     |   0.59   |    0.94    |   0.28   |  0.29  |
|      PM10 [µg/m³]      |  30.33   |   29.61    |  35.27   | 29.77  |
|    PM10 [µg/m³] std    |   9.32   |   15.83    |   2.71   |  3.96  |
|   Total Mass [µg/m³]   |  124.61  |   131.13   |  153.37  | 114.4  |
| Total Mass [µg/m³] std |  51.31   |   83.91    |  25.27   | 23.13  |
|      Mode Dp [nm]      |  125.2   |   116.4    |  129.8   | 134.3  |
|    Mode Dp [nm] std    |  131.3   |   127.5    |  131.6   | 136.2  |
|     Median Dp [nm]     |  224.2   |   211.8    |  227.8   | 228.6  |
|   Median Dp [nm] std   |   68.7   |    77.5    |   66.9   |  67.6  |
|        GMD [nm]        |  203.3   |   195.0    |  201.9   | 213.1  |
|      GMD [nm] std      |   79.7   |    80.7    |   77.9   |  84.2  |
+------------------------+----------+------------+----------+--------+
[4]:
Segment Duration (HH:MM) PNC [cm⁻³] PNC [cm⁻³] std PM1 [µg/m³] PM1 [µg/m³] std PM2.5 [µg/m³] PM2.5 [µg/m³] std PM4 [µg/m³] PM4 [µg/m³] std PM10 [µg/m³] PM10 [µg/m³] std Total Mass [µg/m³] Total Mass [µg/m³] std Mode Dp [nm] Mode Dp [nm] std Median Dp [nm] Median Dp [nm] std GMD [nm] GMD [nm] std
0 All data 00:04 197.44 48.90 1.29 0.11 3.41 0.18 5.77 0.59 30.33 9.32 124.61 51.31 125.2 131.3 224.2 68.7 203.3 79.7
1 Background 00:01 222.08 61.05 1.40 0.08 3.54 0.15 5.63 0.94 29.61 15.83 131.13 83.91 116.4 127.5 211.8 77.5 195.0 80.7
2 Emission 00:00 202.94 42.28 1.31 0.02 3.53 0.10 6.22 0.28 35.27 2.71 153.37 25.27 129.8 131.6 227.8 66.9 201.9 77.9
3 Decay 00:01 179.17 39.19 1.20 0.04 3.26 0.10 5.70 0.29 29.77 3.96 114.40 23.13 134.3 136.2 228.6 67.6 213.1 84.2

You can restrict it to particular metrics or statistics rather than taking the default set.

[5]:
elpi.summarize_activities(metrics=["PNC", "PM2.5"], stats=["mean", "median", "max"])

Summary of aerosol properties (transposed):

+----------------------+----------+------------+----------+--------+
|                      | All data | Background | Emission | Decay  |
+----------------------+----------+------------+----------+--------+
|   Duration (HH:MM)   |  00:04   |   00:01    |  00:00   | 00:01  |
|      PNC [cm⁻³]      |  197.44  |   222.08   |  202.94  | 179.17 |
|  PNC [cm⁻³] median   |  190.1   |   215.57   |  191.12  | 176.09 |
|    PNC [cm⁻³] max    |  442.5   |   442.5    |  314.83  | 309.72 |
|    PM2.5 [µg/m³]     |   3.41   |    3.54    |   3.53   |  3.26  |
| PM2.5 [µg/m³] median |   3.4    |    3.57    |   3.56   |  3.26  |
|  PM2.5 [µg/m³] max   |   3.95   |    3.95    |   3.73   |  3.6   |
+----------------------+----------+------------+----------+--------+
[5]:
Segment Duration (HH:MM) PNC [cm⁻³] PNC [cm⁻³] median PNC [cm⁻³] max PM2.5 [µg/m³] PM2.5 [µg/m³] median PM2.5 [µg/m³] max
0 All data 00:04 197.44 190.10 442.50 3.41 3.40 3.95
1 Background 00:01 222.08 215.57 442.50 3.54 3.57 3.95
2 Emission 00:00 202.94 191.12 314.83 3.53 3.56 3.73
3 Decay 00:01 179.17 176.09 309.72 3.26 3.26 3.60

Exposure assessment

summarize_exposure answers the occupational-hygiene question: given this measurement, what is the exposure, and how does it compare with a limit?

It reports a time-weighted average over twa_window (8 h by convention), the short-term exceedances over short_window (15 min by convention), peak counts, high percentiles, and the time spent above each limit.

[6]:
exposure = elpi.summarize_exposure(
    metric="PM4.2",              # respirable dust
    activities=["Emission"],
    background="Background",     # subtract the mean of this activity
    exposure_hours=None,         # None -> use the measured task duration
    short_limit=1.0,             # STEL, in metric units
    long_limit=1.0,              # 8 h OEL, in metric units
    short_window="15min",
    twa_window="8h",
)
exposure

Exposure summary by segment for metric 'PM4.2' (µg/m³):

+---------------------------------+----------+
|             Metric              | Emission |
+---------------------------------+----------+
|             Metric              |  PM4.2   |
|        Duration [HH:MM]         |  00:00   |
|           Max [µg/m³]           |  7.473   |
|     95th percentile [µg/m³]     |  7.215   |
|     75th percentile [µg/m³]     |  6.951   |
|     50th percentile [µg/m³]     |  6.648   |
|     25th percentile [µg/m³]     |  6.557   |
|     5th percentile [µg/m³]      |  6.276   |
|          Peaks [count]          |    0     |
|          STEL [µg/m³]           |   1.0    |
|      STEL window [offset]       |  15min   |
|      STEL exceedance [min]      |   0.02   |
|      STEL episodes [count]      |    1     |
|     Exposure limit [µg/m³]      |   1.0    |
| Exposure limit exceedance [min] |   0.62   |
|    TWA concentration [µg/m³]    |  6.018   |
|       TWA window [offset]       |    8h    |
+---------------------------------+----------+
[6]:
Segment Metric Duration [HH:MM] Max [µg/m³] 95th percentile [µg/m³] 75th percentile [µg/m³] 50th percentile [µg/m³] 25th percentile [µg/m³] 5th percentile [µg/m³] Peaks [count] STEL [µg/m³] STEL window [offset] STEL exceedance [min] STEL episodes [count] Exposure limit [µg/m³] Exposure limit exceedance [min] TWA concentration [µg/m³] TWA window [offset]
0 Emission PM4.2 00:00 7.473 7.215 6.951 6.648 6.557 6.276 0 1.0 15min 0.02 1 1.0 0.62 6.018 8h

Three arguments carry most of the meaning:

  • ``background`` — either a number, or the name of an activity whose average is used. Subtracting a measured background separates the process contribution from what was already in the room.

  • ``exposure_hours`` — how long the worker is assumed to be exposed. None uses the measured duration of the activity; give a number to extrapolate a short measurement to a full shift.

  • ``metric`` — which quantity the limit applies to.

Compare a short task assumed to last the whole shift against the same task taken at its measured length:

[7]:
measured = elpi.summarize_exposure(metric="PM4.2", activities=["Emission"],
                                   background="Background", exposure_hours=None)
full_shift = elpi.summarize_exposure(metric="PM4.2", activities=["Emission"],
                                     background="Background", exposure_hours=8.0)

twa_cols = [c for c in measured.columns if "TWA" in c]

print("assumed duration = measured task length")
print(measured[["Segment"] + twa_cols].to_string(index=False))
print()
print("assumed duration = 8 h")
print(full_shift[["Segment"] + twa_cols].to_string(index=False))

Exposure summary by segment for metric 'PM4.2' (µg/m³):

+---------------------------------+----------+
|             Metric              | Emission |
+---------------------------------+----------+
|             Metric              |  PM4.2   |
|        Duration [HH:MM]         |  00:00   |
|           Max [µg/m³]           |  7.473   |
|     95th percentile [µg/m³]     |  7.215   |
|     75th percentile [µg/m³]     |  6.951   |
|     50th percentile [µg/m³]     |  6.648   |
|     25th percentile [µg/m³]     |  6.557   |
|     5th percentile [µg/m³]      |  6.276   |
|          Peaks [count]          |    0     |
|          STEL [µg/m³]           |   1.0    |
|      STEL window [offset]       |  15min   |
|      STEL exceedance [min]      |   0.02   |
|      STEL episodes [count]      |    1     |
|     Exposure limit [µg/m³]      |   1.0    |
| Exposure limit exceedance [min] |   0.62   |
|    TWA concentration [µg/m³]    |  6.018   |
|       TWA window [offset]       |    8h    |
+---------------------------------+----------+

Exposure summary by segment for metric 'PM4.2' (µg/m³):

+---------------------------------+----------+
|             Metric              | Emission |
+---------------------------------+----------+
|             Metric              |  PM4.2   |
|        Duration [HH:MM]         |  00:00   |
|           Max [µg/m³]           |  7.473   |
|     95th percentile [µg/m³]     |  7.215   |
|     75th percentile [µg/m³]     |  6.951   |
|     50th percentile [µg/m³]     |  6.648   |
|     25th percentile [µg/m³]     |  6.557   |
|     5th percentile [µg/m³]      |  6.276   |
|          Peaks [count]          |    0     |
|          STEL [µg/m³]           |   1.0    |
|      STEL window [offset]       |  15min   |
|      STEL exceedance [min]      |   0.02   |
|      STEL episodes [count]      |    1     |
|     Exposure limit [µg/m³]      |   1.0    |
| Exposure limit exceedance [min] |   0.62   |
|    TWA concentration [µg/m³]    |  6.722   |
|       TWA window [offset]       |    8h    |
+---------------------------------+----------+
assumed duration = measured task length
 Segment  TWA concentration [µg/m³] TWA window [offset]
Emission                      6.018                  8h

assumed duration = 8 h
 Segment  TWA concentration [µg/m³] TWA window [offset]
Emission                      6.722                  8h

Choosing a metric

For size-resolved data the metric can be any of:

  • "PNC" — total number concentration

  • "MASS" — total mass concentration

  • "PM<x>", "PN<x>", "PS<x>", "PV<x>" — cumulative mass, number, surface or volume below cut diameter <x> in µm, e.g. "PM2.5"

  • "PM<a>-<b>" — the band between two diameters, e.g. "PM1-4", computed with the EN 481 / ISO 7708 penetration curves

Several activities can be summarised in one call, which is the convenient form when comparing tasks.

[8]:
elpi.summarize_exposure(
    metric="PM10",
    activities=["Emission", "Decay"],
    background="Background",
    long_limit=5.0,
    short_limit=10.0,
)

Exposure summary by segment for metric 'PM10' (µg/m³):

+---------------------------------+----------+-------+
|             Metric              | Emission | Decay |
+---------------------------------+----------+-------+
|             Metric              |   PM10   | PM10  |
|        Duration [HH:MM]         |  00:00   | 00:01 |
|           Max [µg/m³]           |  0.246   | 0.208 |
|     95th percentile [µg/m³]     |  0.194   | 0.153 |
|     75th percentile [µg/m³]     |  0.165   | 0.126 |
|     50th percentile [µg/m³]     |  0.154   | 0.111 |
|     25th percentile [µg/m³]     |  0.139   | 0.099 |
|     5th percentile [µg/m³]      |  0.121   | 0.084 |
|          Peaks [count]          |    1     |   0   |
|          STEL [µg/m³]           |   10.0   | 10.0  |
|      STEL window [offset]       |  15min   | 15min |
|      STEL exceedance [min]      |   0.0    |  0.0  |
|      STEL episodes [count]      |    0     |   0   |
|     Exposure limit [µg/m³]      |   5.0    |  5.0  |
| Exposure limit exceedance [min] |   0.0    |  0.0  |
|    TWA concentration [µg/m³]    |  0.132   | 0.131 |
|       TWA window [offset]       |    8h    |  8h   |
+---------------------------------+----------+-------+
[8]:
Segment Metric Duration [HH:MM] Max [µg/m³] 95th percentile [µg/m³] 75th percentile [µg/m³] 50th percentile [µg/m³] 25th percentile [µg/m³] 5th percentile [µg/m³] Peaks [count] STEL [µg/m³] STEL window [offset] STEL exceedance [min] STEL episodes [count] Exposure limit [µg/m³] Exposure limit exceedance [min] TWA concentration [µg/m³] TWA window [offset]
0 Emission PM10 00:00 0.246 0.194 0.165 0.154 0.139 0.121 1 10.0 15min 0.0 0 5.0 0.0 0.132 8h
1 Decay PM10 00:01 0.208 0.153 0.126 0.111 0.099 0.084 0 10.0 15min 0.0 0 5.0 0.0 0.131 8h

Single-channel instruments

A 1D instrument has no size distribution, so the metric is its own measurement — "PNC" for a CPC.

[9]:
cpc = at.load_cpc_file("../../tests/data/Sample_CPC_AIM.txt")
cpc.mark_activities({
    "Task": [("2023-08-14 11:14:00", "2023-08-14 11:18:00")],
})

cpc.summarize_exposure(
    metric="PNC",
    activities=["Task"],
    exposure_hours=8.0,
    long_limit=1e4,
    short_limit=2e4,
)

Exposure summary by segment for metric 'PNC' (cm⁻³) [1D]:

+---------------------------------+----------+
|             Metric              |   Task   |
+---------------------------------+----------+
|             Metric              |   PNC    |
|        Duration [HH:MM]         |  00:03   |
|           Max [cm⁻³]            |  1559.0  |
|     95th percentile [cm⁻³]      |  1511.0  |
|     75th percentile [cm⁻³]      |  1457.0  |
|     50th percentile [cm⁻³]      |  1415.0  |
|     25th percentile [cm⁻³]      |  1366.5  |
|      5th percentile [cm⁻³]      |  1307.0  |
|          Peaks [count]          |    0     |
|           STEL [cm⁻³]           | 20000.0  |
|      STEL window [offset]       |  15min   |
|      STEL exceedance [min]      |   0.0    |
|      STEL episodes [count]      |    0     |
|      Exposure limit [cm⁻³]      | 10000.0  |
| Exposure limit exceedance [min] |   0.0    |
|    TWA concentration [cm⁻³]     | 1412.165 |
|       TWA window [offset]       |    8h    |
+---------------------------------+----------+
[9]:
Segment Metric Duration [HH:MM] Max [cm⁻³] 95th percentile [cm⁻³] 75th percentile [cm⁻³] 50th percentile [cm⁻³] 25th percentile [cm⁻³] 5th percentile [cm⁻³] Peaks [count] STEL [cm⁻³] STEL window [offset] STEL exceedance [min] STEL episodes [count] Exposure limit [cm⁻³] Exposure limit exceedance [min] TWA concentration [cm⁻³] TWA window [offset]
0 Task PNC 00:03 1559.0 1511.0 1457.0 1415.0 1366.5 1307.0 0 20000.0 15min 0.0 0 10000.0 0.0 1412.165 8h

Saving the result

Both summary functions accept filename and append to a CSV or Excel file, so results from several measurements accumulate in one place.

elpi.summarize_activities(filename="campaign_summary.xlsx",
                          sheet_name="2023-09-07")

Next: 5 — Plotting.