Quality control for weather observations. Catches the readings that are wrong but look fine: sensors stuck on one value, dew points above the air temperature, feeds that froze yesterday, a station reading 25 °C while everything around it reads 12.
Flags follow QARTOD, the scheme NOAA's observing systems use, so a consumer that already speaks it needs no translation.
No dependencies. TypeScript, ESM and CJS, works in Node and in the browser.
An API that goes down is obvious. A station that keeps reporting 12.0 °C for three days because its thermometer froze is not — the data looks perfectly reasonable, it arrives on time, and it is completely wrong. That is the failure this library exists to catch.
It was written for snowy.es, which aggregates weather stations from six public Spanish networks — AEMET, SIAR, Meteocat, MeteoGalicia, Euskalmet and Wunderground — and therefore has the problem in its sharpest form: when one network starts lying, nothing in the data says so.
Aggregating several networks also turns out to be the cure. A station can be checked against its neighbours from a different network, which is something no single operator can do for itself.
import { qualityControl, FLAG, explain } from 'weather-qc'
const report = qualityControl({
variable: 'airTemperature',
observations: [
{ timestamp: 1672531200000, value: 8.2 },
{ timestamp: 1672534800000, value: 8.1 },
{ timestamp: 1672538400000, value: 7.9 },
],
})
report.flags // [1, 1, 1] — one per observation
report.counts // { 1: 3, 2: 0, 3: 0, 4: 0, 9: 0 }
explain(report, 2) // [] — nothing to say about a good readingServing only what passed:
import { isUsable } from 'weather-qc'
const trustworthy = observations.filter((_, index) => isUsable(report.flags[index]))A whole station at once, which adds the checks that need more than one variable:
import { qualityControlStation } from 'weather-qc'
const reports = qualityControlStation([
{ variable: 'airTemperature', observations: temperature },
{ variable: 'dewPoint', observations: dewPoint },
{ variable: 'relativeHumidity', observations: humidity },
])
reports.get('dewPoint')?.counts| check | catches | verdict |
|---|---|---|
timeline |
duplicate, reversed or future timestamps | fail |
physicalRange |
outside what an instrument can be measuring | fail |
climatology |
outside what is normal for that place, that month | suspect |
stuckSensor |
a value repeating for longer than the variable plausibly holds still | suspect |
spike |
a reading that jumps away from its neighbours and comes straight back | suspect |
rateOfChange |
movement faster than the variable plausibly moves | suspect |
staleness |
the newest reading is too old to present as current | suspect |
internalConsistency |
dew point above temperature, gust below mean wind, the three humidity variables disagreeing | suspect |
neighbours |
a station disagreeing with the stations around it | suspect |
Only physically impossible readings fail. Everything else is suspect, because real weather
does break statistical limits: a chinook once moved temperature 27 °C in two minutes, and
discarding those readings would discard the interesting ones. fail means the instrument or the
decoding is broken; suspect means look before you trust it.
Checks that cannot reach a verdict return notEvaluated rather than pass. A check that had
nothing to work with should say so, not quietly approve.
The range check only knows what is physically possible, and 35 °C passes everywhere. In Madrid in August it is a warm day; in January it is a broken sensor.
qualityControl(series, {
climatology: {
monthly: {
1: { min: -5, max: 18 },
8: { min: 12, max: 42 },
},
},
})The ranges come from you, because they are a property of the site rather than of the variable, and whoever runs this already has the history to derive them. A month with no entry is not evaluated rather than passed.
Direction is circular: 359° and 1° are two degrees apart, and every linear check reads that as a
358° swing. spike, rateOfChange and neighbours use the shorter arc for differences and a
vector mean for averages, so they work on direction rather than being switched off for it.
The neighbour check also declines when the surrounding stations point every which way — an average direction only means something when there is a direction to average.
Every check is a pure function over the series it is given, which means a service polling every ten minutes would never see a sensor stuck for twelve hours: each call only sees its own window.
Keep the window yourself and hand it in whole:
const window = [...previousReadings, ...justFetched].filter(
(o) => o.timestamp > Date.now() - 24 * 3600_000,
)
qualityControl({ variable: 'airTemperature', observations: window })No state lives in this library, so where that window is kept — memory, Redis, a previous snapshot — stays your decision.
The neighbour check needs geography to be worth anything. Give it locations and a radius:
import { neighbours } from 'weather-qc'
const outcome = neighbours(target, others, {
location: { latitude: 42.45, longitude: -2.33, elevation: 363 },
radiusKm: 100,
})With elevations, temperature and dew point readings are brought to the target's altitude at 6.5 °C per km before comparing, so a valley station is not judged against a mountain one.
Station pressure gets its own correction, barometric rather than linear: a neighbour's reading is scaled by the ratio of standard pressures at the two altitudes before comparing. Without it a sea level barometer and a summit one differ by 200 hPa and both are right.
Humidity gets the opposite treatment. There is no lapse rate for it — above a trade wind inversion it sits near 15% while the valley below reads 90%, and both are right — so instead of correcting, neighbours more than 400 m apart in height are dropped. Correct what you know how to correct; discard what you do not.
Without a location every neighbour stays in, on the assumption that the caller already chose them. Running it on stations picked at random flags about a quarter of everything, which is a good way to prove the point.
Run against the snapshot behind the snowy.es map — 1,855 stations across six Spanish networks, all measured at the same moment:
staleness 588
neighbours 32
todayExtremes 4
network readings flagged rate
AEMET 4162 505 12.1%
SIAR 1979 83 4.2%
METEOCAT 828 19 2.3%
METEOGALICIA 155 3 1.9%
WUNDERGROUND 327 6 1.8%
EUSKALMET 347 2 0.6%
Two of those findings were about the checks rather than the stations, and both are the point of running it on real data:
SIAR came back 100% stale on the first run. It publishes three times a day, so a three-hour
threshold accuses the entire network forever. Freshness has to follow each source's own cadence —
which is why staleness now infers the limit from the series when you do not set one.
The worst neighbour outliers were all mountains. Station pressure at 2,200 m sits 200 hPa below the valley beside it and both readings are right. Without elevations the check does not produce a weak signal, it produces a wrong one, so it now declines to judge station pressure at all unless it knows the altitude. That took neighbour hits from 108,469 to 32.
What is left is real: one station 55 hours stale and still on the map, four reporting a current temperature outside their own daily maximum, and a handful of genuine outliers.
Reproduce this one yourself — the data is public and needs no account:
npx tsx scripts/check-real-data.ts <directory of NOAA ISD-Lite .isd files>A year of hourly observations from ten stations in northern Spain, 430,000 readings, nothing prepared for it:
neighbours 4,736
spike 155
stuckSensor 39
rateOfChange 22
internalConsistency 4
timeline 0
physicalRange 0
staleness 0
The four internalConsistency hits are unambiguous data errors — a dew point of 8 °C recorded
alongside an air temperature of 7.2 °C, which cannot happen. The stuckSensor hits are twelve
straight hours at 3 °C in the Ebro valley in January, which is either persistent fog or a frozen
thermometer, and worth a look either way.
The honest limit is the neighbour check: it cannot tell a broken station from a real microclimate. Most of its 4,736 hits are a coastal station being compared with inland ones 150 km away, which genuinely do read differently. Tighten the radius or raise the threshold for your network.
The defaults come from running the checks over that year of data and looking at what fired. Two that matter:
Stuck sensors are measured in hours, not readings. Six identical readings means six hours at hourly cadence and one hour at ten-minute cadence, which are not the same claim. Pressure and wind direction are allowed to sit still far longer than temperature, because an established synoptic pattern holds them there — a threshold tight enough for temperature reports most of an anticyclone as broken.
Precipitation at zero is never stuck. It is the normal state of a rain gauge.
Every threshold is overridable per call.
Written for snowy.es and calibrated against its network, but not yet running in its production ingest — that integration is planned, not done. Everything above is the library run against real snapshots and archives, which is a different claim and a weaker one.
Said plainly because the whole argument here is that the numbers come from data rather than from guesses, and that argument does not survive one convenient exaggeration.
MIT