A drift detector that measures its own noise

00Install

Python 3.10 or newer. Two runtime dependencies, both of which you already have.

1. Install the package

   No extras, no optional backends. SciPy and scikit-learn are test dependencies and are never imported at runtime.

   pip install driftwatch

2. Compare two windows

   A reference frame and a current frame. The report is a list of typed Finding objects that renders to JSON, Markdown or a terminal.

   import driftwatch as dw
   
   report = dw.compare(reference_df, current_df)
   print(report.to_terminal())
   
   if report.failed:          # any CRITICAL finding
       raise SystemExit(1)

3. Or run it as a CI step

   Same shape from a shell, with the exit code doing the work.

   driftwatch check reference.parquet current.parquet --config drift.toml
   # exit 0 = clean · 1 = critical drift · 2 = the check itself broke

01What it does

You give it a reference window — the training set, a signed-off month, or simply yesterday — and a current window. It compares them column by column and returns a report of Finding objects.

Each finding carries the feature, the metric, both sample sizes, the value, a p-value where one exists, the adjusted p-value, the threshold that produced the verdict, a severity, and any caveats the computation earned along the way. The same objects render as JSON, Markdown or terminal output, and round-trip losslessly.

DriftWatch  CRITICAL
  reference (20,000 rows)  ->  current (20,000 rows)
  4 critical | 3 warning | 13 ok | 20 checks

        FEATURE  METRIC                VALUE      P(ADJ)    THRESH        N REF->CUR
  CRIT  age      wasserstein          0.4194           -    0.2500    20,000->20,000
  CRIT  region   chi2                 0.1959    <1e-308*    0.1500    20,000->20,000
  CRIT  region   frequency_shift      0.1502           -    0.1500    20,000->20,000
  CRIT  income   missingness          0.0877    <1e-308*    0.0500    20,000->20,000
  WARN  age      ks                   0.1700     2e-251*    0.1000    20,000->20,000
  ... 2 more

  10 tests corrected with benjamini_hochberg (alpha=0.05): 3 significant raw -> 3 after correction

  caveats
    - income/missingness: column had no nulls in the reference window and 8.77%
      now; the rate ratio is undefined, judge this on the absolute rate

The exit code is the part that matters in automation:

The split between 1 and 2 is the point. Conflating them turns a broken monitoring job into a silent pass, which is the failure this library exists to make impossible.

02The engineering model

A comparison is six stages. Nothing is stateful; nothing is learned. The same two frames always produce the same report.

Stage 05 is where the library takes its position. A metric never decides its own severity, and a p-value never becomes an alert on its own.

03What gets measured

Metrics are chosen so that they disagree. When PSI is quiet and Wasserstein is loud, that difference is information about where the mass went.

Two deliberate choices

chi2 reports Cramér’s V, not the raw statistic. The raw χ² grows with sample size and is useless as a threshold; V is bounded in [0, 1] and is not. Likewise wasserstein divides by a reference scale by default, so its value reads as standard deviations of mass movement rather than as unusable raw feature units.

The performance metrics ship with no default thresholds at all. There is no universal accuracy floor, and inventing one would produce confident nonsense. You set them from your own baseline or they never fire.

04Three questions, kept apart

Most drift tooling collapses these into one number. Keeping them separate is the library’s main architectural commitment.

A policy escalates on magnitude, and only when the shift is also distinguishable from noise. "Distinguishable from noise" is established with whichever tool the metric actually supports: metrics carrying a p-value must clear alpha; magnitude-only metrics must clear a multiple of their noise floor. The two are never applied to the same metric, because double-gating would silently halve its sensitivity.

A third gate caps severity on small windows. Below min_samples observations on either side, a finding can warn but cannot go critical — you still see it, it just will not fail your build on thirty rows.

05The noise floor

Every divergence in this library is non-negative. PSI, Wasserstein, total variation, Jensen–Shannon and Cramér’s V cannot go below zero, so sampling noise can only push them up. Two independent draws from the same distribution produce a positive value, and the smaller the windows the larger it is.

This is the single biggest source of spurious drift alerts in practice, and it is why "PSI > 0.1 means investigate" is not a decision rule. It is a decision rule conditional on a sample size nobody wrote down.

So DriftWatch computes the floor. Each metric reports what it would read under the null, given the two sample sizes, as details["expected_null"].

What that looks like

Noise-floor values for a four-category feature and a ten-bin numeric feature, equal window sizes:

At 500 rows a side, the default warning line for total variation (0.05) is 1.2× the noise floor. At 10,000 rows it is 5.4×. The same threshold means completely different things.

The guard

A magnitude-only metric must exceed noise_multiple × expected_null — 2.0 by default — before a policy will escalate it. Measured on this project’s synthetic scenarios, 200 fixed-seed replicates per row:

Detection is unaffected: each of four drift scenarios is caught in 40 of 40 fixed seeds at both n=600 and n=4,000, where detection means at least one warning-or-above finding on the feature the scenario actually moved.

06Multiple testing

A report over 200 features × 2 test-bearing metrics runs 400 hypothesis tests. At α = 0.05 with nothing whatsoever wrong, roughly 20 come back significant. Any monitoring job that alerts on raw p-values therefore alerts every single run, and the team learns to ignore it.

Benjamini–Hochberg runs across the whole report by default, and both counts land in the metadata so the correction is visible rather than implied:

16 tests corrected with benjamini_hochberg (alpha=0.05): 9 significant raw -> 8 after correction

Holm and Bonferroni are available when a single false alarm is expensive — a page at 3am, an automated rollback. Benjamini–Yekutieli is there for when positive dependence cannot be assumed; at 400 tests it is about 6.6× more conservative.

07Labels that arrive late

A model scores a loan today and learns whether it defaulted in eighteen months. Evaluating it means joining prediction events to labels, and the join is where the interesting mistakes live.

Scoring on whatever labels have arrived conditions on label arrival — and arrival time is very often correlated with the outcome. Confirmed fraud lands faster than cleared transactions. Churn is only observable at the end of a subscription period. A ticket resolved in an hour is not a random sample of tickets.

In a synthetic log where positives are confirmed in 1–5 days and negatives take 25–35, the churn rate among arrived labels is 54%. Among mature events it is 47%. More rows only estimate the biased quantity more precisely.

The defence is to evaluate only mature events — those old enough that a label would have arrived by now if one were ever going to. The join is deliberately a left join, so unlabelled predictions stay visible as NaN and are never quietly coerced to a negative class. When label coverage moves between the two windows, every performance finding carries an explicit caveat saying part of the change may be selection rather than the model.

08What it will not claim

Every metric here measures P(X), the distribution of the inputs, or P(Ŷ), the distribution of the predictions. Model quality depends on P(Y|X), the relationship between inputs and outcome. Those are different objects.

P(X) can be completely unchanged while P(Y|X) changes.

That is concept drift, and no metric in this library can detect it. A fraud model can degrade to uselessness while every feature distribution is pixel-identical, because the fraudsters changed what the same features mean. A demand model can fail after a competitor’s price change that never appears in its own inputs.

The reverse holds too: P(X) can change a great deal while P(Y|X) is stable, in which case the model is fine and the alert is noise you should not act on.

Only the performance.* and calibration.* findings — which need labels — are evidence about the model. A green report with labels still pending means your inputs look familiar. It does not mean your model works.

The assumption you must check yourself

The noise floors are derived for independent observations. Clustered, repeated-entity, longitudinal or autocorrelated data carries less information than its row count implies, so its true floor is higher than DriftWatch reports and the guard built on it is correspondingly too permissive. Nothing in the library detects dependence. reference_n and current_n are raw row counts, not effective sample sizes. Deduplicate to one row per entity, or raise the multiple.

09Verification

Every statistical quantity the library computes is pinned against a reference implementation, and every noise floor against simulation. SciPy and scikit-learn are test dependencies; the library itself imports neither, and a CI job asserts they are not importable at runtime.

Cost

Measured on an Apple M-series laptop, single core, median of seven repeats. These numbers describe that machine and workload only — wall-clock figures move 10–20% between runs and much more across hardware. The shapes are what transfer.

Peak additional allocation — working memory beyond the frames themselves — tracks rows and not columns: roughly 16–19× the size of one float64 column. That is 13–16 MB at 100k rows whether the frame has 25 columns or 500, and 208 MB at 1.6M rows. Columns are compared one at a time, so width costs time rather than memory. It is O(rows) with a small constant, not constant memory.

What an audit found

A final adversarial pass over the finished library turned up five defects worth naming, because each is the kind that hides:

Three of the project’s own published claims were corrected in the same pass, for being measured at one sample size and stated as general.