pkgxray calibration — methodology

What was scanned

5,000 published packages in one static pass, across download-ranked target lists resolved around 2026-07-21:

1,000 — top-1000 calibration list: the false-block denominator; committed with resolved name@version rows. inputs
1,000 — deeper download-ranked sweep (first): false-block hunt beyond the head. inputs
3,000 — deeper download-ranked sweep (second): false-block hunt beyond the head. inputs

Counted separately and not part of the 5,000-package denominator above:

300 — MCP cohort: hunting list — NOT part of the 5,000 denominator. inputs
21 — reconstructed known-malware corpus: catch-rate corpus — NOT part of the 5,000 denominator. inputs

The top-1000 list is committed with resolved name@version rows. The 4,000 extended-sweep targets were selected from the committed download-rank pool (validation/calibration-2026-07-22/pool-npm-high-impact-1.13.0.json) via scripts/build-extended-target-list.js --want 4000; the resolved 4,000-name snapshot was not committed, so that portion is reproducible in method but not byte-identical.

Exact tool and command

pkgxray 1.0.6 (build 76c0768, Node v22.22.2), one invocation per package:

pkgxray guard npm:<name>@<version> --format json

Engine 1.0.6 was a pre-release build, not published to npm — npx [email protected] will not resolve. The public npm release at run time was 1.0.4. To reproduce on the exact engine, check out the commit:

git checkout 76c0768 && node bin/audit.js guard npm:<name>@<version> --format json

Static only — the scanner reads the tarball's bytes and queries OSV; the static scan never executes package code and never connected to any hosted endpoint. Exit codes: 0 safe, 2 block, 3 review. A scan that failed to produce a parseable verdict is recorded as a scan error, never counted as safe.

How false blocks were adjudicated

A block has two very different causes, and they are split before anything is called a false positive:

OSV block — the only high finding is a known CVE. Blocking here is by design and is not a false positive.
Heuristic block — a malware-signal finding (install hook, exec, exfil, obfuscation, credential/agent access…). These are the reputation-staking calls, and every one was read by hand.

Every top-1000 block was a real CVE (OSV, by design); zero were heuristic false positives. The one pre-retune heuristic false positive — a ~30-line utility that reads .npmrc solely to return the configured registry URL, never touching the auth token, with no network egress at all — was minimized into a benign corpus fixture and the over-firing heuristic retuned, so on this engine it resolves to review. Re-running the full 1,000-package list on the fixed engine confirms 0.0% = 0 / 1,000, with no new false block.

Reconciliation note (read this)

This run is the re-scan on the retuned engine. An earlier provisional run (2026-07-21, kept unedited in run history) published the as-measured pre-retune figure of one top-1000 false block, flagged as pending a re-run. It now has one: re-running the full 5,000-package scan on the fixed engine yields zero top-1000 false blocks, and the earlier run's number was left in place rather than silently edited. This is also the scoping correction that reconciled the README's earlier "0 false blocks" line: that claim held only on a stale 2019 dependents list; the figure here is corpus-gated and scoped to "the top-1000 most-downloaded."

Catch rate and the corpus

npm removes confirmed malware, so live-registry recall is untestable — of 22 curated known-malicious versions, only one is still downloadable. Catch rate is therefore measured against a committed, reconstructed corpus run through the real static engine (node benchmark/run.js): 20 of 21 malicious samples block outright (95.2%), and 0 passed as safe — the remainder are routed to review by policy (download-then-exec and geo/locale-gated payloads), not missed. CI hard-fails on any false block, any full miss, and any recall regression.

Reproduce it

The committed top-1000 target list (names, versions, and download counts — inputs only, no verdicts) is here: https://github.com/adamsjack711-ux/pkgxray/tree/main/validation/calibration-2026-07-19.

Check out the exact engine and run one invocation per package over the top-1000 list to re-derive the verdict distribution and the false-block count:

git checkout 76c0768 && node bin/audit.js guard npm:<name>@<version> --format json

The corpus and benchmark harness live in the repository under benchmark/ (node benchmark/run.js).