0.2.0

Published Sep 10, 2026

Bug Fixes

• metadata: Resolve versioned profile paths and send a User-Agent (ddb5833)

Two bugs on the same path, ported from the fairspec-typescript fix.

The registry keyed the five vendored profiles only on https://fairspec.org/profiles/latest/<name>.json, but every save_* action stamps a versioned $schema from FAIRSPEC_VERSION, and both lookups match the URL by exact string equality. So the library wrote a URL its own bundle could never match: every saved descriptor missed the cache and went remote.

That remote load then failed outright. _load_remote_descriptor passed a bare URL to urlopen, so requests carried the default Python-urllib agent, which fairspec.org (Cloudflare) answers with 403:

saved $schema: https://fairspec.org/profiles/0.5.0/dataset.json validate : HTTPError: HTTP Error 403: Forbidden

Bump FAIRSPEC_VERSION to 0.6.0 to match what the bundle already is -- the vendored dataset.json carries the object Integrity that shipped as spec 0.6.0 -- and derive the registry so each profile answers to both "latest" and the pinned version. Versions the bundle does not carry still go remote, which is correct: they have different rules.

Add a USER_AGENT setting and send it from both urlopen callers that target fairspec.org: the descriptor loader and resolve_basepath, which 403s the same way.

Collapse load_profile into load_json_schema, which already does registry-then-network plus caching, removing the duplicated lookup.

Every fixture and default used "latest", so nothing covered this. Add an offline save -> validate round-trip, registry resolution tests, and a User-Agent assertion; all fail without the fix.

• table: Emit tuple for nullable column types in sqlite plugin (717ede5)

The sqlite table_schema conversion carried a copy of the shared _make_property_nullable helper with tuple changed to list, so it assigned ["string", "null"] to a field annotated as a tuple of literals. Pydantic models do not validate on assignment, so the bad value was stored silently and only surfaced later at model_dump() time as serializer warnings.

Extract the helper as a public set_property_nullable in fairspec_metadata and call it from both the shared polars inferrer and the sqlite plugin, so the two cannot diverge again. Add filterwarnings = ["error::UserWarning"] to fail the suite on any future occurrence.

Closes #2

• table: Tag cell and row errors instead of embedding a template (8bb9a49)

A polars literal is broadcast to every row, so putting the JSON error template in the then branch materialized a full-length string column for each of the 14 cell checks. On a 500MB file that made one column's check cost ~2.7GB, and site B validates columns concurrently, so the cost was multiplied by the worker count and 500MB no longer fitted in 6GB.

Carry a UInt8 tag in the error column instead and resolve it back to the template in Python once the frame has been filtered to the reported errors. Row key checks own a single template each, so they only need a boolean flag and can read the template from the closure.

Isolated on the same file, 14 layers cost 2754MB with the JSON payload and 652MB with a short one; the layering itself is free. End to end, 500MB and 1GB validations that were OOM-killed under a 6GB cap now complete, and the JSON round-trip disappears from the hot path.

Output is unchanged: 1472 tests pass, and the invalid 500MB fixture still reports exactly its one error.

• test: Point fixture URLs at the current repo and path (7f09ffa)

Both specs fetch a fixture from the fairspec-typescript repo, still naming the fairspec org that was renamed to datisthq and a path that predates the -test folder layout. The csv spec ran against the live URL and had been failing with a 404.

VCR matches on URI, so the two prefetch cassettes are re-recorded.

Chores

• Omit _test folders from coverage (ff3fff0)

The specs live inside the measured source packages, so until now they counted toward their own coverage. The _test convention makes them matchable by path for the first time.

• Rename test files to _unit.py to match typescript (fe4d303)

The TypeScript repo names every test file <module>.unit.ts. Rename the 139 Python specs from <module>_spec.py to <module>_unit.py, and the taskipy task from spec to unit so uv run task unit mirrors pnpm unit.

Python cannot use the literal .unit.py: a dot is not legal in a module name, and 132 of the specs import their subject relatively, so the package would be computed one level too deep and every from ..load import would break. AGENTS.md now records that alongside the existing note on why directories are _test and _shared rather than -test.

Also updates the gitleaks allowlist, which is keyed on the filename suffix and would otherwise have silently stopped exempting tests from the secret scan.

Test files are renames only, with no content change; all 1467 tests still pass and the 3 VCR cassettes are untouched, since their names key on the test class rather than the module.

• Update GitHub org to datisthq (9a868c7)

• deps: Bump vcrpy in the uv group across 1 directory (#18, df3adeb)

Bumps the uv group with 1 update in the / directory: vcrpy.

Updates vcrpy from 8.1.1 to 8.2.1

• Release notes
• Changelog
• Commits

updated-dependencies:

• dependency-name: vcrpy dependency-version: 8.2.1

dependency-type: indirect

dependency-group: uv ...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

• deps: Migrate to polars 1.44 (dc69b6f)

polars 1.44 tightens the DataFrame constructor signature, so the decoded JSON buffer needs a cast at the call site. That is the only source change the upgrade needs — unlike nodejs-polars 0.26, the Python binding keeps the Expr.dt namespace unchanged.

Not a performance win. On the 1 GB validation benchmark it measured 31% slower than 1.38.1 (5.7s to 7.4s) for 18% less peak memory (2393MB to 1970MB), and it carries the scan_csv memory regression tracked in pola-rs/polars#27539. It also does not bring a usable streaming engine for this workload: CSV is not a streaming source.

• metadata: Track fairspec spec 0.6.1 (eb21f8c)

0.6.1 re-tags 0.6.0 with no rule changes -- the only difference is the version segment in dataset.json's internal $refs, which the bundled copy writes as "latest" and resolves through the registry either way. The vendored profiles are already content-identical to live 0.6.1, so this is the constant alone; the registry re-keys itself from it.

• project: Add a project package owning repo docs and the docs guard (6696bee)

The docs-consistency guard lived in fairspec/fairspec/_test/, inside the umbrella package whose entire source is init.py and main.py. It reads AGENTS.md, README.md and CONTRIBUTING.md and reaches up to the repo root, so it has nothing to do with the package hosting it.

Port the convention from fairspec-typescript: a directory that nothing depends on, holding the skills Claude Code reads through .claude/skills and the guard that checks this repo's prose. Deliberately not a uv workspace member and carrying no pyproject.toml -- it ships nothing, and a manifest would invite adding it to version_toml later.

skills/ carries a .gitkeep because git cannot commit an empty directory and the symlink would otherwise dangle on a fresh clone, failing the new assertion in CI while passing locally.

Also corrects a stale line: fairspec-typescript renamed agent to mcp-server, so the cross-repo note named a package that no longer exists.

Documentation

• Port AGENTS.md structure from mycarro-next (b17675c)

Rewrite AGENTS.md around the sections mycarro-next uses: Rules, Modules with the package DAG, Code structure including the new _test/_shared convention, Conventions, and a calibrated comment rule in place of a blanket ban.

Correct a claim that was never true: the Specs section asked for test_<module>.py files, while pyproject.toml has always set python_files = "*_spec.py" and no file in the repo matched the documented pattern. The taskipy tasks for build, publish and coverage and the whole pnpm side — leaks, vulns, scan, deps, docs:start, docs:build — went undocumented.

CONTRIBUTING.md still cloned from github.com/yourusername, described tests as merely collocated, and gave a placeholder path as the single-test example.

• Update citation author email (526c8cd)

• project: Move the documentation site into the project package (5bf0613)

The livemark sources sat at the repo root alongside config. They belong with the package that owns documentation.

livemark.config.ts stays at the root -- livemark resolves its config from the cwd, and docs:build/docs:start run from there -- so only the include glob moves. README.md and CONTRIBUTING.md stay at the root too: the patches block keys off those filenames and GitHub resolves them there.

Every page carries an absolute path: in its frontmatter, so the published URLs are unchanged. The guard's package list drops "docs" for "project" now that docs/ is no longer a root directory.

Features

• Run validation and inference in parallel (dd47e96)

The TypeScript codebase fans out concurrently in six places; the Python port turned every one into a sequential loop, and the workspace had no concurrency at all.

Add a shared bounded ThreadPoolExecutor in fairspec_dataset.helpers and restore all six sites: per-resource validate and infer, per-column and per-row-key checks, prefetch, and hash reads. A single concurrency option on InferTableSchemaOptions is inherited by LoadTableOptions and ValidateTableOptions, defaulting to os.cpu_count().

Report output is unchanged. Each task's results are collected separately and concatenated in input order, and the column and row-key loops are chunked so the existing max_errors early exit still applies. A 143KB report is byte-identical at concurrency 1, 4 and 12, and the whole suite passes with the default forced to 1.

Two deliberate differences from the TypeScript original:

• Nested fan-out runs inline. Submitting into a bounded pool from inside that pool deadlocks once every worker blocks on an inner future, so a thread-local marker makes inner calls run in the current worker. Total threads can never exceed the pool width; TypeScript instead applies its limit per call, allowing cpu^2 tasks in flight. - infer_hash parallelises the file read rather than only the stream open, since concat_file_streams reads them serially afterwards.

Threads rather than processes, so the remaining GIL-bound work scales for free once polars ships free-threaded wheels.

Closes #3

Performance Improvements

• table: Cap column check concurrency at four (1e278d3)

Each column check scans the whole source, so peak memory grows with the number in flight while the speed stops improving well before it. On the NYC 311 sample, 1M rows across 41 columns, validation takes 6.8s in 384MB serially, 3.5s in 1GB at four checks, and 3.4s in 1.5GB at one per core: the last eight workers buy a tenth of a second for half a gigabyte.

Four is where the curve flattens. Wide tables are what expose this — the earlier 5 and 10 column benchmarks never had enough columns in flight to show it. Callers can still override with the concurrency option, and row key checks are left alone since a schema rarely has more than a couple.

• table: Collect checks with the streaming engine, restore concurrency (5cd6efc)

The in-memory engine holds the whole source while it collects, so peak memory grew with the file and checks had to run one at a time. py-polars exposes the streaming engine on collect(), which keeps a bounded working set: the minimum memory a 1 GB validation survives stops tracking file size and sits at 384MB for 50MB, 500MB and 1GB alike, of which 128MB is just importing the library.

That makes per-core concurrency affordable again, so the two INSPECT_* limits added in fc463ea are gone and the checks run at cpu_count once more. Against that serial-and-in-memory baseline: 100MB 0.93s to 0.34s, 500MB 4.0s to 1.1s, 1GB 7.0s to 2.6s, all now fitting in a 1G cap where the pre-cap code needed more than 6G.

Applied to the collect sites that scan the whole source. The bounded head(sample_rows) and head(HEAD_ROWS) collects are left alone; they read a fixed number of rows and gain nothing.

Note this diverges from fairspec-typescript, which keeps its checks serial: nodejs-polars exposes streaming on collectSync only, so the same fix is not reachable there.

• table: Execute save plugins with the query engine (0aee3a5)

The parquet and arrow plugins already sink, so they only needed the engine passed. The csv plugin collected the whole frame and then wrote it, even though sink_csv accepts the same four options it was building — the local was already called sink_options — so it now sinks directly. Saving a 500MB CSV drops from a 1536MB floor to 384MB, the same bounded working set validation gets.

json, xlsx and sqlite have no polars sink: the first decodes the written text back to Python to reshape it, and the last two hand a materialized frame to openpyxl and sqlite3. They still collect, but with the streaming engine so the collect itself is bounded.

• table: Run column and row checks one at a time (fc463ea)

Every check builds its own LazyFrame over the source, and a LazyFrame caches nothing, so N checks means N independent scans of the same file. polars materializes a CSV in full however the query is collected, so peak memory was the file size times the number of checks in flight.

On a 1 GB file that was 12GB at one check per core, against 1.9GB serially. Time goes the other way, 5.0s to 7.0s, but the concurrency sweep flattened above two workers while memory kept climbing, so the parallelism was buying little of what it cost.

The two limits are separate settings: row-key checks are usually one or two tasks, columns are the many. Callers can still override with the concurrency option.

Refactoring

• Complete the init.py package chain (73e2943)

metadata carried exactly one init.py and relied on PEP 420 namespace packages; dataset was missing the markers under actions/. The other four packages already have them nearly everywhere.

pytest resolves a test module by walking up the init.py chain from the file, and stops at the first directory without one. With a gap, a spec one level below its subject resolves as a top-level module and a relative import of that subject reaches beyond the top-level package.

Prerequisite for moving specs into _test folders.

• Move unit tests into _test folders (683e171)

Adopt the convention fairspec-typescript took from mycarro-next: unit tests and their fixtures move into a collocated _test/ directory, so the module surface reads without spec files interleaved.

TypeScript marks these folders with a leading -, which is illegal in a JS identifier and so can never be mistaken for an importable module. Python needs the marker to stay a valid identifier, because _shared is imported by its siblings, so it uses a single leading underscore instead.

Discovery is unaffected — pytest matches on the *_spec.py suffix, and both the fixture paths and the vcr_cassette_dir fixture derive from the test file's own directory, which now carries its fixtures with it.

dataset's conftest.py moves down from the package root to the one folder whose spec consumes it, taking the two cassettes with it.

plugins/xlsx/actions/table/test.py was production-located but read only by two specs; it moves into _test/ as test_data.py, named after its exports.

• metadata: Drop the dead format="" literal on the number base (f2f91a8)

BaseNumberColumnProperty was the only plain column base declaring a format, and it declared the empty-string literal that every other plain model had already migrated away from. NumberColumnProperty overrides it with Literal[None], so the field was shadowed and unreachable — the public surface already rejected format="".

Removing it makes the number base match the string, integer and object bases, which declare no format at all.

• table: Move action helpers into _shared folders (f332be3)

Both helpers.py files sit inside action folders where every other file is a single action. _shared marks production code shared by those siblings without being an action itself, and can hold its own _test.

actions/table/_shared/helpers.py is used by normalize.py and denormalize.py. actions/column/_shared/helpers.py has no production caller here — the column types each implement their own inspection loop, unlike the TypeScript tree where seven of them import the equivalent helper.

• table: Name the polars engine setting QUERY_ENGINE (e508881)

INSPECT_ENGINE was named for the one caller it started with, but the setting is already read by the foreign-key join and the dialect probe, and polars accepts engine= on fifteen methods rather than collect alone — including sink_parquet and sink_ipc, which the save plugins already call. QUERY_ENGINE names what it selects: how a query plan is executed, for collects and sinks alike.

Testing

• fairspec: Guard docs against stale scripts and paths (4a035d0)

Fail the build when AGENTS.md, README.md or CONTRIBUTING.md names a taskipy task, a pnpm script or a repo path that does not exist, or when .claude/CLAUDE.md stops resolving to AGENTS.md.

The docs had drifted to a test-file convention the repo never used; this is what stops that happening again.

• table: Record the remote csv fixture request (c8f400a)

The spec fetched the fixture over the network on every run, so it broke whenever the fixture moved — it had been failing with a 404 until the URL was corrected. It now replays a cassette like the prefetch spec.

The vcr_cassette_dir fixture moves to a root conftest: the cassette directory rule is repo-wide, and a second copy would have been the only alternative.

Detailed Changes: v0.1.7...v0.2.0