Founding release · v0.2.0 · Python 3.12
Dataset Preflight
Catch dataset problems locally. Put the evidence in CI.
One command inventories a directory, surfaces duplicate, declaration, and secret-risk findings, applies your thresholds, and can emit SARIF and JUnit for the CI viewers your team already uses.
Built for teams that assemble, receive, or move folders of training and evaluation data. It is a file-level preflight gate, not a spreadsheet profiler or model-quality score.
- Scan
- Review
- Decide
- Enforce
One organization. No subscription. Any 0.x updates that are released are included. Checkout and delivery are handled by Polar.
A narrow operational gate
Use it when a dataset becomes expensive to misunderstand
Dataset Preflight records bounded, reviewable facts before a pipeline run. It does not score the dataset or replace the person responsible for accepting it.
Before a dataset leaves your machine
Create a review packet before an upload, training run, evaluation, or handoff changes the source directory.
When a handoff needs evidence
Give another team a deterministic inventory, duplicate findings, declaration coverage, and checksums instead of a verbal clean-data claim.
When CI needs a real gate
Turn selected limits into stable exit codes, SARIF findings, and JUnit cases that fit existing review workflows.
Why buy a small local tool?
Skip maintaining the unglamorous evidence layer
A quick script can list files. This release also keeps JSON, HTML, SARIF, JUnit, checksums, exit codes, privacy exclusions, and deterministic ordering aligned behind one command.
Audit the public packet first. Buy only if that exact scope saves your team more than $39 of work.
Inspect before purchase
A real five-artifact packet from the released v0.2.0 wheel
These are the real, unedited outputs generated from a tiny synthetic six-file, 149-byte fixture. They demonstrate one exact duplicate, one narrowly normalized duplicate, one filename-only secret-risk signal, four intentionally uncovered files, no scan issues, and a passing explicit policy.
The manifest records SHA-256 checksums for the JSON, HTML, SARIF, and JUnit reports.
The sample shows the output contract and review experience. It is not customer data or a performance benchmark, and it does not prove that another dataset is safe, lawful, suitable for training, or able to pass the same policy.
Local by construction
Review the data without sending it anywhere
Dataset Preflight runs on your machine. The runtime has no dependencies, makes no network requests, performs no telemetry or update checks, and reports paths relative to the scanned root.
The dataset stays local.
Files are read to build the inventory and findings. Outputs are written only to the new local directory you choose. The report excludes absolute source paths, hostnames, usernames, timestamps, network state, and output paths.
yy-preflight scan C:\path\to\dataset --output C:\path\to\preflight-reportNew in v0.2.0
Review locally. Surface the same findings in CI.
Every run writes JSON, HTML, and a manifest. Add --evidence sarif,junit when your CI system also needs SARIF and JUnit outputs.
report.json
Canonical, machine-readable inventory and findings with sorted keys and stable output bytes.
report.html
A standalone offline review generated from the same report object, with inline CSS and no JavaScript or external resources.
report.sarif
An opt-in SARIF 2.1.0 export for findings and policy failures, suitable for GitHub Code Scanning and other SARIF viewers.
junit.xml
An opt-in JUnit XML export with fixed observability checks and configured policy cases for test-report tooling.
manifest.sha256
SHA-256 checksums for every generated report artifact, ordered by filename.
Machine-readable without inventing evidence.
SARIF carries scan issues, secret-risk signals, uncovered files, missing declaration evidence, and failed thresholds. JUnit failures align with configured policy failures on completed scans; input, configuration, filesystem, and internal errors still exit without a partial packet. Neither export includes matched secret values, absolute paths, real or wall-clock timestamps, usernames, or invented line and column numbers. JUnit uses a fixed Unix-epoch timestamp to keep its bytes deterministic.
SARIF finding severity is separate from the configured policy. This sample includes error-level uncovered-file findings while its explicit policy still passes.
Measured scope
Concrete findings before a pipeline run
The report states observable facts and bounded signals. Optional declarations and policies let your team add its own evidence and acceptance limits without changing scan findings.
Deterministic inventory
Records normalized relative paths, fixed local file types, byte sizes, and SHA-256 content hashes in a stable order.
Exact duplicates
Groups files with the same raw SHA-256 and reports excess-file counts you can threshold in CI.
Narrowly normalized duplicates
Finds strict UTF-8 text made identical only by a documented normalization of Unicode, newlines, trailing spaces and tabs, and outer blank lines.
Provenance coverage
Checks optional customer-authored declarations against file scopes and reports uncovered paths, conflicts, and whether named evidence files exist.
Secret-risk signals
Surfaces conservative filename and eligible-text signal codes without copying matched text, secret values, offsets, or line numbers into the report.
CI thresholds
Applies selected count limits as stable exit codes. On a completed scan, failed thresholds also appear in SARIF and as JUnit failures aligned with policy exit code 1.
CI-native evidence
Optionally emits deterministic SARIF 2.1.0 and JUnit XML so selected findings and configured policy results can enter existing code-scanning and test-report workflows.
Sensitive outputs
Keep the reports with the dataset
Reports intentionally contain filenames and content hashes. Treat them as sensitive operational artifacts even though secret values and matched text are never included.
Strict boundary
What this release does not claim
A preflight packet supports a human decision; it does not replace one. Secret-risk signals are conservative heuristics with false positives and false negatives.
- Not a security scanner, security audit, certification, or proof that a dataset is safe
- Not legal advice, license clearance, or a determination that data is lawful to use
- Not a determination that data is suitable or ready for model training
- Not semantic, fuzzy, or embedding-based near-duplicate detection
- Does not validate the meaning or legal sufficiency of provenance declarations
Founding release
$39 once
Licensed for internal use by one organization, including all 0.x updates that are released.
Support and future features are not promised by this founding release.