Training data with a provenance story you can defend.
Every AI team is one licensing dispute away from a data problem. Public records are the corpus with no publisher on the other side: government filings, registries, permits, court dockets — published to be public. Recordpipe builds them into training-ready datasets: structured, deduplicated, delivered as JSONL or Parquet, with per-record source documentation your counsel can actually read.
Corpus builds to your spec
Record types, jurisdictions, time ranges, and schema defined in scoping; delivered as one clean, versioned corpus.
Provenance per record
Every row carries its public-source citation and capture date — the documentation diligence asks for.
Refresh for temporal splits
Ongoing collection on cadence gives you clean train/eval time splits and continuously fresh RAG indexes, not a frozen snapshot.
Structured for machines
JSONL, Parquet, or an API; free-text fields optionally classified and normalized by our AI analysis layer before delivery.
LLM training datasets built from a public records corpus
Most LLM training datasets start from a crawl and work backwards, trying to reconstruct where the text came from after the fact. A public records corpus inverts that. Government filings, court dockets, corporate registrations, permits, and licensing records are published by public bodies for public consumption — the origin is the starting point, not a forensic exercise.
Recordpipe builds these records into training-ready corpora to your spec. Scoping defines the record types, jurisdictions, time ranges, and target schema. Delivery is JSONL or Parquet, sharded the way your ingestion pipeline expects, with a corpus manifest listing every shard, its checksum, and its version. Free text arrives as published; structured fields arrive normalized to one schema across jurisdictions that publish in wildly different formats. The corpus you get is the one your data-acquisition team specified — not a dump you spend a quarter cleaning.
Training data provenance, documented per record
Provenance is the field diligence actually reads. Every record in a Recordpipe corpus carries its own source citation — the publishing body and jurisdiction — plus its capture date and a note on the public-source basis of the collection. When a governance reviewer, an acquirer, or an enterprise customer asks where a document came from, the answer is in the row, not in an engineer's memory.
That per-record documentation is the honest version of what the market calls rights-clean training data. We do not certify legal outcomes; no vendor honestly can. What we deliver is a corpus whose construction is defensible: public sources only, nothing login-walled, no scraped private platforms, no breached datasets, and a citation trail your counsel can evaluate against your application. Data licensing for AI is a legal determination that belongs to your lawyers; our job is handing them a dataset they can actually assess instead of a shrug.
RAG datasets that stay current, not frozen snapshots
Retrieval systems fail differently than pretraining runs: a stale index answers confidently from a record that changed last month. RAG datasets from public records need a refresh mechanism, and Recordpipe contracts one — collection on a fixed cadence, down to daily, delivered as deltas by webhook or file drop so your retrieval layer re-embeds what changed instead of rebuilding the world.
Each delivery carries a version, and each record a dedupe hash, so your pipeline can distinguish a new filing, an amended one, and a duplicate published through a second channel. For agent platforms the same feed doubles as a grounding layer: an agent citing a permit or docket entry can surface the source citation and capture date alongside the answer — the difference between a tool users trust and one they double-check. The cadence is a contract term, not a subscription you are pushed into.
Temporal splits, deduplication, and corpus hygiene
Contamination and leakage are corpus-construction problems, and public records make them tractable. Every record is stamped with the event date the source published and the date we captured it, so train/eval splits cut on real time boundaries — models evaluated on filings from after their training window, not on paraphrases of what they already saw. Frozen snapshots are available where a reproducible evaluation set matters more than freshness.
Deduplication runs before delivery: records are hashed on normalized content, near-duplicates across jurisdictions and channels are collapsed, and the surviving record keeps references to its variants. Records that fail quality rules are flagged in the manifest, not dropped — a documented gap is worth more to a serious data team than an artificially smooth corpus. Where free text needs structure, our AI analysis layer classifies and normalizes it before delivery, with the method documented so your team can trust the derived fields or recompute them.
AI training data pricing that fits how labs buy
Data acquisition teams evaluate before they commit, and the pricing is built for that. Scoping is $500, credited against the contract, and returns feasibility, a fixed quote, and a sample of the actual corpus — real records, real manifest, real provenance fields — within 5 business days. Contracts start at $5,000 for a scoped vertical corpus and run to $3 million for multi-year programs with contracted refresh. Usage rights are set in the contract in plain terms your counsel negotiates directly.
The infrastructure underneath is not built to order: the same pipelines already process over one million public records nightly for our own products. What is built to order is the corpus — your record types, jurisdictions, schema, and cadence — which is why the sample comes first.
What a delivery looks like: the corpus manifest
| Field | Description |
|---|---|
| record_id | Stable identifier per record, consistent across refresh deliveries and versions |
| source_citation | Publishing body and public-source citation for the record — the provenance field diligence reads |
| capture_date | When Recordpipe collected the record; paired with event_date to cut clean temporal splits |
| jurisdiction | State, county, or registry that published the record, normalized to one scheme |
| record_type | Category per the taxonomy agreed at scoping — filing, docket entry, registration, permit, license |
| text | Free-text content as published — the substrate for pretraining, fine-tuning, and embedding |
| structured | Normalized structured fields (parties, dates, statuses) per the contracted schema |
| dedupe_hash | Content hash used to collapse duplicates across sources; variant references preserved |
| version | Corpus and record version, incremented on refresh so deltas apply deterministically |
| license_note | Per-record note on the public-source basis of collection, for your counsel's licensing review |
In the field.
Data acquisition at a foundation lab
A foundation lab's data-acquisition team can commission a multi-year corpus of public legal and administrative text — deduplicated, sharded for ingestion, with per-record source citations — and hand governance a paper trail instead of a crawl of unknown origin, testing tokenization and quality on the scoping sample before signing.
Fine-tuning corpus for a legal-AI startup
A legal-AI startup can specify dockets, filings, and case-record text from its target jurisdictions as one versioned corpus, with capture dates supporting honest train/eval splits — domain depth its general-purpose base model never had, on a contract its enterprise customers' vendor reviews can read.
Property intelligence for a proptech-AI startup
A proptech-AI team can train and ground its models on permits, ownership records, and transaction filings across its coverage area, refreshed on cadence — so valuations and risk calls reflect current records, and every model output can trace back to a cited public filing.
Grounding layer for a RAG or agent platform
An agent-platform team can contract a daily delta feed of the record types its users ask about, keyed by dedupe hash and version, so the retrieval index stays current without rebuilds — and every agent answer can show the source citation behind it.
Built for labs and vertical-AI teams
Foundation labs need breadth; a legal-AI or proptech-AI startup needs one deep, current vertical corpus. Both run on the same pipelines that already process over one million public records nightly for our own products. Fixed-price from $5,000 for a scoped corpus to $3 million programs; scoping is $500, credited, with a sample in 5 business days.
Is this data licensed for AI training?
Can you keep a RAG index current?
What formats do you deliver?
How do you handle deduplication?
Can we get clean temporal splits for evaluation?
What won't you collect?
Can you build a small pilot corpus first?
Do you label or annotate the data?
More solutions: Sales & lead intelligence · Tenant screening · Property management · Real estate & title · HR & onboarding · Legal & PI firms · RPA at scale · Custom Data API