INGEST — Autonomous Data Ingestion Agent

Every ingestion tool on the market classifies sensitive data after it lands in the warehouse. By then the SSNs are already in a Delta table, in the logs, and in three downstream copies. INGEST decides the masking policy in the plan step, then proves in the verify step that every flagged column was actually masked. Compliance is not a scan you run later. It is a gate the data passes through.

The agent never depends on the LLM to finish. Claude refines the plan and explains failures. A deterministic engine validates every suggestion and does the work. Cut the network, set --no-llm, and it still runs end to end. That is why it fits AIRLOCK, GovCloud, and any enclave your auditors care about.

— The loop

Six steps, zero hand-written pipeline.

01
DiscoverSniffs format, encoding, and delimiter. Profiles every column: inferred type, null rate, cardinality, primary-key candidates, date columns. Handles messy real-world files: mixed date formats, currency strings, leading-zero identifiers, duplicate rows.
02
ClassifyFourteen PII / PHI detectors run on names and values together: SSN, EIN, credit card (Luhn-checked), email, phone, date of birth, MRN and member IDs, NPI, ICD-10 and CPT codes, person names, addresses, IPs, account numbers, embedded secrets. Each hit carries its HIPAA, PCI, GLBA, or NIST citation.
03
PlanProposes the target table, business-meaningful column names, canonical types, primary key, dedupe strategy, partition key, and a per-column masking policy: hash, redact, year-only, ZIP-3. Claude refines; the validator rejects anything that drops a column, invents a type, or weakens masking.
04
GenerateEmits the Airflow DAG, Databricks Delta DDL with PII column comments and table properties, dbt schema.yml with tests, and a standalone loader. Pipeline-as-code your team can read, review, and commit.
05
Run and self-healCoerces types, masks per policy, dedupes, loads to Parquet, SQLite, or Databricks. Rows that fail coercion are quarantined with a reason, never silently dropped. A column rejecting too many rows is a wrong type guess, so the agent widens it and retries. Bounded, logged, explainable.
06
VerifyReconciles row counts, proves clean + quarantined + duplicates equals source, checks key uniqueness and NOT NULL constraints, and reads the target back to prove every PII column is actually masked. PASS or FAIL, with an HTML evidence report and a JSON manifest an auditor can diff.

— What ships

One agent, any source, any target.

Sources

Files, databases, APIs, folders

CSV, TSV, JSON, JSONL, Parquet, Excel, SQLite, any SQLAlchemy database URL, REST JSON endpoints with bearer auth, and whole folders of mixed files stamped with their origin.

Format and encoding auto-detected

Targets

Parquet, SQLite, Databricks Delta

Hive-partitioned Parquet for lakes, SQLite for local proof, Databricks via the SQL Statement Execution API with generated Delta DDL. Snowflake and Fabric targets follow the same adapter contract.

Zero cloud footprint by default

Compliance gate

PII / PHI masked before landing

Masking policy is decided in the plan and proven in verify. Column comments and table properties carry the classification into Unity Catalog so downstream tools inherit it.

HIPAA · PCI · GLBA · NIST 800-53

Migration ready

Built for application migrations

Every legacy application is a set of tables and feeds that must move with proof. INGEST profiles the source, lands it governed, and hands you the reconciliation report the cutover meeting needs. Repeat per app, per environment.

ERP · EHR · claims · legacy databases

Airlock mode

Runs with the network cut

Deterministic engine completes the whole loop offline. LLM refinement is an optional enhancement, never a dependency. Fits DMZ, IL4 / IL5, and GovCloud enclaves.

--no-llm · no outbound calls

Evidence

Audit trail on every decision

The manifest records what the agent saw, what it decided, what Claude suggested, what the validator kept, what was quarantined and why. Learning state improves detection across runs.

manifest.json · HTML report · rejects parquet

— Proof

Synthetic claims file, one command.

$ python agents/ingest.py --source claims_2026_08.csv --target parquet

[INGEST] discover <- claims_2026_08.csv

[INGEST] 17 columns, 5120 rows, PII/PHI: Member ID, MemberFirstName, MemberLastName, SSN, DOB, Phone, Email, Zip, Rendering NPI, Dx Code

[INGEST] plan -> table=claims pk=[claim_id] partition=service_date load_mode=merge planner=claude+validated

[INGEST] generate -> 4 artifacts (Delta DDL, Airflow DAG, dbt schema, loader)

[INGEST] run -> 5120 source -> 4977 loaded, 25 quarantined, 118 duplicates removed

[INGEST] verify -> PASS (25/25) report: ingest-report.html

Mixed date formats normalized, currency strings parsed, rows with text in a numeric column quarantined with reasons, 118 exact duplicates removed on the detected key, ten PII / PHI columns masked and proven masked on read-back. Synthetic data, no real people.

Point it at the data. It does the rest.