What does tablespec define?

tablespec defines table contracts in UMF: types, validation criteria, relationships, aliases, keys, and derivations. The ingested bronze contract is source-semantic; silver and gold specs declare business decisions. All compile to runtime artifacts.

Does tablespec stop at bronze?

No. Ingested bronze is the layer tablespec covers most completely. Cross-source conformance, survivorship, and derived gold tables are declared in the same UMF format and compiled by the same step, not hidden inside ingestion.

From raw records to gold tables

TABLESPEC / SOURCE CONTRACT

One UMF. Every runtime artifact.

Define table types, quality rules, and transformations once. tablespec compiles those contracts into SQL, dbt, Lakeflow, schemas, and Great Expectations suites, plus Excel workbooks and guidebooks for review. Keep source meaning in source-semantic ingested bronze; declare business choices in silver and gold. Review the runtime artifacts before they run.

Start with a UMF View the worked example

compile path review surface

source flat files, dumps, APIs records as delivered by another system

raw source records transport shape kept for audit and replay

ingested bronze typed, validated, keyed source meaning captured in Delta-compatible tables

silver conform, resolve, enrich business choices across one or more sources

UMFtables/claims/table.yaml

SQLclaims.ingest.sql

dbtmodels/claims.sql

Lakeflowpipeline.yml

GXsuite.json

Guidebookindex.html

types declared validation generated keys recorded relationships defined artifacts committed

SHARED FORMAT / PIPELINE COMPILER

UMF carries the metadata. tablespec compiles the pipeline.

Universal Metadata Format (UMF) is now a separate project. Its umf-core Python package supplies shared document models, schemas, serialization, and validation machinery. tablespec owns the tablespec.pipeline extension: native table types, source declarations, contextual nullability, keys, relationships, and pipeline policies.

Existing split-directory specs remain supported. Shared documents preserve unknown vocabularies; compilation requires an explicit tablespec execution mapping. Read the format and compatibility boundaries or explore the UMF project.

EVALUATE

Define once. Generate. Review.

Define the source table once

Capture names, source types, nullability, keys, relationships, aliases, and provenance in Universal Metadata Format.

Compile committed artifacts

Generate the SQL DDL, ingest transform, dbt model, Lakeflow pipeline, JSON Schema, Great Expectations suite, Excel workbook, and guidebook from that spec.

Generate checks for each layer

Run checks against raw source records, typed ingested tables, and the derived tables built from them, all generated from the same specs.

WORKED EXAMPLE

Medical claims from UMF to reviewable artifacts

Start with a split-format UMF directory for `medical_claims`: one table-level YAML file plus one YAML file per column. tablespec validates that source-table contract, then emits raw-to-ingested SQL, a dbt project, a Great Expectations suite, and a guidebook from the same spec.

Open the example

terminal deterministic output

tablespec validate tables/
tablespec generate tables/medical_claims -f ingest > claims.ingest.sql
tablespec emit tables/ out/dbt --backend dbt --dialect databricks
tablespec validation-sync tables/medical_claims --out gx/
tablespec guidebook tables/ -o site/guidebook

DECIDE

Bronze records source meaning. Silver records business choices.

Both are UMF contracts compiled by tablespec. The layer names say what each contract may decide.

Question Ingested bronze Silver

Renames columns? No. Preserve source names. Yes, when conformance requires it.

Resolves duplicates across sources? No. Yes.

Defines business survivorship? No. Yes.

Feeds downstream systems? Yes. Typed, validated, keyed Delta-compatible artifacts. Consumes bronze and applies governed business decisions.

OPERATE

Compile the UMF contract. Review the diff.

Install tablespec, write one UMF spec, generate runtime artifacts, then review the diff before anything runs.

Get started Read raw, ingested, and silver