data-lake-platform

Data Lake Platform

Build and operate production data lakes and lakehouses: ingest, transform, store in open formats, and serve analytics reliably.

Quick Reference

Quick-Start Decision Table

Format State as of July 2026

Key July 2026 facts (verify before quoting further out):

• Iceberg v3 is GA on Snowflake (May 7 2026) and AWS (S3 Tables, Glue, EMR — Nov 2025). On Databricks it is Public Preview (announced Apr 9 2026, Databricks Runtime 18.0+ with Unity Catalog), not GA — do not tell a Databricks-only user v3 is production-ready there without checking current Databricks release notes.
• Trino's Iceberg connector treats format-version-3 support as experimental in its own docs: base read/write works, but row-level updates, deletes, and OPTIMIZE on v3 tables are unsupported. Starburst's Trino-based product has broader v3 support (deletion vectors, variant type) than open-source Trino — don't conflate the two when a client asks about "Trino."
• Delta Lake shipped 4.1 (Mar 2026, catalog-managed tables GA), 4.2 (Apr 2026, native Flink connector, geospatial + Variant GA in Kernel), and 4.3 (Jun 2026, Unity Catalog Delta APIs route every table operation, streaming/CDF on catalog-managed tables). Treat "4.1" as a floor, not the current version.
• Apache Polaris graduated to Apache Top-Level Project on Feb 19 2026 (~100 contributors, 6 releases, 2800+ PRs at graduation) and is now on a monthly release train with federation and credential vending in production.
• DuckLake v1.0 GA Apr 13 2026: SQL-based catalog (SQLite/PostgreSQL/DuckDB), Iceberg-compatible deletion vectors, clients for Spark/Trino/DataFusion/Pandas. DuckLake 1.1 is expected ~Sept 2026 — re-check compatibility notes before committing to a long-lived DuckLake deployment.
• Amazon S3 Tables: Iceberg v3 deletion vectors and row lineage supported (GA, rolled out from Nov 2025); simplified IAM permissions (Mar 2026); GovCloud GA (Feb 2026); two new regions (May 2026).
• Apache Gravitino: TLP graduated Jun 2025; 1.2.0 (Mar 13 2026) added Table Maintenance Service, ClickHouse catalog, end-to-end UDF management; 1.2.1 (May 11 2026) is a stability/correctness patch — prefer it over 1.2.0 for new deployments.
• Apache Hudi 1.1 is an active development branch (pluggable table-format framework, indexing rework) — not recommended for production as of July 2026; production Hudi deployments should stay on the 1.0.x line.

Any number here (release dates, contributor counts, benchmark percentages) can drift within weeks in this space — re-verify against the primary source in data/sources.json before repeating it in a client-facing recommendation, especially GA-vs-preview status, which vendors routinely blur in marketing copy.

Decision Tree

Choosing a lakehouse path:
    ├─ Databricks is the primary platform?
    │   └─ Delta 4.x + Unity Catalog; add UniForm only if external readers matter
    │
    ├─ Need open multi-engine access across Trino/Spark/DuckDB?
    │   └─ Iceberg v2/v3 + Polaris / Glue REST / Nessie / Open Catalog
    │       Note: default to v2 if Trino must write; v2 avoids row-update/delete/OPTIMIZE
    │       gaps that v3 still has on open-source Trino as of July 2026
    │
    ├─ Need heavy CDC, mutable tables, or streaming-first semantics?
    │   ├─ Flink-native stack -> Paimon first, compare with Hudi
    │   └─ Spark-heavy stack -> Hudi first (CoW for read-heavy, MoR for write-heavy)
    │
    ├─ Single-engine embedded or analyst workstation?
    │   └─ DuckLake v1.0 (PostgreSQL catalog for multi-instance) or DuckDB + Parquet
    │
    ├─ Need low-latency dashboards or embedded analytics?
    │   ├─ High concurrency BI -> Add ClickHouse / StarRocks / Doris
    │   └─ Local, notebook, CI -> DuckDB + Parquet/Iceberg
    │
    └─ Heterogeneous multi-format estate (Hive + Kafka + RDBMS)?
        └─ Apache Gravitino as federated "catalog of catalogs"

Catalog Landscape (July 2026)

Workflow Checklist

1. Architecture and Ingestion

• Choose architecture pattern: references/architecture-patterns.md — medallion, mesh, lambda, kappa, lakehouse
• Define CDC or batch ingestion path: references/ingestion-patterns.md — dlt, Airbyte, Debezium, Flink CDC
• Configure streaming semantics if needed: references/streaming-patterns.md — Kafka, Flink, Spark Structured Streaming
• Define data mesh domain ownership if org is distributed: references/data-mesh-patterns.md

2. Storage and Catalog

• Select table format and version: references/storage-formats.md
  • Iceberg: confirm v2 or v3 — check all engine support before enabling v3
  • Delta: use 4.x catalog-managed tables for new workloads
  • Hudi/Paimon: validate engine stack before committing
• Select catalog: references/governance-catalog.md
  • Polaris for open Iceberg; Glue REST for AWS; Nessie for branching; Unity for Databricks
  • Consider Gravitino for heterogeneous multi-catalog federation
• Set compaction, snapshot retention, and orphan-file cleanup schedule
• Test interoperability with all intended engines before declaring multi-engine support

3. Transformation

• Choose transformation tool: references/transformation-patterns.md — dbt, SQLMesh, Spark
• Verify dbt or SQLMesh supports target catalog and table format
• Set up incremental models with idempotency guarantees
• Add orchestration layer: references/orchestration-patterns.md

4. Query and Serving

• Choose lake query engine: references/query-engine-patterns.md
• Add serving layer only when concurrency/latency requirements are proven
• Configure BI and visualization layer: references/bi-visualization-patterns.md

5. Quality, Security, and Ops

• Define data quality contracts and checks: references/data-quality-patterns.md
• Enforce access control per engine: references/security-access-patterns.md
• Build operational runbooks: references/operational-playbook.md
• Enforce file size targets, compaction policy, and cost guardrails: references/cost-optimization.md

Quick Commands

# Generate DDL for partitioned Iceberg table (REST catalog, format v2):
python scripts/scaffold_iceberg_table.py \
  --catalog rest \
  --name analytics.events \
  --columns "event_id BIGINT, user_id BIGINT, event_type STRING, ts TIMESTAMP" \
  --partition ts_month,event_type \
  --format-version 2 \
  --target-file-size-mb 256

# Inspect S3-backed Iceberg table layout:
./scripts/inspect_iceberg_metadata.sh \
  --location s3://my-bucket/warehouse/analytics/events \
  --backend s3

# Iceberg maintenance (run in Spark or Trino):
CALL catalog.system.expire_snapshots('db.events', TIMESTAMP '2026-01-01');
CALL catalog.system.remove_orphan_files('db.events');
CALL catalog.system.rewrite_data_files('db.events');
CALL catalog.system.rewrite_manifests('db.events');

# DuckLake: create catalog and attach (DuckDB v1.5.2+):
INSTALL ducklake; LOAD ducklake;
ATTACH 'ducklake:postgres:dbname=catalog host=localhost' AS lake;

# Delta: check table version and history:
DESCRIBE HISTORY delta.`s3://bucket/path/to/table`

Reference Architectures

• Open multi-engine: object storage + Iceberg v2/v3 + Polaris or Glue REST + Trino/Spark + DataHub/OpenMetadata + OpenLineage
• Databricks-centered: Delta 4.x + Unity Catalog; add external interoperability only where there is a real consumer requirement
• Streaming-first mutable: Kafka/Flink CDC + Hudi or Paimon + Trino/Spark readers + strict replay and retention rules
• Serving-heavy analytics: Iceberg/Hudi/Delta upstream + ClickHouse/StarRocks/Doris downstream for fast dashboards
• Small-team embedded: DuckLake v1.0 + PostgreSQL catalog + DuckDB compute; upgrade path to Iceberg when multi-engine needed

Do / Avoid

Do

• Define data contracts, owners, and retention rules before first write.
• Make every pipeline idempotent, replayable, and safe to backfill.
• Keep catalog, lineage, and access-control choices explicit.
• Test interoperability on real engines before committing to multi-engine promises.
• Confirm the exact Iceberg v3 feature and engine support matrix (GA vs preview vs experimental — they are not the same) before enabling v3 in production, and re-check it close to launch since GA dates and preview scopes move monthly.
• Use a serving layer only when workloads prove the need.
• Distinguish "the spec supports X" from "our vendor's build of the engine supports X" — Trino open source and Starburst's Trino-based product diverge on Iceberg v3 coverage, and Databricks' own docs may say "Public Preview" even when a partner blog calls it "available."

Avoid

• Treating Delta, Iceberg, Hudi, and Paimon as interchangeable.
• Enabling Iceberg v3 writes on Trino in production — as of July 2026 Trino's own docs mark v3 as experimental with row-level updates, deletes, and OPTIMIZE unsupported on v3 tables.
• Repeating a vendor's "v3 support" headline without checking whether it means GA, public preview, or read-only — Databricks Iceberg v3 is Public Preview, not GA, even though it is easy to find blog copy that reads as if it shipped.
• Hiding governance inside a single vendor-specific default.
• Shipping CDC without delete handling, retention policy, and replay drills.
• Recommending managed services or format/spec-version claims without current-source verification — this space re-ships GA announcements monthly, and a fact that was accurate in the last training pass is a coin flip a month later.

Known Traps

• Choosing a table format for vendor fit before validating engine support, catalog behavior, delete semantics, and maintenance tooling across the actual estate.
• Treating object storage plus an open table format as a complete platform while leaving compaction, snapshot retention, metadata cleanup, and orphan-file controls unmanaged.
• Mixing CDC upserts, streaming ingestion, and batch rewrites into the same tables without explicit idempotency, late-arrival, and rollback rules.
• Assuming all engines interpret schema evolution, partition pruning, delete files, and time travel consistently across formats.
• Copying warehouse-style small-table habits into the lake and creating severe small-file, manifest, and metadata amplification at scale.
• Assuming DuckLake is interchangeable with Iceberg REST for multi-engine workloads — DuckLake is SQL-catalog-native and not designed for concurrent Spark/Flink write access.
• Recommending a coordinated-commits or catalog-managed-tables migration (Delta 4.x, Iceberg REST) as a drop-in change. It changes who owns the commit path and can require a client/connector version bump across every reader and writer — sequence it as a migration with a rollback plan, not a config flag.
• Picking the format with the best headline feature (row lineage, deletion vectors, Variant type) without asking whether the catalog and compute stack the client already has can actually exercise that feature today. A GA spec feature is not GA for that client until their engine, catalog, and client library all agree on it.
• Treating "Apache project" or "Top-Level Project" status as a maturity signal by itself. TLP graduation (Polaris, Gravitino) is a governance and community milestone, not a production-readiness certification — check adoption, release cadence, and operator experience separately.
• Letting a proof-of-concept's convenient single-engine choice (e.g., DuckLake, embedded DuckDB) silently become the production architecture once a second team needs concurrent writes or a different query engine. Name the upgrade trigger and the target format up front.

Navigation

References (load on demand)

Templates

• Core blueprints: assets/cross-platform/template-medallion-architecture.md, assets/cross-platform/template-data-pipeline.md, assets/cross-platform/template-migration-checklist.md, assets/cross-platform/template-partitioning-strategy.md
• Ingestion: assets/cross-platform/template-ingestion-governance-checklist.md, assets/cross-platform/template-incremental-loading.md, assets/ingestion/dlt/template-dlt-rest-api.md, assets/ingestion/airbyte/template-airbyte-connection.md
• Storage and quality: assets/cross-platform/template-schema-evolution.md, assets/cross-platform/template-data-quality.md, assets/cross-platform/template-data-quality-governance.md, assets/cross-platform/template-data-quality-backfill-runbook.md
• Open formats: assets/storage/iceberg/template-iceberg-table.md, assets/storage/iceberg/template-iceberg-maintenance.md, assets/storage/delta/template-delta-table.md, assets/storage/hudi/template-hudi-table.md
• Transforms: assets/transformation/dbt/template-dbt-project.md, assets/transformation/sqlmesh/template-sqlmesh-project.md, assets/transformation/sqlmesh/template-sqlmesh-incremental.md, assets/transformation/sqlmesh/template-sqlmesh-testing.md, assets/transformation/sqlmesh/template-sqlmesh-layering-and-access.md
• Engines and BI: assets/query-engines/duckdb/template-duckdb-analytics.md, assets/query-engines/clickhouse/template-clickhouse-optimization.md, assets/query-engines/clickhouse/template-clickhouse-materialized-views.md, assets/visualization/metabase/dashboard-request.md

Related Skills

• ai-mlops - MLOps, deployment, platform operations
• ai-ml-data-science - Analytics and feature engineering workflows
• data-sql-optimization - OLTP tuning and relational operations
• ops-devops-platform - Infra, Kubernetes, observability, and runbooks

Tool/Platform Recommendation Protocol

1. Read data/sources.json and start from primary docs.
2. Verify current platform behavior for catalogs, connectors, security, and interoperability.
3. Distinguish open-standard behavior from vendor-specific behavior.
4. Report defaults, tradeoffs, and what is still vendor-bound.
5. If browsing is unavailable, state the limitation and mark recommendations as unverified.

Fact-Checking

• This skill's dated facts (spec-version GA status, release numbers, contributor/PR counts, benchmark percentages) reflect a snapshot and go stale within weeks — the lakehouse catalog and format space ships GA and preview announcements monthly.
• Before repeating a GA/preview/experimental status claim, verify it against a trust_tier: primary source in data/sources.json (project release notes, vendor release notes) rather than a blog or aggregator, which often blur preview and GA language.
• Explicitly separate "the format spec supports X" from "this specific engine/vendor build supports X" — these routinely diverge (see Trino vs. Starburst on Iceberg v3; Databricks Public Preview vs. Snowflake/AWS GA on Iceberg v3).
• If web access is unavailable, say so explicitly and mark version- or GA-status-dependent recommendations as unverified rather than presenting a frozen snapshot as current.

Learnings Loop

Before applying this skill on a non-trivial task, read learnings.consolidated.md in this directory (and learnings.md if present).

After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to learnings.md via agents-skills-feedback-loop/scripts/append_learning.py. Do not modify SKILL.md itself.