Data Engineering Agent Skills

🌐 Project Website | 📦 VS Code Marketplace | 🔌 JetBrains Marketplace | ⬇️ GitHub Releases

Production-grade data engineering skills for AI agents.

The open skill registry and execution toolkit for data engineering agents.

This repository packages repeatable workflows, quality gates, hooks, installer surfaces, and example packs (runnable scaffolds plus architecture blueprints) so agents can build data systems with the same discipline used by strong data engineering teams.

The goal is not to give agents generic prompts. The goal is to give them operating procedures for defining, planning, implementing, validating, replaying, and shipping reliable data products.

Agent Skills Registry Compatibility

This repository is structured to work with open Agent Skills registries:

• every capability lives in a directory containing a SKILL.md
• every SKILL.md starts with YAML frontmatter including at least name and description
• descriptions are written for progressive disclosure so agents can decide when to load the full skill
• supporting materials can live in references/, templates/, examples/, hooks/, and scripts/

Real setup paths:

git clone https://github.com/vaquarkhan/data-engineering-agent-skills.git
scripts/install.sh --tool all --target /path/to/project

Windows-friendly file install:

pwsh scripts/install.ps1 --tool all --target C:\path\to\project

If your agent or editor supports importing a GitHub-hosted skill registry directly, use that tool's documented repository-install command against this repository URL rather than the example syntax from another project.

For Python-based proof assets and validators, install local dependencies with:

pip install -r requirements.txt

Contract validation quickstart

The dataset contract validator checks sample data against templates/dataset-contract.yaml-style contracts. It requires a contract file and a data source (--data or --duckdb); validating a template alone will error by design.

python scripts/validate_dataset_contract.py --help

python scripts/validate_dataset_contract.py \
  --contract examples/aws-s3-glue-athena-iceberg/contracts/customers-contract.yaml \
  --data examples/aws-s3-glue-athena-iceberg/data/customers.jsonl \
  --reference-time 2026-05-02T00:00:00Z

For dbt + DuckDB local proof:

python scripts/validate_dataset_contract.py \
  --contract examples/dbt-warehouse-marts/contracts/fct_daily_revenue-contract.yaml \
  --duckdb examples/dbt-warehouse-marts/build/dbt_warehouse_marts.duckdb \
  --query "select * from fct_daily_revenue order by order_date"

Bootstrap install shortcuts:

./bootstrap.sh /path/to/project auto

pwsh .\bootstrap.ps1 C:\path\to\project auto

Quick Start

Start Here

1. load skills/using-data-engineering-agent-skills/SKILL.md
2. pick the closest platform preset from presets/
3. choose the safest next command from the lifecycle below
4. use one starter pack, template, or example to reduce guessing

Install Surfaces

📦 Plugin And Release Downloads

⬇️ Install from VS Code Marketplace — search "Data Engineering Agent Skills" in VS Code Extensions panel

⬇️ Install from JetBrains Marketplace — IntelliJ, PyCharm, DataGrip, WebStorm, GoLand

⬇️ Download VS Code Extension (.vsix) — manual install for VS Code, Cursor, Windsurf, VSCodium

⬇️ Download JetBrains Plugin (.zip) — alternative manual download

How to install the .vsix:

1. Download the .vsix file from the latest release
2. Open VS Code / Cursor / Windsurf / Kiro
3. Press Ctrl+Shift+P (or Cmd+Shift+P on Mac) to open the Command Palette
4. Type Extensions: Install from VSIX... and select it
5. Browse to and select the downloaded .vsix file
6. Reload the editor when prompted
7. Use the Command Palette again — search for Data Engineering Skills: to see all available commands

How to install the JetBrains .zip:

1. Download the .zip file from the latest release
2. Open IntelliJ / PyCharm / DataGrip
3. Go to Settings → Plugins → ⚙️ gear icon → Install Plugin from Disk...
4. Select the downloaded .zip file
5. Restart the IDE when prompted

Additional plugin resources:

• plugin publishing and marketplace workflow setup: docs/plugin-publishing.md
• marketplace publish workflows exist, but this repository should be treated as GitHub Releases-first unless a marketplace listing is explicitly published
• Claude plugin bundle: use .claude-plugin/, .claude/commands/, and CLAUDE.md

Install By Tool

Tool or surface	Best starting link	Install path
VS Code	vscode-extension/README.md	Install from Marketplace or download .vsix from Releases
Cursor	docs/cursor-setup.md	use .cursor/rules/ or scripts/install.sh --tool cursor
Claude	docs/claude-setup.md	use .claude/commands/, .claude-plugin/, CLAUDE.md, or scripts/install.sh --tool claude
JetBrains	docs/jetbrains-setup.md	Install from Marketplace or download .zip from Releases
Copilot	docs/copilot-setup.md	use .github/copilot-instructions.md or scripts/install.sh --tool copilot
Kiro	docs/kiro-setup.md	use .kiro/steering/ or scripts/install.sh --tool kiro
Windsurf	docs/windsurf-setup.md	use .windsurfrules.example or scripts/install.sh --tool windsurf
OpenCode	docs/opencode-setup.md	use .opencode/ or scripts/install.sh --tool opencode
Codex	docs/codex-setup.md	use AGENTS.md, CLAUDE.md, skills-index.md, and docs/codex-setup.md
generic AGENTS.md consumers	docs/getting-started.md	use scripts/install.sh --tool all or copy the core files manually

One-Line Script Install

scripts/install.sh --tool claude --target /path/to/project
scripts/install.sh --tool kiro --target /path/to/project
scripts/install.sh --tool windsurf --target /path/to/project
scripts/install.sh --tool opencode --target /path/to/project
scripts/install.sh --tool all --target /path/to/project
scripts/install.sh --tool auto --target /path/to/project
scripts/install.sh --tool cursor,claude --target /path/to/project
scripts/install.sh --tool all --target /path/to/project --symlink

Windows-friendly install path:

pwsh scripts/install.ps1 --tool all --target C:\path\to\project

Evaluation runner:

python evals/run.py

Tool Setup Guides

• docs/getting-started.md
• docs/cursor-setup.md
• docs/kiro-setup.md
• docs/claude-setup.md
• docs/copilot-setup.md
• docs/windsurf-setup.md
• docs/opencode-setup.md
• docs/codex-setup.md
• docs/jetbrains-setup.md
• docs/plugin-publishing.md

Tutorials

• tutorials/README.md
• tutorials/using-data-engineering-agent-skills.md
• tutorials/cloud-data-engineering-architecture-patterns.md
• tutorials/pipeline-orchestration-patterns.md
• tutorials/streaming-architecture-patterns.md
• tutorials/data-resiliency-testing-patterns.md
• tutorials/etl-elt-modernization-and-cutover.md
• tutorials/regulated-data-and-compliance-workflows.md
• tutorials/installing-vscode-and-jetbrains-plugins.md

Case Studies

• case-studies/README.md
• case-studies/incident-bad-publish-recovery.md
• case-studies/replay-safe-backfill-after-corruption.md
• case-studies/regulated-data-release-gate.md

Core Principles

• Spec before pipeline code
• Contract-first source and output design
• Idempotent, replayable, backfill-safe execution
• Data quality checks before publish
• Lineage, ownership, and governance by default
• Clear evidence for every change

Feature Highlights

• Spec-first lifecycle with /spec, /plan, /build, /validate, /review, /backfill, and /ship
• 73 workflow skills covering ingestion, transformation, orchestration, streaming, lakehouse, warehousing, governance, quality tooling, legacy modernization, release, incident recovery, and platform operating concerns, plus 1 compatibility alias entry skill
• focused resilience testing coverage for failure drills, replay safety, restart behavior, backlog catch-up, publish protection, and disaster recovery readiness
• 14 platform presets spanning AWS, Azure, GCP, Databricks, Snowflake, Alibaba Cloud, Informatica, Talend, and Apache-first stacks
• Multi-agent packaging for Cursor, Claude, Copilot, Gemini, Codex, Kiro, OpenCode, Windsurf, AGENTS.md, and CLAUDE.md consumers
• Install surfaces for VS Code family editors, JetBrains IDEs, setup guides, one-line install scripts, and plugin release artifacts
• Plugin delivery and discovery workflows for VS Code and JetBrains, including release downloads, install smoke tests, markdown linting, and optional marketplace publish automation
• 5 runnable example scaffolds with contract validation, rollback demonstrations, and smoke-test proof paths, plus 9 architecture blueprint examples (spec/plan/tasks only)
• Agent benchmark pack with quantified with-skills coverage improvement (23→67 on the included task set)
• Structured operational templates for dataset contracts, compliance controls, backfills, schema changes, release gates, and incident response

Feature Coverage

Area	What is included	Good starting point
Core delivery workflow	Spec-driven delivery, planning, validation, review, publish readiness, replay safety, and rollback-aware release flow	skills/using-data-engineering-agent-skills/SKILL.md
Cloud and platform coverage	AWS, Azure, GCP, Databricks, Snowflake, Alibaba Cloud, multi-cloud and hybrid guidance	presets/
Apache and OSS data stack	Spark, Flink, Airflow, Kafka, Iceberg, Hudi, Beam, Trino, ClickHouse, DuckDB, LakeFS, OpenMetadata, DataHub, OpenLineage, Great Expectations, Deequ, Cuallee, Soda, Superset, and schema-registry patterns	skills-index.md
Orchestration and streaming	Scheduler-driven, event-driven, and lakehouse-native orchestration patterns plus replay-safe streaming architecture guidance	references/pipeline-orchestration-patterns.md, references/streaming-architecture-patterns.md
Security, governance, and regulated data	PII, PCI, HIPAA, lineage, retention, deletion, privacy, audit evidence, platform security, publish controls, and cloud-native governance for Glue Data Catalog, Lake Formation, Unity Catalog, Purview, and Dataplex	skills/data-security-compliance-and-regulated-data/SKILL.md, references/platform-native-governance-patterns.md
Regional compliance and reporting	GDPR, sovereignty and residency patterns, SAMA, Europe, USA, India, Saudi Arabia, CSRD, ESRS, BRSR, and ESG reporting workflows	references/regional-compliance-and-data-sovereignty-checklist.md, references/esg-and-sustainability-reporting-checklist.md
Modernization and enterprise ETL	ETL versus ELT, data migration, cutover, enterprise ETL modernization, mainframe offload, and guidance for Informatica, Talend, DataStage, SSIS, Matillion, and mainframe-origin estates	skills/etl-elt-and-modernization-strategy/SKILL.md, skills/mainframe-modernization-and-data-offload/SKILL.md
Testing and lower environments	Contract testing, testcase patterns, data-quality tools, resiliency testing, failure drills, reconciliation, anti-patterns, synthetic data, masked lower environments, and validation-security review starter packs	references/data-quality-tooling-and-rule-management.md, starter-packs/resiliency-testing-starter.yaml
Language-specific engineering	Python pipeline packaging, Scala JVM data jobs, and Java integration or metadata services	skills/python-data-engineering-and-pipeline-packaging/SKILL.md, skills/scala-data-engineering-on-jvm-runtimes/SKILL.md, skills/java-data-engineering-and-integration-services/SKILL.md
Adoption and automation	VS Code extension, JetBrains plugin, Kiro steering, hooks, starter packs, examples, MCP templates, and machine-readable asset registry	docs/getting-started.md, registry/assets.json

Lifecycle Commands

These commands are the clearest way to understand the repo. They can be mapped to slash commands, prompts, hooks, or local workflows.

What you're doing	Command	Key principle
Define the data product	/spec	Contract, SLA, lineage, schema, retention, ownership
Plan the work	/plan	Small, atomic, verifiable tasks
Build incrementally	/build	Safe slices over big rewrites
Prove publish safety	/validate	Quality, contract compliance, and reconciliation are proof
Review the change	/review	Reliability, governance, cost, and operability
Replay or cut over safely	/backfill	Backfills, reruns, and cutovers need explicit guardrails
Ship and operate	/ship	Safe rollout, observability, and rollback paths

Choose Your Path

• New pipeline -> use /spec
• dbt project -> use starter-packs/warehouse-analytics-starter.yaml
• Streaming system -> use starter-packs/streaming-reliability-starter.yaml
• Production reliability and failure recovery -> use starter-packs/production-reliability-starter.yaml and tutorials/production-reliability-and-mcp-observability.md
• Serverless Spark or MSK hardening -> use examples/aws-serverless-spark-msk-reliability/
• Streaming architecture review -> use references/streaming-architecture-patterns.md
• Cloud architecture review -> use references/cloud-data-engineering-architecture-patterns.md
• Pipeline orchestration review -> use references/pipeline-orchestration-patterns.md
• Validation and testcase review -> use starter-packs/validation-security-review-starter.yaml
• Resiliency testing -> use starter-packs/resiliency-testing-starter.yaml
• Python data pipeline work -> use python-data-engineering-and-pipeline-packaging
• Scala Spark or JVM jobs -> use scala-data-engineering-on-jvm-runtimes
• Java connectors or data services -> use java-data-engineering-and-integration-services
• MySQL versus NoSQL choice -> use operational-datastore-selection-relational-and-nosql
• Anti-pattern review -> use references/data-engineering-anti-patterns.md
• ETL or ELT modernization -> use etl-elt-and-modernization-strategy
• Partner file or SFTP ingestion -> use file-and-partner-feed-ingestion
• Glue Data Catalog or Lake Formation governance -> use glue-data-catalog-and-lake-formation-governance
• Unity Catalog governance -> use unity-catalog-and-lakehouse-governance
• Azure governance or Purview -> use microsoft-purview-and-azure-data-governance
• Dataplex or BigQuery governance -> use dataplex-and-bigquery-governance
• Snowflake-native pipelines -> use snowflake-native-pipelines-and-governance
• BigQuery or Dataform platform work -> use bigquery-and-dataform-platform-engineering
• Data-quality tool strategy -> use data-quality-platforms-and-rule-management
• Incident recovery -> use incident-triage-and-pipeline-recovery
• Disaster recovery planning -> use data-platform-disaster-recovery-and-business-continuity
• Platform operating model or golden paths -> use data-platform-operating-model-and-service-ownership
• Schema migration -> use schema-evolution-and-contract-migrations
• Cutover or backfill -> use safe-backfill-and-replay-orchestration and data-migration-and-platform-cutover
• CI/CD and progressive release -> use starter-packs/data-platform-cicd-release-starter.yaml
• Regional compliance or sovereignty -> use regional-data-compliance-and-sovereignty
• ESG reporting -> use starter-packs/regional-compliance-and-esg-reporting-starter.yaml
• Regulated data (PII, PCI, HIPAA) -> use data-security-compliance-and-regulated-data
• Test data or synthetic fixtures -> use test-data-preparation-and-synthetic-data
• Lower-environment masked refresh -> use lower-environment-data-masking-and-obfuscation
• Informatica or Talend estate -> use enterprise-etl-and-data-integration-modernization
• Mainframe modernization -> use mainframe-modernization-and-data-offload

Hooks

The hooks/ directory adds a lightweight operating layer:

• session-start.sh
• contract-check-pre.sh
• pipeline-review-pre.sh
• incident-mode.sh
• backfill-guard.sh
• schema-change-guard.sh
• cost-check.sh
• release-guard.sh

Use these hooks when you want the repository to behave more like a workflow system than a static library.

Initial Skill Pack

This repository includes 73 workflow skills plus 1 compatibility alias (using-data-agent-skills redirects to using-data-engineering-agent-skills). The alias exists for older installs; do not count it as a separate workflow.

This repository now includes a broader production-grade skill pack:

• using-data-engineering-agent-skills (canonical start-here skill)
• using-data-agent-skills (compatibility alias — redirect only)
• data-specification
• pipeline-planning-and-task-breakdown
• data-quality-and-contract-testing
• data-resiliency-testing-and-failure-injection
• orchestration-and-backfills
• lineage-pii-and-governance
• data-security-compliance-and-regulated-data
• regional-data-compliance-and-sovereignty
• esg-and-sustainability-regulatory-reporting
• lower-environment-data-masking-and-obfuscation
• spark-and-distributed-processing
• airflow-and-workflow-orchestration
• streaming-and-messaging-systems
• file-and-partner-feed-ingestion
• lakehouse-table-format-engineering
• data-lake-and-zone-architecture
• warehouse-and-schema-design
• python-data-engineering-and-pipeline-packaging
• scala-data-engineering-on-jvm-runtimes
• java-data-engineering-and-integration-services
• operational-datastore-selection-relational-and-nosql
• data-mesh-and-domain-oriented-design
• delta-lake-and-medallion-architecture
• dbt-and-analytics-engineering
• cdc-and-incremental-loading
• schema-evolution-and-contract-migrations
• warehouse-performance-and-cost-optimization
• data-platform-disaster-recovery-and-business-continuity
• data-observability-and-sla-management
• incident-triage-and-pipeline-recovery
• data-platform-operating-model-and-service-ownership
• terraform-and-data-platform-infrastructure
• snowflake-native-pipelines-and-governance
• bigquery-and-dataform-platform-engineering
• semantic-layer-and-metric-governance
• notebook-to-production-hardening
• data-sharing-and-publishing-contracts
• data-catalog-and-discovery
• glue-data-catalog-and-lake-formation-governance
• unity-catalog-and-lakehouse-governance
• microsoft-purview-and-azure-data-governance
• dataplex-and-bigquery-governance
• privacy-retention-and-right-to-delete
• etl-elt-and-modernization-strategy
• mainframe-modernization-and-data-offload
• test-data-preparation-and-synthetic-data
• reverse-etl-and-operational-data-serving
• feature-store-and-ml-data-pipelines
• data-migration-and-platform-cutover
• api-and-saas-ingestion-patterns
• source-reliability-and-extraction-resilience
• enterprise-etl-and-data-integration-modernization
• data-reconciliation-and-financial-controls
• data-platform-ci-cd-and-release-management
• master-data-and-entity-resolution
• debezium-and-kafka-connect-cdc
• apache-beam-unified-batch-and-stream
• apache-hudi-lakehouse
• trino-presto-federated-query
• openmetadata-datahub-and-openlineage
• great-expectations-deequ-and-cuallee
• data-quality-platforms-and-rule-management
• lakefs-and-data-versioning
• clickhouse-real-time-analytics
• superset-and-metrics-serving
• avro-protobuf-json-schema-registry
• duckdb-local-analytics-and-dev
• `data-contract-testin