CLAUDE.md Template: ETL Pipeline Projects

Overview

A CLAUDE.md file placed in your ETL pipeline project root gives build-cli persistent context about the project — programming language, framework, orchestration tool, source and target systems, coding conventions, and data quality requirements. This article provides a ready-to-use CLAUDE.md template for Python and PySpark ETL pipelines (AWS Glue, Airflow, Step Functions, or plain Python).

Copy the template below, fill in the placeholders, and commit it to your project root as CLAUDE.md.

Background

ETL pipeline projects have highly project-specific conventions — credential handling patterns, logging utilities, schema enforcement approaches, and idempotency requirements. Without persistent context, build-cli may generate code that uses hardcoded credentials, prints instead of logging, or appends instead of upserts, violating your project’s standards. A well-filled CLAUDE.md ensures the AI generates code that fits your project conventions from the first prompt.

The template includes the most common ETL patterns (S3-to-Redshift, RDS-to-Snowflake, etc.) and is designed to be adapted for any Python or PySpark pipeline project.

Main Content

Copy this template into CLAUDE.md at your project root and replace all [PLACEHOLDER] values:

## Project

**[PROJECT NAME]** — [One sentence: what this pipeline does, what it moves where.]

## Tech stack

- Language: Python [VERSION]
- Framework: [PySpark / Pandas / AWS Glue / plain Python]
- Orchestration: [Airflow / Step Functions / cron / none]
- Source: [SOURCE SYSTEM AND FORMAT — e.g., S3 Parquet, RDS PostgreSQL]
- Target: [TARGET SYSTEM — e.g., Redshift, S3, Snowflake]
- Infrastructure: AWS ([services used — e.g., Glue, S3, Secrets Manager, Lambda])

## Pipeline structure

[Describe the high-level flow, e.g.:] Ingest (S3) → Validate → Transform → Load (Redshift)

Key files:
- `src/ingest.py` — [what it does]
- `src/transform.py` — [what it does]
- `src/load.py` — [what it does]
- `config/` — pipeline configuration and schema definitions

## Conventions

- Schema: all DataFrames use explicit schema definitions — no inferred schemas in production
- Credentials: use AWS Secrets Manager; never hardcode
- Logging: `from utils.logger import get_logger` — use this, not `print()`
- Error handling: raise on schema violations; log and skip (with metrics) on individual record errors
- Idempotency: all loads must be safe to re-run (use upsert or truncate-load, not append)
- Naming: snake_case for all variables and files

## Data quality

- DQ checks run after ingest, before load
- Quality gate: fail the job if critical DQ rules fire; warn and continue for non-critical
- DQ rule definitions are in `config/dq_rules.yaml`

## Running locally

```bash
python src/main.py --env dev --date [YYYY-MM-DD]

Testing

pytest tests/

Tests use synthetic data from tests/fixtures/. Never use production data in tests.

Known constraints

[Any known limitations, upstream dependencies, or fragile areas to be careful about]

## Summary

- Place `CLAUDE.md` in the project root to give build-cli accurate context for every session.
- The "Conventions" section is critical: it prevents the AI from hardcoding credentials, using `print()` instead of your logging utility, or generating non-idempotent load patterns.
- Fill in "Known constraints" with upstream dependencies and fragile areas — this prevents the AI from suggesting refactors that would break production.
- The "Data quality" section ensures the AI knows about your DQ gates and does not generate code that bypasses them.
- Commit `CLAUDE.md` to version control so all team members work with consistent context.
- See the other CLAUDE.md example templates in this hub for data governance, dbt, and SQL analytics projects.