AI-Assisted Python and PySpark Pipeline Review

Who This Guide Is For

Software engineers who work with Python or PySpark data pipelines and want to use build-cli to catch issues before submitting code for human review. Experience with Python data engineering (AWS Glue, Spark, Pandas) is assumed.

What You Will Learn

How to give build-cli the context it needs to review a pipeline effectively
A structured five-step review workflow covering correctness, error handling, data quality, performance, security, and maintainability
How to drill into specific findings and generate a PR review comment
A CLAUDE.md snippet for pipeline projects to give every session relevant context
Known limitations of AI-assisted code review

Step-by-Step Instructions

Step 1: Start with Context

Before pasting code, give build-cli the context it needs:

I'm going to share a Python data pipeline for review. Here's the context:

Purpose: [one sentence — what this pipeline does]
Input: [data source, format, approximate volume]
Output: [target system, format]
Tech stack: [Python version, PySpark/Pandas, AWS Glue / local, etc.]
Key business rules: [any transformation rules the code should implement]

Step 2: Request the Structured Review

Please review this pipeline code. Focus on:
1. Correctness — does it implement the business rules described above?
2. Error handling — what happens when the input is empty, malformed, or late?
3. Data quality risks — schema drift, type coercion surprises, silent null propagation
4. Performance — unnecessary collects, missing partitioning, broadcasting opportunities
5. Security — hardcoded credentials, unsafe file paths, unvalidated inputs
6. Maintainability — magic numbers, unclear variable names, missing type hints

[paste code here]

Step 3: Drill into Specific Findings

On the error handling point — can you show me specifically where the pipeline would silently
continue if the source schema changes? What's the minimal change to make it fail loudly instead?

Step 4: Check Against Requirements

Based on the business rules I described, are there any transformations missing from this code?

Step 5: Generate the PR Review Comment

Write a structured code review comment for this pipeline.
Format: markdown, grouped by severity (blocking / non-blocking / suggestion).
Include: what I verified works, what I found, and what was fixed before submission.

Verification

Your review is complete when:

build-cli has assessed all six review dimensions (correctness, error handling, DQ risks, performance, security, maintainability)
You have drilled into any blocking findings and confirmed the fix
You have a formatted PR review comment ready for your pull request

Next Steps

See the ‘AI-Assisted SQL Code Review’ guide for the SQL equivalent of this workflow
See the ETL Pipeline CLAUDE.md example article for a ready-made CLAUDE.md template for pipeline projects

CLAUDE.md Snippet

Add this to your project’s CLAUDE.md to give build-cli pipeline context for every session:

## Pipeline conventions
- Language: Python 3.10+
- Framework: [PySpark / Pandas / AWS Glue]
- Error handling: raise exceptions on schema violations; log and skip on individual record errors
- Logging: use the team's standard logger (import from utils.logger)
- No hardcoded credentials — use AWS Secrets Manager or environment variables
- All DataFrames must have explicit schema definitions; inferred schemas are not allowed in production

## Review focus
When reviewing pipeline code, check: schema enforcement, null handling on join keys,
idempotency (can it be re-run safely?), and Secrets Manager usage for credentials.

Example

Pipeline description: “Reads customer records from S3 (Parquet), joins with reference data from RDS, applies a de-duplication rule (keep most recent by updated_at), and writes to Redshift.”

Excerpt reviewed:

df = spark.read.parquet(f"s3://{bucket}/{prefix}/")
ref = spark.read.jdbc(url=jdbc_url, table="ref_customers", properties=props)
joined = df.join(ref, df.customer_id == ref.id)
deduped = joined.orderBy("updated_at", ascending=False).dropDuplicates(["customer_id"])
deduped.write.jdbc(url=redshift_url, table="customers_clean", mode="overwrite")

AI review findings (excerpt):

BLOCKING — Overwrite mode on Redshift target: mode="overwrite" will drop and recreate the entire target table on every run. If the pipeline fails mid-write, the table is empty. Consider mode="append" with a pre-truncate step inside a transaction, or use a staging table + swap pattern.

BLOCKING — No schema enforcement: spark.read.parquet(...) infers the schema at runtime. If the source Parquet schema changes, the join will silently fail or produce nulls. Define an explicit schema and validate against it on read.

NON-BLOCKING — Join type not specified: df.join(ref, ...) defaults to inner join. If ref doesn’t have a matching record, the customer row is silently dropped. Confirm this is intended; if not, use a left join and handle nulls explicitly.

Known Limitations

The AI cannot run the pipeline or profile the data — performance findings are structural, not empirical.
Very large files (1000+ lines) should be split into logical sections for review.
PySpark-specific optimisations (broadcast hints, partition counts) depend on cluster configuration the AI doesn’t know.
Not suitable for code that processes PII or sensitive data without confirming the review doesn’t leak data context into prompts.