The data engineer portfolio: what hiring managers actually open
A data engineer portfolio is not a data analyst portfolio with fewer charts. Hiring managers open it looking for pipelines, models, and reliability thinking. Here are the four projects that prove you can build systems, and how to document infrastructure work so it survives a skim.
Most portfolio advice is written for analysts, and following it produces a data engineer portfolio that impresses nobody: notebooks, charts, a dashboard. A hiring manager for a data engineering role opens your portfolio looking for something else entirely, evidence that you can build systems that move data reliably. That means pipelines, models, and documented reliability thinking. This is what to build, and how to present work that has no pretty output layer.
The four projects that make the portfolio:
- An orchestrated ETL pipeline with retries and alerts
- A dimensional data model with a documented schema
- A batch or streaming job that justifies its own architecture
- Infrastructure as a README: setup, decisions, trade-offs
What changes when the role is engineering
The difference is not the tools, it is what gets evaluated. Analyst work is judged on the finding; engineering work is judged on the design. Nobody asks a pipeline what the business should do next, they ask what happens when Tuesday's run fails halfway through.
- A pipeline that runs end to end on a schedule
- Data modeling decisions you can defend
- Failure handling: retries, idempotent reruns
- Awareness of batch vs streaming trade-offs
- Reproducibility: someone else could run it
- Polished dashboards and BI layers
- Machine learning models
- Exploratory notebooks with narrative
- A dozen small script repos
- Paid cloud infrastructure running 24/7
That last point matters for juniors: you do not need a production cloud bill to prove engineering judgement. Local Airflow in Docker, a free-tier warehouse, and public data sources are enough, because what is being evaluated is the design and the documentation, not the hosting.
The four projects that cover the role
An orchestrated ETL pipeline
The centerpiece. Extract from a real API or files, load to a warehouse, transform, on a schedule, with retries and alerts thought through. Choosing where transformation happens is itself a design decision worth documenting: the ETL vs ELT cheat sheet frames the trade-off your README should address.
A dimensional data model
Take raw transactional data and design the warehouse layer: facts, dimensions, and the reasoning behind the grain of each table. Document why you chose a star schema over a snowflake, and build the transformations as tested dbt models rather than loose SQL files (dbt basics covers the core workflow).
A batch or streaming job
One project that processes data in motion or in volume: a Kafka consumer aggregating events, or an incremental batch job over a large dataset. The point is showing you know when each is the right call, which is exactly the trade-off in our batch vs streaming cheat sheet.
Infra as README
Not a separate build, a discipline across all three: an architecture diagram, a docker-compose that brings the stack up with one command, and a README that explains the design decisions. For infrastructure work, the documentation is the demo.
Together these mirror what the job actually is, which you can sanity-check against a day in the life of a data engineer: building flows, modeling data, and keeping both reliable.
For source data, favour feeds over files: a public API with fresh data every day (weather, transit positions, market prices, city open data) gives your pipeline a reason to be scheduled, which a static CSV never does. A pipeline that has genuinely run every night for three weeks, with the run history to show for it, is a quietly powerful line in an interview.
Reliability is the story: tell it
Here is what separates a junior repo from a hireable one. Anyone can write a script that moves data once. Engineers are paid for what happens on the thousandth run, so your READMEs should answer the failure questions before the interviewer asks them.
- 1
Make reruns safe, then say so
Idempotency is the single highest-signal word in a junior DE portfolio, because it shows you have thought past the happy path. If the concept is fuzzy, the idempotency cheat sheet is the five-minute version; your pipeline README should then demonstrate it: "rerunning a day is safe because loads overwrite the partition."
Questions to ask- What happens if the same run executes twice?
- Upserts and partition overwrites over blind inserts
- State it explicitly in the README
- 2
Document the failure modes
A short "failure modes" section in the README, three or four scenarios and how the pipeline responds, does more for your candidacy than another project. It is the exact conversation a DE interview becomes.
Questions to ask- Source API down: what happens?
- Bad data arrives: where is it caught?
- Late data: how is it handled?
- 3
Show the checks
Data quality checks prove you think about the data, not just the plumbing. Even simple dbt tests, documented, signal a production mindset.
Questions to ask- Row counts and freshness tests
- dbt tests on keys and accepted values
- What alerts, and who would know
Analysts are hired for answers. Engineers are hired for what happens when the answer has to arrive every morning at six.
What a finished DE portfolio looks like
Calibrate against a real one. This is a public D8A portfolio built from exactly the four-project arc above, with each project validated and documented.
Note the shape: four projects, not ten, each named for what it does, each one click from a description a non-engineer could follow. The portfolio page carries the presentation; the GitHub repos behind it carry the code and the design docs.
This split matters more for engineers than for analysts. DE work is the least visual of the data disciplines, so a bare GitHub profile undersells it badly: a recruiter without an engineering background sees folders of Python and YAML and has no way to tell a real orchestrated pipeline from a homework script. A portfolio page translates the work into terms the first screener understands, validated projects, named tools, a skills list, while keeping the repositories one click away for the technical interviewer who wants to read your DAG definitions and dbt tests. Both audiences screen you; build for both.
Building it without the blank page
The honest difficulty with DE portfolio projects is setup drag: standing up Airflow, a warehouse, and streaming infrastructure alone can eat the weeks you meant to spend engineering. That is the part the D8A Data Engineer path removes. The path is this exact sequence, data modeling, dbt, Airflow, Kafka, as guided projects on real data, with the environment decisions made for you. Each project is auto-validated by a structural check on your GitHub repo when you finish, then published to a public portfolio page like the one above. You still make the design decisions, because those are the point. You just skip the three weekends of configuration standing between you and them.