Building the pipeline orchestration engine itself as a DAG of stages, letting independent stages run in parallel, and the artifact storage and promotion mechanics underneath build-once-promote-everywhere.
Published September 23, 2026
CI/CD Pipeline Design (in the DevOps subject) covered how a LEAD designs and structures a pipeline for their team. This case is different: designing the PIPELINE ENGINE itself — the system (like Jenkins, GitHub Actions, or GitLab CI under the hood) that actually executes pipeline definitions.
Design a CI/CD system that triggers on a code commit, executes a defined sequence of stages (build, test, deploy), supports stages running in parallel where possible, and supports rolling back a deployment.
Functional: trigger pipeline execution on a commit/webhook event; execute a configurable sequence of stages; support rollback to a previous successful deployment; store and retrieve build artifacts. Non-functional: minimize total pipeline execution time (parallelize independent work); handle many concurrent pipeline runs across many repositories/teams; artifact storage must scale with build volume over time.
┌─→ [Unit Tests] ─┐
[Build] ─┤ ├─→ [Integration Tests] ─→ [Deploy to Staging] ─→ [Deploy to Prod]
└─→ [Static Analysis] ┘
Modeling a pipeline as a DAG (Directed Acyclic Graph) of stages, rather than a fixed linear sequence, is what enables genuine PARALLELISM — Unit Tests and Static Analysis both depend only on Build completing, and have NO dependency on each other, so they can run CONCURRENTLY once Build finishes, rather than being forced into an arbitrary sequential order (Unit Tests then Static Analysis) that a purely linear pipeline definition would impose. This directly extends CI/CD Pipeline Design's stage-sequence discussion with the actual EXECUTION MODEL underneath: the DAG structure is what an engine like this needs to represent, schedule, and execute correctly.
This is structurally the SAME problem as Design a Workflow Engine, applied specifically to CI/CD stages — a pipeline execution engine needs to track which stages have completed, determine which NEWLY-eligible stages can now start (every one of their dependencies has succeeded), and run independently-eligible stages in parallel via a worker pool. The Step/WorkflowContext shape from that LLD exercise maps directly: a pipeline Stage is a Step, pipeline-run state (which commit, which artifacts produced so far) is the WorkflowContext.
Build stage produces an artifact (a container image, a JAR) → stored ONCE in an
artifact registry (versioned, immutable) → later stages (deploy-to-staging,
deploy-to-prod) REFERENCE that SAME stored artifact, never rebuilding it
This is the concrete storage-layer implementation of CI/CD Pipeline Design's build-once-promote-everywhere principle — the pipeline engine needs a dedicated ARTIFACT REGISTRY (conceptually similar to S3 with versioning, or a container registry) that stores each build's output durably and immutably, with later pipeline stages referencing that exact stored artifact by its version/digest rather than triggering a fresh build. Artifact storage volume grows continuously with build frequency, making lifecycle policies (S3's aging/expiration model, applied to old build artifacts) a genuine, necessary cost-management concern here too.
Q: How does the engine handle a stage that fails partway through a DAG with other stages still in progress? A: In-progress SIBLING stages (with no dependency on the failed one) can reasonably be allowed to finish (their results might still be useful for diagnosis), but any DOWNSTREAM stage depending on the failed one must not start — this requires the engine to track dependency satisfaction explicitly per stage, not simply react to 'something failed' by halting everything indiscriminately.
Q: Why is rollback implemented as 'redeploy a PREVIOUS artifact' rather than 're-running the pipeline with old code'? A: Re-running the full pipeline against old code risks producing a DIFFERENT artifact than what was originally deployed (dependency versions could have moved, environment could have changed) — redeploying the EXACT previously-stored, immutable artifact (from the registry) guarantees you're rolling back to precisely what was verified working before, directly reflecting CI/CD Pipeline Design's build-once-promote-everywhere correctness argument.
Q: How would this system scale to handle many concurrent pipeline runs across many teams/repositories? A: The worker pool executing individual stages needs to scale horizontally and be shared/multiplexed across concurrent pipeline runs (similar to Design a Distributed Task Scheduler's worker-pool model) — with per-team/per-repo resource quotas or priority tiers to prevent one team's high pipeline volume from starving another team's pipeline runs of available workers.
Q: Does representing the pipeline as a DAG create any new failure modes compared to a simple linear sequence? A: A malformed pipeline DEFINITION could accidentally introduce a CYCLE (stage A depends on B, B depends on A) — the engine needs to validate the DAG is genuinely acyclic at definition/parse time, rejecting an invalid pipeline definition upfront rather than discovering the problem only when execution deadlocks waiting on a dependency cycle that can never resolve.