The release of GitLab 14.2 brings an exciting new feature to the management of CI/CD pipelines. Stages can now be completely omitted. Instead, the dependencies between pipeline jobs can be specified using the needs keyword. In this article, I will go into these changes and show you an example pipeline definition file making use of stageless pipelines.
Traditionally, CI/CD pipelines in GitLab would consist of multiple stages and each stage would contain multiple jobs:
The reasoning behind this structure was that all jobs of a single stage would be executed before jobs of the next stage, while all jobs of the same stage could be run in parallel. This execution model is fine for simple pipelines, but it restricts the performance of more complex ones.
Instead, a better solution is to specify for each job individually on which other jobs it depends on. This is why the needs keyword was introduced to GitLab. However, until GitLab 14.2, it was still required to define stages. Gladly, this has changed. Let’s have a look at this new way of writing pipelines.
Simple Pipeline Examples
We will start with a basic pipeline template as provided by gitlab.com:
stages:
- build
- test
- deploy
build-job:
stage: build
script:
- echo "Compiling the code..."
- echo "Compile complete."
unit-test-job:
stage: test
script:
- echo "Running unit tests... This will take about 60 seconds."
- sleep 60
- echo "Code coverage is 90%"
lint-test-job:
stage: test
script:
- echo "Linting code... This will take about 10 seconds."
- sleep 10
- echo "No lint issues found."
deploy-job:
stage: deploy
script:
- echo "Deploying application..."
- echo "Application successfully deployed."
This pipeline defines three stages (build, test and deploy) which consist of one, two, and one job, respectively. The only stage with more than one job is the test stage that contains both the unit-test-job as well as the lint-test-job.
We can transform this simple pipeline into a stageless one in two steps:
- Remove the
stages:definition block - Replace each
stage:keyword with aneeds:keyword that references all the job dependencies
The result of this transformation is the following configuration:
build-job:
script:
- echo "Compiling the code..."
- echo "Compile complete."
unit-test-job:
needs: [build-job]
script:
- echo "Running unit tests... This will take about 60 seconds."
- sleep 60
- echo "Code coverage is 90%"
lint-test-job:
needs: [build-job]
script:
- echo "Linting code... This will take about 10 seconds."
- sleep 10
- echo "No lint issues found."
deploy-job:
needs: [unit-test-job, lint-test-job]
script:
- echo "Deploying application..."
- echo "Application successfully deployed."
As you can see, this new configuration is both shorter (30 lines before, 24 lines after) and more explicit (for each job, you can see the dependencies directly in the job’s block).
When you execute this pipeline, you can also visualize the dependencies in GitLab’s Pipeline view:
With this new configuration, we have actually just replicated the same pipeline as before just without stages – but where is the advantage? To demonstrate why this flexibility can result in better overall performance, consider adding a new job that only depends on the result of the lint-test-job. If we still had stages, we would need to add this new job in a new stage between the test and deploy stage. However, then the new job would also need to wait for the unit-test-job to complete, even though the new job doesn’t care at all about those results.
In the new stageless world, this change is as simple as adding this configuration block:
post-linting-job:
needs: [lint-test-job]
script:
- echo "Postprocessing linting results... This will take about 10 seconds."
- sleep 10
- echo "Postprocessing completed."
The resulting pipeline automatically schedules the new job after the lint-test-job is completed:
And here is the most beautiful thing: the new post-linting-job is already executed while the unit-test-job is still running, thereby improving the overall execution time of the pipeline.
Complete template
I encourage you to experiment with this new feature as it will improve both the clarity and the performance of your CI/CD pipelines. Feel free to start based on my template:
build-job:
script:
- echo "Compiling the code..."
- echo "Compile complete."
unit-test-job:
needs: [build-job]
script:
- echo "Running unit tests... This will take about 60 seconds."
- sleep 60
- echo "Code coverage is 90%"
lint-test-job:
needs: [build-job]
script:
- echo "Linting code... This will take about 10 seconds."
- sleep 10
- echo "No lint issues found."
post-linting-job:
needs: [lint-test-job]
script:
- echo "Postprocessing linting results... This will take about 10 seconds."
- sleep 10
- echo "Postprocessing completed."
deploy-job:
needs: [unit-test-job, lint-test-job]
script:
- echo "Deploying application..."
- echo "Application successfully deployed."