Updated: Apr 22
Keep your workflows DRY Github Reusable Workflows
It is common in an organizational environment to have multiple applications built with the same technologies or frameworks, and even having them sharing a common CI/CD pipeline. This, most of the time ends up in having to repeat a lot of code to, for example, deploy a Terraform infrastructure, upload a container image to a registry and other similar jobs.
For this reason, to keep your pipelines DRY (Don’t Repeat Yourself) we bring you Github Reusable Workflows
Here we have an example of how a normal workflow to upload a container image to ECR looks like:
build_n_push:
name: Build and Push to ECR
runs-on: ubuntu-latest
outputs:
image_tag: ${{ steps.set-image-tag.outputs.image_tag }}
latest_tag: ${{ steps.set-latest-tag.outputs.latest_tag }}
steps:
- name: Checkout code
uses: actions/checkout@v2
with:
submodules: true
fetch-depth: 0
- name: Set Image tag
id: set-image-tag
shell: bash
run: |
echo "::set-output name=image_tag::$(echo ${{ github.sha }} | cut -c1-12)"
- name: Set LATEST tag
id: set-latest-tag
shell: bash
run: |
if [ ${{ github.ref }} == 'refs/heads/develop' ]; then echo "::set-output name=latest_tag::latest"; else echo "::set-output name=latest_tag::latest-staging"; fi
- name: Configure AWS Credentials
id: configure-credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-east-1
- name: Login to Amazon ECR
id: login-container-registry
uses: aws-actions/amazon-ecr-login@v1
- name: Build and Push Container Image
id: build-container-image
env:
ECR_REGISTRY: ${{ steps.login-container-registry.outputs.registry }}
ECR_REPOSITORY: my_ecr_repo
run: |
docker build . \
--progress plain \
--file Dockerfile \
--tag $ECR_REGISTRY/$ECR_REPOSITORY:${{ steps.set-image-tag.outputs.image_tag }}
docker tag \
$ECR_REGISTRY/$ECR_REPOSITORY:${{ steps.set-image-tag.outputs.image_tag }} \
$ECR_REGISTRY/$ECR_REPOSITORY:${{ steps.set-latest-tag.outputs.latest_tag }}
docker push $ECR_REGISTRY/$ECR_REPOSITORY:${{ steps.set-image-tag.outputs.image_tag }}
docker push $ECR_REGISTRY/$ECR_REPOSITORY:${{ steps.set-latest-tag.outputs.latest_tag }}
Now imagine having to copy this workflow on each one of your applications (even you will need to copy your deployment workflows too!). This is a lot of tedious and repetitive work, also this involves a lot of human interaction, therefore making our pipelines more prone to have errors.
It would be great to have all this standar code in a place, where we can maintain it more easily and which we can use from other repos in our organization. Here is where Reusable Workflows come handy.
It is easy to create a reusable workflow for your Github Actions CI/CD pipeline, but we have to keep in mind some limitations related to them before starting:
Reusable workflows can’t call other reusable workflows
You can’t call reusable workflows in a private repository unless you are in the same repository.
Environment variables aren’t propagated from caller workflow to called workflow (don’t worry we can solve this with inputs)
So, now we can define our first reusable workflow:
name: Build and push a Docker image to ECR
on:
workflow_call:
inputs:
ECR_REPO:
required: true
type: string
AWS_REGION:
required: false
type: string
default: "us-east-1"
outputs:
image_tag:
description: Image tag created on the workflow
value: ${{ jobs.build_n_push.outputs.image_tag }}
latest_tag:
description: Latest tag created on the workflow
value: ${{ jobs.build_n_push.outputs.latest_tag }}
secrets:
AWS_ACCESS_KEY_ID:
required: true
AWS_SECRET_ACCESS_KEY:
required: true
jobs:
As you can see, with just adding a few things we can create a reusable workflow. We have the following parameters:
on.workflow_call: this tells Github that this workflow will be triggered via call of another workflow.
inputs: this is non sensitive data that we can pass to our workflow, in this case we are passing the ECR Registry and the AWS Region.
outputs: this is data that our workflow will output to other jobs in the caller workflow. In this example we are setting as output the image tag and the latest tag for that image (we can chain this output to our deploy workflow 😉😉 )
secrets: sensitive data that will be used by our workflow.
After all these parameters, all we have to do is to define our workflow like a normal one, specifying all the jobs that will run along with their steps.
To make use of our new reusable workflow first of all you can have it on the same repo of your application, or a better solution would be to have this workflow on a public repository of your organization. This makes your workflow publicly accessible unless you do a trick I will teach you in the end 🤫.
We have two ways to invoke our workflow, the first one if it is on the same repo:
name: Continuous Deployment
on:
push:
branches:
- main
- develop
jobs:
build_and_push_image:
name: Build container image
uses: .github/workflows/docker_build_and_push.yaml@master
with:
AWS_REGION: us-east-1
ECR_REPO: my-repo
secrets:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
And now if we have our public repo with the workflow:
name: Continuous Deployment
on:
push:
branches:
- main
- develop
jobs:
build_and_push_image:
name: Build container image
uses: our-org/our-workflow-repo/.github/workflows/docker_build_and_push.yaml@master
with:
AWS_REGION: us-east-1
APP_NAME: my-repo
secrets: inherit
About the inherit, you can use it when both your repos have access to the same secrets, and they have the same name in each repo (like using organizational secrets).
Now… maybe you want to have this workflow on a public repository so you can call it from all your applications repositories but you may not like that everyone else can call your workflow. Well there is a simple trick to ensuring only your organization can call your workflow, you can use the following template:
jobs:
check_org:
name: Check Caller
runs-on: ubuntu-latest
steps:
- name: Check the calling organization
if: ${{ github.repository_owner != 'my-org'' }}
uses: actions/github-script@v3
with:
script: |
core.setFailed('This reusable workflow can only be used by My Org.')
my_normal_job:
needs: check_org
name: This is my normal job
runs-on: ubuntu-latest
steps:
#....
As you can see, we are using a Github context variable called repository_owner, this ensures that if the calling repo isn’t from our organization it can’t use the workflow (at least from our public repo). Remember to add the needs: check_org on your actual job.
Juan Wiggenhauser
DevOps
Teracloud