CircleCI - Running special workflows on demand

Tags: circleci, continuous integration

Recently, I had the pleasure of setting up the CI pipeline for my iOS project at work. Since other projects were already using CircleCI, it was quite natural for us to use it as well.

One hurdle that took me considerable time to solve was figuring out how to trigger workflows on demand. Since I have not been able to find any up to date resources on it, I decided to document my approach in the hopes of helping someone.

Use-case

Since I am working on an iOS application, the pipeline has the usual steps of running tests, after which it deploys a build for our QA team. Following the best practices, the QA build points to the staging environment, where QA can do whatever they want, without affecting live users.

However, once in a blue moon, we need to deploy a build that is pointing to the production environment. Unfortunately, CircleCI does not support triggering jobs on demand from their user interface.

Fortunately, after digging through their documentation, I found out that you can do this through their REST API, with the use of pipelines.

Initial setup

First, we need to set up our .circleci/config.yml file with the two workflows. It should look something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65

# Use CircleCI version 2.1
version: 2.1

aliases:
  # Define a shared alias for what Xcode version you want to use
  - &xcode_version "11.4.1"

workflows:
  version: 2.1

  # The standard workflow you want to run on every commit
  standard-workflow:
    jobs:
      - tests
      - deploy-to-qa:
        requires:
          - tests

  # The special workflow you want to run on demand
  on-demand-workflow:
    jobs:
      - deploy-to-qa-prod

jobs:
  tests:
    macos:
      xcode: *xcode_version
    steps:
      - setup
      - run-fastlane:
          lane-name: tests

  deploy-to-qa:
    macos:
      xcode: *xcode_version
    steps:
      - setup
      - run-fastlane:
          lane-name: deploy_to_qa

  deploy-to-qa-prod:
    macos:
      xcode: *xcode_version
    steps:
      - setup
      - run-fastlane:
          lane-name: deploy_to_qa_prod

# Shared commands that are used by multiple jobs
commands:
  setup:
    description: "Initial setup, shared across all jobs"
    steps:
      - checkout

  run-fastlane:
    description: "Run the fastlane lane passed in as argument"
    parameters:
      lane-name:
        type: string
    steps:
      - run:
          name: "Fastlane << parameters.lane-name >>"
          command: "bundle exec fastlane << parameters.lane-name >>"
          no_output_timeout: 30m

For sake of completeness, I included the whole config.yml file. There is quite a lot to unpack, but the relevant part is to have two workflows, standard-workflow and on-demand-workflow in our case.

If you were to use the previous config.yml file, both workflows would be run for every commit. Considering most of the time, the on-demand-workflow is not needed, this would result in wasted build time, which costs money.

Allow on demand triggering

For security reasons, CircleCI does not allow you to trigger any workflow from their REST API. Only once you defined them in config.yml file, you are able to use them.

First, add another section to your config.yml file, after aliases:

1
2
3
4

parameters:
  api_on_demand_workflow:
    type: boolean
    default: false

With this, you define a boolean variable that is false by default. You will be passing this parameter as true in the API request you will be making. For this reason, I like to make the name of the parameter the same as the name of the workflow and prefix it with api_ to indicate that it will come from an API request.

Then, you need to specify that your on demand workflow should only be run, when this paramter is true, by modifying the workflow section:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

workflows:
  version: 2.1

  # The standard workflow you want to run on every commit
  standard-workflow:
    unless: "<< pipeline.parameters.api_on_demand_workflow >>"
    jobs:
      - tests
      - deploy-to-qa:
        requires:
          - tests

  # The special workflow you want to run on demand
  on-demand-workflow:
    when: "<< pipeline.parameters.api_on_demand_workflow >>"
    jobs:
      - deploy-to-qa-prod

With the when clause, we define that we only want on-demand-workflow to be run, when api_on_demand_workflow is true. However, this is not enough, because our API call would still trigger our standard-workflow. We do not want this, since this likely ran already, when it was triggered by our commit. To avoid this happening, we need the unless clause.

NOTE: If you have multiple on demand workflows, to avoid adding unless clauses for each parameter, I create a api_default_workflow parameter, which is true by default, and pass it as false in all my API requests. This is a temporary solution that I am not too happy with, so if you know any better ways, please let me know.

You final config.yml file should look something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72

# Use CircleCI version 2.1
version: 2.1

aliases:
  # Define a shared alias for what Xcode version you want to use
  - &xcode_version "11.4.1"

parameters:
  api_on_demand_workflow:
    type: boolean
    default: false

workflows:
  version: 2.1

  # The standard workflow you want to run on every commit
  standard-workflow:
    unless: "<< pipeline.parameters.api_on_demand_workflow >>"
    jobs:
      - tests
      - deploy-to-qa:
        requires:
          - tests

  # The special workflow you want to run on demand
  on-demand-workflow:
    when: "<< pipeline.parameters.api_on_demand_workflow >>"
    jobs:
      - deploy-to-qa-prod

jobs:
  tests:
    macos:
      xcode: *xcode_version
    steps:
      - setup
      - run-fastlane:
          lane-name: tests

  deploy-to-qa:
    macos:
      xcode: *xcode_version
    steps:
      - setup
      - run-fastlane:
          lane-name: deploy_to_qa

  deploy-to-qa-prod:
    macos:
      xcode: *xcode_version
    steps:
      - setup
      - run-fastlane:
          lane-name: deploy_to_qa_prod

# Shared commands that are used by multiple jobs
commands:
  setup:
    description: "Initial setup, shared across all jobs"
    steps:
      - checkout

  run-fastlane:
    description: "Run the fastlane lane passed in as argument"
    parameters:
      lane-name:
        type: string
    steps:
      - run:
          name: "Fastlane << parameters.lane-name >>"
          command: "bundle exec fastlane << parameters.lane-name >>"
          no_output_timeout: 30m

Obtaining a Personal API Tokens

To trigger builds through the API, you need to obtain a Personal API Tokens. You can do that in your CircleCI User Settings.

Make sure you save this API token, as you will not be able to see it again.

Triggering your workflow

With all that setup out of the way, it is finally time to test our workflow. For my own use, I have a bash file with a simple cURL request, which looks like this:

1
2
3
4
5
6
7
8

CIRCLECI_TOKEN=<<Your Token>>

curl -u ${CIRCLECI_TOKEN}: -X POST -H "Content-Type: application/json" -d '{
  "branch": "master",
  "parameters": {
      "api_on_demand_workflow": true
    }
}' https://circleci.com/api/v2/project/{project-slug}/pipeline

You can replace master with the name of the branch you want your workflow to run on.

NOTE: Your project slug in the form vcs-slug/org-name/repo-name. For example github/my-organization/my-repo-name. If you have opted into CircleCI’s new UI, when you visit your Pipelines page, you can also see your project slug as the last 3 components of the URL.

If you did everything correctly, hopefully you should see a success message in your console:

1
2
3
4
5
6

{
  "number" : 1080,
  "state" : "pending",
  "id" : "<some unique identifier>",
  "created_at" : "2020-05-04T20:24:02.005Z"
}

Conclusion

In this article, we have seen how you can trigger on demand workflows through CircleCI’s REST API. While not as intuitive as having a dedicated interface for it on their website, like Bitrise for example, at least it is possible to do it.

If you encounter any issues that I did not cover, feel free to get in touch at moc.htikmsofi@gomlb!