Original article in Japanese: Renovate の大量の Pull Request を処理する技術

In this post, I’d like to introduce techniques for handling a large number of pull requests from Renovate in a Terraform Monorepo.

Background

We manage a Terraform Monorepo, and recently we’ve migrated its CI from AWS CodeBuild to GitHub Actions and tfaction.

2022-02-25 Migrate Terraform CI from AWS CodeBuild to GitHub Actions

We have about 400 working directories (Terraform States), and the following tool versions are managed in each working directory.

• Terraform
• Terraform Provider
• tflint
• tflint plugin
• tfsec
• etc

If a single package is used in multiple services in Monorepo, by default, Renovate updates them in a single pull request. We use additionalBranchPrefix to separate pull requests per working directory.

e.g.

{
  "additionalBranchPrefix": "{{packageFileDir}}-",
  "commitMessageSuffix": "({{packageFileDir}})",
  "matchManagers": [
    "terraform",
    "regex"
  ]
}

This way, when a tool is updated, nearly 400 pull requests need to be merged. Reviewing such a large number of pull requests one by one by humans is difficult and not worth the effort. Therefore, if CI is successful and the result of terraform plan is no change, it is desirable to merge automatically. If the number of pull requests that can be merged a day is too small, we wouldn’t be able to fully process pull requests and tools wouldn’t be updated properly.

Solution

To handle a large number of pull requests from Renovate automatically, we did the following actions.

1. Enable automerge
2. Enable platformAutomerge
3. Set prHourlyLimit to 0
4. Set prConcurrentLimit to 5
5. Limit branchConcurrentLimit too
6. Update the feature branch and enable automerge automatically when the automerge is disabled due to the update of base branch
7. Close the pull request and delete the feature branch immediately when CI fails
8. Skip terraform plan and apply for updates other than Terraform and Terraform Provider
9. Install not only Renovate Approve but also Renovate Approve 2 to make pull requests certainly approved
10. Set prPriority to prevent some tools from blocking other tools’ update
11. Replace GITHUB_TOKEN to GitHub App’s token to prevent API rate limiting

1. Enable automerge

If you enable automerge, Renovate will merge pull requests automatically. If one approval is required to merge pull requests, you can use Renovate Approve.

However, it is known that it takes a little long time to merge pull requests with automerge. It can take several hours. So, you can enable platformAutomerge.

2. Enable platformAutomerge

When platformAutomerge is enabled, pull requests are merged as soon as the conditions are met by GitHub Automerge feature.

Notes on GitHub Automerge

Please use GitHub Automerge carefully, otherwise your pull requests will be merged even if CI fails.

• You have to enable Allow auto-merge in the repository setting
• The base branch must be protected by Branch Protection Rule
  • You must select the status checks in Status checks that are required least at one, otherwise Automerge cannot be enabled

Be aware that the pull request will be merged even if the checks other than the ones you checked in Status checks that are required fail. If a GitHub Actions job is skipped by if, it will be merged.

We run GitHub Actions’ multiple jobs in parallel by build matrix, but it is difficult to add those jobs to Status checks that are required because executed jobs are changed dynamically. So we add a job which depends on the build matrix, and add it to Status checks that are required.

There is still a problem that if other workflows fail the pull request would be merged, but we tolerate this because it rarely happens and we can fix it if it happens.

3. Set prHourlyLimit to 0

Renovate has several limits that restrict the creation of pull requests. Note that even if they are unlimited by default, it may be limited by the preset config:base.

config	default	config:base
prHourlyLimit	0	2
prConcurrentLimit	0	10
branchConcurrentLimit	prConcurrentLimit

prHourlyLimit is limited to 2 by config:base, which means that only 2 pull requests will be created per hour. So, explicitly set it to 0 so that an unlimited number of pull requests can be created.

4. Set prConcurrentLimit to 5

Renovate tries to create as many pull requests as possible within the above limit. When you run terraform plan and apply in Terraform CI, probably CI would fail due to API rate limiting if you run a lot of them at the same time. Also, GitHub Automerge may be automatically disabled when the base branch is updated.

For this reason, we set prConcurrentLimit to 5.

5. Limit branchConcurrentLimit too

branchConcurrentLimit is a limit based on the number of branches. I thought we didn’t have to limit pull requests by the number of branches, so I set it to 0 at first, but that was a mistake. It seems that branches are created even if no pull requests are created, so more than 1000 branches were created unnecessarily. Since branchConcurrentLimit is the same as prConcurrentLimit by default, we explicitly set only prConcurrentLimit and not branchConcurrentLimit.

6. Update the feature branch and enable automerge automatically when the automerge is disabled due to the update of base branch

GitHub Automerge may be automatically disabled when the base branch is updated.

To merge these pull requests automatically, you can update feature branches and re-enable automerge automatically by GitHub Actions.

https://github.com/suzuki-shunsuke/reenable-automerge-action

7. Close the pull request and delete the feature branch immediately when CI fails

Since we have set prConcurrentLimit and branchConcurrentLimit, leaving Renovate pull requests open will limit the number of new pull requests that can be created. Therefore, we decided to close pull requests that could not be automerged and delete feature branches automatically.

https://github.com/suzuki-shunsuke/renovate-autoclose-action

You can search closed pull requests with simple query like is:pr is:unmerged author:app/renovate, and can also be found in Renovate’s Dependency Dashboard.

8. Skip terraform plan and apply for updates other than Terraform and Terraform Provider

With tfaction, CI would fail if the result of terraform plan of pull request by Renovate is not No Change to prevent dangerous changes from being applied by terraform apply. However, sometimes tools such as tfsec and tflint couldn’t be updated due to this failure.

tfsec and tflint are not related to terraform plan and apply, so you don’t have to run terraform plan and apply to update them.

Since tfaction v0.4.9, tfaction supports skipping terraform plan and apply in Renovate pull requests, so we’re using the feature.

This also speeds up CI and prevents API rate limiting.

9. Install not only Renovate Approve but also Renovate Approve 2 to prevent approve omissions

We don’t know the reason, but sometimes Renovate approve does not approve pull requests expectedly. So we also installed Renovate Approve 2 to prevent the omission of approve. This app is supposed to be used when two approval are needed, but we think it can also be used to prevent approval leaks. So far, we haven’t had any approval leaks after we installed Renovate Approve 2.

10. Adjust prPriority properly

Tools like Terraform and AWS Provider may disturb other tools’ updates for a long time, because they are frequently updated. If you want to prioritize other tools updates, you can adjust the prPriority.

11. Replace GITHUB_TOKEN to GitHub App’s token to prevent API rate limiting

tfaction takes GitHub Access Token as input. By default secrets.GITHUB_TOKEN is used, but if the number of builds’ an hour increases, API rate limiting may occur. So we switched to a GitHub App token which has a less strict rate limit than secrets.GITHUB_TOKEN. About the rate limit, please see the document.

• https://docs.github.com/en/rest/overview/resources-in-the-rest-api#requests-from-github-actions
• https://docs.github.com/en/developers/apps/building-github-apps/rate-limits-for-github-apps

To switch, you need to modify GitHub App permissions (issues: read is required). Furthermore, you also need to switch GitHub Access Token for github-comment hide, because github-comment hide only hides comments from the same user.

Conclusion

As a result of the above actions, we are now able to create and merge about 500 pull requests a day into a single repository. This number still has room for improvement (we think it could be up to 700 or so), but it is still sufficient for the current situation. We used to check and respond to open pull requests from time to time, but by automating tasks as much as possible, we only have to deal with those that really need to be dealt with by humans, and this has reduced our workload.