ChatOps Checks Manager

Purpose/Goals

• Use "ChatOps" to run scripts that require access to secrets or write privileges on pull requests

• Create and manage pull request checks for specific jobs without requiring them to be triggered by pull_request

• Easily integrate new workflows into this scheme without having to pollute them with tons of extra code in order to work

This project can by no means be called an ideal solution to any of the problems that called it into existence, but it does solve some inconveniences:

• Github only creates automatic code checks for workflows triggered on pull_request or pull_request_target and there is no mechanism to exclude specific jobs from being counted as checks. If you have some jobs that are not actually checks they will clog up the mergebox. This project solves that by only having one job triggered by pull_request_target which in turn creates checks for every job whose id begins with _isCheck_ and updates them after they run via workflow_run.

• When triggered by pull_request from a forked repository, a workflow run has no write access. This is, of course, a good safety measure, but it makes things like building and pushing a test Docker image impossible. You can use pull_request_target instead, but using this to checkout untrusted code is a security risk. Note that this project actually uses pull_request target in checks.yaml, but it does not check out any code from the PR.

There are other solutions to these problems. For instance, you could use webhooks and a 3rd party service to manage your checks, but I wanted to keep everything on Github.

There is also John Bohannon's ok-to-test template, but it has unfortunately been broken by an update from Github on 31 March 2025 which took away the ability to update the status of automatically created checks.

If you just want chatops and don't care about checks you can use peter-evans/slash-command-dispatch directly.

Running this Demo

This is a template repository so you can use it to create your own repository to play around with. To do this you will need to set up an authorization method. I use a GitHub App mainly because it can be managed by an organization and performs actions (replying to a comment, for example) under its own name rather than under my name as a PAT would do. See more on this below.

After you get this out of the way you can create a PR on your new repository. You will immediately see two checks running

• Checks Handler / Setup Checks (pull_request_target)
  • This is running in the context of the base branch, and has access to secrets and write permissions, so it should never check out pull request code.
• Pre-commit / Pre-commit checks (pull_request)
  • This is running in the context of the pull request and has no access to secrets or permissions. Use it as an example for creating workflows like pre commit that don't change anything.
  • Its check status is managed by Github.

After a few seconds, a bunch of new checks will appear. They will all follow the pattern:

Checks Handler / Job Name (pull_request_target)

Clicking on them will bring you to a summary page where, if it is a job triggered by a slash command, you will see a hint as to how to run it; something like "To run this check, comment /fail + any required args on the pull request."

In order to run them, someone with the appropriate permission level ('write' in this case) needs to comment that slash command.

In this example the only command that requires an extra argument is /build. It needs the SHA of the PR head commit like /build sha=d3d8117. This is to make sure that, if a new commit gets pushed right as the maintainer is writing the slash command comment, the job will be skipped and will not run the new, potentially malicious code.

I think the rest should be fairly obvious from reading the code and watching it run.

Usage

I currently do not have the time to document all aspects of usage, but will be glad to answer any questions in issues and will be even happier to merge pull requests improving the documentation.

Authentication

peter-evans/slash-command-dispatch Needs some sort of authentication to send repository_dispatch events. Choose one of the following:

• Personal Access Token (PAT)
• OAuth app token. ( I only know of this from here and have not tested it)
• A GitHub App. This is what I use

Creating a GitHub App

• Go to Settings > Developer Settings > GitHub Apps, and select New GitHub App.

• Enter a name for your app (this needs to be unique across GitHub), and fill in the required URL fields. You can fill in these URLs with fake values - they do not need to resolve, so you can use: Homepage URL = http://example.com

• You can ignore the 'Callback URL' field, and untick 'Webook' > Active

• Under Repository Permissions, set:

  • Checks = 'Read and write' (If you want to have the custom checks in their own section)
  • Contents = 'Read and write'
  • Issues = 'Read and write' (If you need to post comments under your app's name)
  • Metadata = 'Read-only'

• Under Organization Permissions, set:

  • Members = 'Read-only'

• Additional permissions may be required depending on what else you are doing with it.

• Click Create GitHub app

• Click Generate Private key (this will be downloaded to your computer), and take a note of the App ID field (A 7 digit number)

• Install the GitHub App into your user or organization, by clicking Install under the 'Install App' tab, and choose whether you want to give the app access to all of your user's / org's repositories, or just specific ones

• Go to the organization or repository that you want to use chatops-checks-manager with, and then Settings > Secrets and variables > Actions and create two new entries:

  • A variable called APP_ID with the value for the App ID field that you noted earlier
  • A secret called PRIVATE_KEY, copying and pasting in the full contents of the Private Key file that you generated and downloaded earlier. Make sure it does not have a whitespace at the end.

Credits

• ok-to-test was the original inspiration and I also copied half of this README from there.
• peter-evans/slash-command-dispatch
• Gemini Code Assist in VS Code. Yeah, I vibe coded this :)

Contributing

Pull requests are welcome!

Wishlist/Roadmap

• Add the ability to manage checks for workflows triggered by pull_request on a per job basis. This would probably be accomplished by using repository-dispatch in checks.yaml

• Add a mechanism to get the args required for running any given slash command and past it in the check summary to provide a clue.

• The same kind of mechanism for checks not triggered by slash command, although I don't really see a use case. If you have an actual need for this please let me know!