Features That GitHub Actions Should Have

Lately I haven’t been feeling inspired to write anything tech-related, in fact I’ve had a hard time finding motivation to do any software stuff in my free time. Maybe it’s a temporary slump, but more likely it’s a side effect of having a full-time tech job and more non-technical hobbies. I’m counting it as a net win, though—life needs balance. In any case, here’s something; I thought I’d serialize some of my recurring gripes since I spend a lot of time with GitHub Actions at work.

GitHub Actions, since its proper release¹ about a year and a half ago, has quickly become the de facto CI system for open source Git repositories. It owes its success to GitHub being the de facto software development platform—as well as Travis CI’s recent self-destruction—and while it’s powerful enough for most automation, it’s downright clunky for some tasks. For GHA to become universally useful, there are a few features that I think it needs, and that I can’t believe it doesn’t already have.

On the JuliaLang Slack, I’ve complained dozens of times and fielded dozens more complaints from others about GHA’s lack of allowed-to-fail jobs. If you’re testing a library of any sort, then you probably want to test on multiple versions of your runtime or compiler, and for this you’d use a matrix. Here’s a fairly typical (but incomplete) example that runs the same job three times with different language versions:

jobs:
  tests:
    strategy:
      matrix:
        version:
          - '1'
          - '2'
          - 'nightly'
    steps:
      - uses: FooLang/setup-foo@v1
        with:
          foo-version: ${{ matrix.version }}

The nightly version is unstable and we don’t mind if it fails because it’s not necessarily our fault, so we’d like to do something like this:

jobs:
  tests:
    strategy: 
      allowed-failure: ${{ matrix.version == 'nightly' }}

In this case, a failure on the nightly job and passes on the others would result in a workflow pass. Ideally, the status on the allowed failure would be yellow to indicate that it failed but was allowed to do so. But alas, such an allowed-failure setting does not exist on GHA.

There’s continue-on-error, but it’s not the same thing. Jobs that fail with this setting enabled are reported as successes, making it easy to miss unless you specifically go looking for it on every run. It’s also never worked for me at the job level as documented. Instead, I have to put it only on the step running tests, which means that following steps will always run even if the tests fail to run all the way through. If I submit code coverage to something like Codecov in a later step, it yells at me about my 0% coverage until the passing jobs catch up.

To complain about this, see this issue and this forum post.

I probably complain about this one almost as much as the allowed failures. Jobs can fail for all kinds of reasons outside of your own code—maybe a service you depend on was down, or maybe GitHub itself had an error. In any case, it’s useful to be able to restart failed jobs without having to push another commit. GHA actually does support this, but not without also restarting all of the jobs in the workflow. So if you’re running the same unit tests on twenty different compiler versions, you have to re-run the nineteen jobs that passed if you want to retry the one that failed.

To complain about this, see this issue and this forum post.

Other, more mature CI platforms mostly support enabling some form of remote access on job failures. This is incredibly useful to debug CI-specific issues. GHA has no such feature at this time, meaning I spend a lot of debugging time pushing countless commits to run jobs over and over again with tiny configuration changes. There’s a third-party Action to run tmate, but it still requires me to push changes to my workflow to start debugging, and to push another commit removing those changes once I’m done.

To complain about this, see this forum post.

Oftentimes, you want to save some generated files to be downloaded at a later date, such as a compiled executable or logs. Maybe you want to save your release binary on pushes to your main branch, and then download and publish it on tag pushes. Maybe you want to save output from previous runs and compare it as some kind of regression test. The possibilities are endless, but it’s all mostly impossible with GHA, because you can’t download artifacts from any workflow run but the current one². That means that you can pass artifacts between steps of a job and jobs of a workflow, but you can’t use yesterday’s artifacts in today’s workflow run.

There are a couple of alternatives in the meantime. You can, of course, roll your own persistent artifact system with an object store like S3, but you shouldn’t have to! I’ve not tested this, but you could also probably abuse actions/cache for the same purpose.

To complain about this, see this forum post.

GHA prides itself on being a full-featured, all-in-one automation platform, but I don’t think it deserves that title until it supports interactions between multiple repositories. There are a million reasons you might want to respond to events in other repositories in some kind of PubSub fashion. For example, you might want to subscribe to new releases of your dependencies to trigger upgrades. Having this feature would have saved me loads of work developing a convoluted, finnicky approximation of it with issue comments.

To complain about this, see this forum post.

All that being said, I still really like GitHub Actions, and I look forward to seeing its continued development. Hopefully that development will include some of the features described above! Thanks for reading.