Stanki Guild

Guide · Jira

Our nightly tests went red on the first run. The app was right.

Approval Flow for Jira 3.0.0 is the production version (30 October 2026). Last updated 1 November 2026. On the Atlassian Marketplace.

By Bartosz Stankiewicz, founder, Stanki Guild.

On 1 November we started running the full test suite every night, twice: once on the code of the next release, and once on the exact version customers have installed. The second one is the one that matters to you, and it's easy to skip, because the installed version "already passed" when it shipped.

It went red on the first run. One test out of 1,256 in Approval Flow.

The test checks something in the AI agent approval gate: when an agent asks a human to approve an action, the person the agent works for can never approve it themselves. To build that case the test creates a request with an expiry time. The expiry time was written into the test as a fixed date, 31 October, 11:00 UTC. On 1 November the request in the test had expired, the app refused the approval ("the request expired, the agent has to ask again"), and the test, which expected a different answer, failed.

So the app did the right thing. The test was the thing that aged. We had already fixed the test in the next release a day earlier, with a fixed clock instead of the real one. The installed version still carries the old copy, and will until the next release ships.

A red result on day one is still better news than a green one that checks the wrong version.

What we did about it

#

The tempting move is to delete the test, or to switch off the alarm. Both are wrong. A deleted test is gone for the next person, and an alarm that rings every night with the same known cause teaches everyone to ignore it, including on the night it rings for something new.

So we keep a short file of known failures: which app, which version, how many tests, and why. The nightly run still runs that test and still reports it. It just doesn't raise the alarm while the count matches the explanation. If a second test fails on that version, the alarm goes off. When the next release ships, the entry stops matching and goes away.

Two other things fell out of that first night. One release tag existed only on a laptop, not in the repository the server reads, so the server tested an older version than the one installed. And the servers run a newer Node.js than the laptops. Neither was a bug for customers, but both would have made the nightly result quietly wrong.

I don't know yet how often this will catch something real. One night is one night.

Do you test the version your users actually have, or only the branch you're working on?

#