Is this ticket ready for an AI coding agent?
Last updated
Why tickets fail agents
A ticket is a short written request for a change, usually a GitHub issue. When a person picks one up, the ticket is only the start. They ask in chat what "make it faster" means, or check with the product owner which of two designs to build. The ticket can be rough because a conversation fills the gaps.
An unattended agent has no conversation. "Unattended" means it runs with nobody watching: it reads the ticket and the code, makes a change, and hands back a pull request, which is a proposed change for someone to review. If the ticket leaves a gap, the agent fills it with a guess. It doesn't stop and say it's unsure. The usual result is a tidy, well-tested pull request for something nobody asked for.
That's worse than no pull request. You pay for the run, and then a reviewer has to spend time working out that the change is wrong. So the most useful thing you can do before an agent starts is to answer a simple question: could someone who can't ask questions finish this ticket?
The five checks
BacklogBeast breaks that question into five narrower ones. Each is a separate way a ticket defeats an agent that works alone. Each is answered yes or no, and a ticket is ready only if all five are yes.
| Check | The question | Falls short when… |
|---|---|---|
| Checkable done state | Does it say what should be true when the work is finished, clearly enough that a reviewer could check it? | It says "improve", "look into" or "make better" and stops there. |
| Says where | Can you tell which part of the product or code is involved: a page, feature, file, endpoint or error message? | Nothing points to a place. |
| No open decision | Can the work be done without first making a product, design or policy call that the ticket leaves open? | It lists options with none chosen, or says "we need to decide". |
| Fits one pull request | Is it a single change one reviewable pull request can deliver? | It's a project, an epic or several unrelated changes. |
| Code-only work | Can it be done and checked entirely by changing files in the repository? | It needs production access, live data, a manual step, or can only be checked in a browser or after a deploy. |
Two details make these checks fair to the ticket writer.
- How to build it is not a decision. "Should this be a new function or part of the existing one?" is the agent's job. The "no open decision" check only asks whether it's settled what the product should do.
- Later answers count. If the ticket body says "needs a decision" and the repository's owner answers in a comment, that settles it. The check reads recent comments as well as the body.
Why five questions and not one "is this well written?" score? A single score blends different problems into one number. A ticket can be beautifully clear about something that needs production access, and it would score well. Separate checks can't hide a failure behind strengths elsewhere, and they let the note back to the author name the exact gap.
- Vague ticket, "Improve checkout": checkable done state, no. Says where, no. No open decision, no. Fits one pull request, no. Code-only work, yes.
- Result: sent back, with four things missing.
- Ready ticket, "Trim spaces in discount codes": all five checks, yes.
- Result: ready, so it can go to a run.
Technical detail: how the checks are scored
Each check is a yes-or-no question put to a small judgment model, which returns a probability that the answer is yes. All five questions go in one request, and the model sees the ticket's title, body, labels and up to the 15 most recent comments, each trimmed to a fixed length. Comments by the repository's owner are marked as such.
The model supplies probabilities; the rule stays in code. Every check must reach the threshold (0.5 by default) on its own. The checks are never averaged, so a strong score on four can't carry a fail on the fifth.
The questions were reworked after we checked them against 30 real tickets we'd labelled by hand. The earlier version treated "which way to implement this" as an open decision and sent back two good tickets, and it passed a ticket with three competing designs and none chosen. Raising the threshold wouldn't have fixed either: it would have bounced a ticket that turned into a clean pull request and still passed the undecided one. Changing what the questions ask did fix them. Across those 30 tickets, the older checks let 8 into runs that were then discarded, ran out of steps, or repeated work that had already shipped. The reworked ones let 2 through.
If the check can't run at all, for example because the judgment service returns an error or a missing answer, nothing is started for that repository and no ticket is marked. The check fails closed: when in doubt, it doesn't spend.
Check before you spend
A run costs real money: a virtual machine in your cloud account, and model usage billed by your AI model provider. It also costs time. In our own runs, preparing the machine (installing tools, fetching the code's dependencies and building it) takes about four minutes before the model even starts, and a full run takes about 15 minutes.
Reading a ticket's text costs almost nothing by comparison. So the check happens first, before any machine starts. A ticket that isn't ready never reaches the part that costs money.
- A person adds the ready label to a ticket.
- The readiness check runs its five yes/no checks. Nothing has been spent yet.
- If every check passes, a run starts in your cloud account. This is where spending starts.
- The run ends in a draft pull request for review.
- If any check fails, the ticket is sent back with a comment naming what is missing.
- A person fixes the ticket and re-adds the ready label, and it is checked again.
The cost of getting this wrong isn't only the run. A run on an unclear ticket usually doesn't fail loudly. It produces a pull request that looks finished. Someone then reads it, compares it with what they actually wanted, and throws it away. Sending the ticket back costs the author a few minutes of writing, which they would have spent explaining it to a person anyway.
- Run it anyway: about 4 minutes to prepare the machine (tools, dependencies, build).
- Then about 11 minutes of model work, paid to your AI model provider, up to the per-run cap.
- Result: a pull request for the wrong thing, or none, plus a reviewer’s time to find that out.
- Check first: one short request reads the ticket text. The ticket is sent back with what is missing, and nothing starts in your account.
Technical detail: the numbers behind the timeline
From six measured runs on our own test repository: installing the toolchain took 55–62 seconds (one outlier at 171), and the repository's own setup (dependencies and build) took 161–168 seconds (one outlier at 427). Model spend was $0.19–$2.35 per run. Runs finish in about 15 minutes.
Each run also has a hard dollar cap. In our configuration, a run needs $5 of monthly budget headroom to start, and the model session itself is capped at 80% of that, $4, because the cap is checked after each model response and can be overshot by one response. These are our settings, not universal figures; see what an AI coding agent costs per ticket for how the caps work.
When a ticket is sent back
"Sent back" means three things happen on the GitHub issue, and nothing else:
- A comment is added that names each check the ticket failed, in plain words.
- A "not ready" label is added (label names are set per repository; ours is
needs-info). While it's there, the ticket isn't checked again on every pass. - The ready label is removed, so the ticket leaves the agent's queue.
Here is what the comment looks like for a ticket that failed two checks:
Not picked up: this ticket needs more detail
An unattended agent works only from the ticket and cannot ask questions, so it
checks five things before spending a run. This ticket fell short on:
- A checkable done state: what should be true, observably, when this is finished.
- Where the change belongs: the page, feature, component, file, endpoint or error involved.
Labelled `needs-info` and `agent-ready` removed. Once the ticket covers the
above, remove `needs-info` and add `agent-ready` back.To fix it, edit the ticket (or add a comment) so it answers what's listed, then swap the labels back. Putting it back in the queue is always a person's decision. The agent never re-labels a ticket itself, so nobody gets surprised by a run on something they meant to revisit.
If the note says there's an open decision, make the call in the ticket before re-labelling. A comment from the repository owner such as "Go with option B" counts. Leaving the choice to the agent is how you get a pull request for option A.
Technical detail: scores, and which tickets get checked
Under the note, a collapsed table lists each check's probability of "yes" and marks the ones below the threshold. That helps when a ticket is borderline: a check that failed at 0.45 needs a small clarification, one at 0.05 needs rewriting.
Each time BacklogBeast looks for work, it checks only a few of a repository's ready-labelled tickets (three in our configuration), most recently updated first. Tickets that pass are the only ones handed to the run, so it never works on a ticket that wasn't checked. Separately, a ticket that already looks fixed (a recent commit on the main branch names it) is skipped quietly, with no comment. If someone starts a run by hand and names a specific ticket, the check is skipped: that's an explicit human decision.
A vague ticket, rewritten
Here's an example of a ticket that would be sent back:
Improve checkout
Checkout is confusing and people complain about discount codes.
Maybe we should redo the layout, or move the code field higher?
Also the confirmation email looks off.It fails four checks. "Improve" and "confusing" give no checkable result. "Checkout" is a whole flow, not a place. Two layout options are offered and neither is chosen. And it bundles at least three changes: a layout, a discount bug and an email. It does pass the last check: everything it mentions lives in the code.
Here's one part of it, rewritten as a ready ticket:
Trim spaces in discount codes
Where: checkout page, discount code field (src/checkout/discount.ts).
Now: " SAVE10 " (with spaces, e.g. pasted from an email) is rejected
as "Invalid code".
Expected: leading and trailing spaces are removed before the code is
checked, so " SAVE10 " applies SAVE10. Add a unit test for it.
Out of scope: the layout of the form, and the confirmation email
(separate tickets).Each line answers one check. "Expected" gives a result a reviewer can test. "Where" points to the place. There's nothing left to decide about how the product should behave. "Out of scope" keeps it to one pull request, and the layout and email become their own tickets. A unit test, a small automated check that lives in the repository, proves it without a browser or live data.
A good habit: write the "Expected" line first. If you can't, the ticket isn't ready for a person either.
Ready, but not for an agent
Some tickets are perfectly clear and still shouldn't go to an unattended run, because the run would be thrown away at the end. BacklogBeast asks three more questions of a ticket that passes. These don't send it back, because nothing is wrong with the ticket. They hand it to a person instead, with a comment saying why.
- Too large, or builds a new page or subsystem? If yes, hand it to a person.
- Names a protected file the agent may not change? If yes, hand it to a person.
- Must the fix change an existing test, and has no one approved that? If yes, hand it to a person.
- If all three are no, the ticket goes to a run.
- Too large. A change that cuts across much of the code, or builds a new page or subsystem from scratch, doesn't finish in one unattended session. The comment suggests splitting it into smaller tickets.
- Needs a protected file. Each repository can list files the agent is never allowed to change, such as its build settings or automated workflows. A branch that touches one is discarded after the run. If the ticket names one, it goes to a person. If the change doesn't actually need that file, say so in the ticket and re-label it. Letting an AI agent open pull requests safely covers protected files.
- Changes what an existing test checks. If the ticket reverses behaviour that a test currently pins down, a correct fix has to edit that test. The agent may not do that on its own, because "make the test agree with my change" is also how a real bug slips through. A person can approve it with a separate label, and every removed test line is then listed in the draft pull request for review.
Technical detail: how the hand-back questions were tuned
The size and "builds something new" questions are answered by the same judgment model, in the same request as the five checks. On our 30 hand-labelled tickets, the two tickets that built something new scored 0.87 and 0.92, and nothing else scored above 0.41. The "changes an existing test" question caught 9 of the 13 tickets that needed it, with no false positives, at a 0.5 threshold.
The protected-file check is plain code, not the model: it looks for paths in the ticket's text that match the repository's protected list. A model question for it ranked the tickets worse; the pattern match caught every real fix on our test repository that touched a protected file, and nothing else.
In practice with BacklogBeast
You hand a ticket over by adding a label to a GitHub issue. People who only write tickets never need a BacklogBeast seat. Before anything starts in your cloud account, BacklogBeast runs the five checks and the three hand-back questions. The check runs on BacklogBeast's side at no charge to you. It reads the ticket's title, body, labels and recent comments, and keeps only the verdict.
- A ready ticket goes to a run, which ends in a draft pull request for your team to review. BacklogBeast never merges and never deploys.
- An unclear ticket is sent back with a comment naming what's missing, and nothing is spent in your account.
- A clear ticket that a run couldn't finish is handed to a person, also for free.
- Re-queuing is always a person adding the label back.
The practical effect is that your agent's budget goes on tickets it can actually finish, and the people who write tickets learn quickly what a good one looks like. If you'd like to try it on your own repository, join the waitlist.