A job that deduplicated against itself, and went quiet forever
2026-09-02 · Vorluno
The report was four words: the agent went quiet. A contact wrote asking about advertising, got an answer, and a minute later asked about pricing. Nothing came back. No error, no escalation, no failed job in the queue, no alert. From every dashboard we had, the system was healthy.
The cause was a scheduled job that reschedules itself. To pace replies, a turn can defer its own continuation — enqueue the next step with a delay. The job was enqueued with a deterministic identifier derived from the conversation, which is the standard trick for making an enqueue idempotent: if the same job is already scheduled, the queue drops the duplicate instead of running the work twice. That is exactly what you want, right up to the moment the job doing the enqueuing *is* the job whose identifier it is reusing. The queue saw an identifier it already knew, dropped the new one as a duplicate, and the currently running job finished. Nothing was scheduled. Nothing failed. The conversation had no next step, forever.
The detail that makes this hard to reproduce is worth writing down, because it cost time: the deduplication only bites when the existing job is *active*, not merely delayed. Reproduce it with a delayed job sitting in the queue and the second enqueue is accepted, everything works, and you conclude the bug is somewhere else. You have to reproduce it from inside a running job to see it at all — the state that makes it fail is the state the naive test does not create.
The fix is unremarkable; the class of bug is not. Deduplication by identifier is a correctness feature that turns into a silent-drop feature the moment the producer and the deduplicated thing share a name. It fails closed in the worst possible way: no error surface anywhere, because from the queue's point of view nothing went wrong — it did precisely what it was configured to do.
What we changed beyond the fix was the monitoring assumption. We had alerts for failed jobs and none for conversations that simply stop having a next step, because "nothing happened" was not a state anything watched. A queue can only tell you about work it is holding. It cannot tell you about the work nobody asked it to hold.