Worker.drain waits for every item in a batch before it dequeues the next
one, so a single item that never returns stops the whole pool. This is what
happened in discussion #6232: filepath.Rel looped forever on a UNC share
root, and all artist artwork stopped. Go can't stop a goroutine that
spins, so the only way to keep the pool going is to stop waiting.
process now runs acquire in its own goroutine and waits at most
itemTimeout (5 minutes). On timeout it logs an error, adds a trace step,
and settles the row as failed, so the normal backoff and give-up budget
apply. A per-worker busy set keeps one item from running twice at the
same time: if the row comes due again while the old run is still stuck,
it is settled as failed without starting a second run. This keeps the
existing guarantee that the drain resolves each item serially.
Shutdown still waits for in-flight items, but now at most itemTimeout
for one that is stuck.
d513b1ea48