claim() reserves the whole batch before dispatch, but the cancellation
path returned without releasing what it had not yet started, leaving
those items in the in-flight set permanently — no later drain could
claim them again.
Harmless until the batch grew past the pool size; now a cancel strands
up to a full batch. The e2e harness cancels mid-drain after every
acquire, so it surfaced there first: one spec timed out waiting for an
item that had been claimed and abandoned, and the suite went from 59s
to 87s on CI.
A first backfill enqueues artists before albums at a single priority, so
the drain took them in that order. Artists resolve through a
rate-limited agent, and gate() waits for its permit while holding a
worker slot, so the whole pool sat asleep in the limiter with every
album queued behind it.
Measured on a 96k-track library (29,115 artists to 6,949 albums, 4:1):
zero albums resolved in seven minutes, and roughly 3.3 hours before the
first album cover would have appeared. Splitting the drain gives each
class its own slots: albums now finish in under eight minutes while
artists trickle at the same 2/s they were always limited to.
The two budgets are carved out of MaxOpenConns so a second pool cannot
take connections the scanner and the UI need. Dequeue filters by kind,
and the drain index leads with item_kind so each pool seeks to its own
work instead of scanning past the other's backlog.
The pool was fed from a batch sized to the pool itself (2x concurrency)
with a WaitGroup barrier before the next dequeue, so one item burning
its external timeout idled every other slot until it finished. The
legacy cache warmer had no such barrier: it streamed through a pipeline
of 4.
Dequeuing well past the pool keeps the slots fed for the whole pass at
no extra cost, since DequeueBatch does not mark rows taken and was
already one query per pass. Acquiring a slot now also observes
cancellation, so a larger batch cannot delay shutdown.
Warming the resize cache re-read the two rows and the file the
acquisition had just written, so every acquired image cost two extra
queries and a second full read of a file whose bytes were still in
memory. processItem now hands back what it persisted and precache warms
from that, under the same cache key the serving path computes.
Resolving the admin user also moves behind the empty-queue check: it is
needed only to resolve private playlists, so an idle server no longer
runs a user lookup on every poll.
Exhausting the 12h budget called writeAbsent unconditionally, so an
entity whose art was already resolved and serving lost it to a long
upstream outage: the hash went empty, clients fell back to the
placeholder, and the now-unreferenced bytes were freed by the next
prune even though nothing about the image had changed.
Exhaustion means the source stayed unreachable, not that the cover
disappeared, so absent is now recorded only when there is nothing to
keep.
Retry backoff now starts at 5s (was 15s) so a transient failure recovers on
essentially the next drain, and jitter widens to ±40% so a wave of correlated
failures doesn't re-clump into one poll.
Add a 12h give-up budget measured from enqueued_at: once the next backoff would
land past it, the worker stops retrying instead of grinding at a cap forever. A
bare failure settles absent (handed to the 24h stale-absent sweep, and still
recoverable on a page view); a found-stale keeps its already-served art. The
budget bounds the tail, so the separate 48h backoffCap is removed.
The exponential retry (base × 4^attempts, cap 48h) started at 5 minutes, so a
single transient failure — a timeout under load, an external blip — parked a
cover for 5 minutes even though a retry seconds later would have resolved it.
Start at 15s instead: transient failures recover almost immediately (15s → 1m
→ 4m → 16m …), while persistent failures still escalate to the 48h cap (now at
the 8th attempt instead of the 5th).
Entity-level artwork queries now take a typed model.Kind instead of a bare
prefix string. GetItemArtwork, DeleteForItem(s), GetInfoForItems,
EnqueueStaleAbsent, hydrateItemImages, enqueueBackfillKind and artwork.Refresh
convert to the prefix string only at the two real boundaries: the SQL
item_kind column (kind.Prefix() inside each repo) and external string inputs
(a new model.ParseKind for the nativeapi URL param, which also validates it).
The Backfill/stale-absent kind slices, the resolve.go dispatch switch, and the
kind→resource / kind→table lookup maps now use the Kind vars directly. The
queue lifecycle methods (MarkFailed/Delete*) keep string kinds — they operate
on a dequeued item's raw ItemKind column, which stays a string field, always
populated via kind.Prefix().
Removes every bare "al"/"ar"/… prefix literal from non-test code (27 -> 0);
behavior is unchanged.
Two worker/enqueue fixes from review:
- playlistRepository.Put assigned the generated id to the caller's Playlist but passed
the stale copy (empty id) to refreshCounters, enqueueing a pl|"" row the worker
failed until the daily dangling purge while the real playlist went unresolved. Set
the id on the copy before enqueueing.
- The drain refresh batch only included found/foundStale, so a cover removed by a scan
(found -> absent) never notified clients, leaving the old immutable image displayed.
Broadcast absent outcomes too; precache still only warms found/foundStale.
The artwork worker's drain speed was governed by two hidden Dev flags,
DevArtworkWorkerConcurrency and DevArtworkExternalRPS, both defaulting to 2.
On a large library's one-time backfill the external rate limiter is the real
ceiling: every art-less item waits on it before the (rate-limited) external
lookup, so the drain crawls at ~RPS items/sec while local-art items are
unaffected.
Promote both to documented, supported options: ArtworkWorkerConcurrency
(default 4) sets local-resolution parallelism, ArtworkExternalMaxRPS
(default 2, 0 = unlimited) caps external-agent lookups to stay polite to
Last.fm/Deezer/etc. Operators can now trade first-backfill speed against
external-API rate limits. The old Dev names still map for backward compat.
With CoverArtPriority="external,cover.jpg", a transient external failure
followed by a folder hit dropped the external error: the worker recorded
found and deleted the queue row, so the configured higher-priority external
art was never retried. Carry extError onto the fallback resolution and add
an outcomeFoundStale that persists+serves the art but reschedules via
MarkFailed, giving the external source another chance. When external later
answers definitively-not-found, the hit is not stale and the row is deleted.
Queue rows for deleted entities failed forever (Get -> ErrNotFound -> failed -> capped retries, unbounded). Add ArtworkQueueRepository.PurgeDangling, called from Prune next to the item_artwork purge. Separately, the found/absent path unconditionally deleted the dequeued row, erasing a concurrent scan re-enqueue; switch to DeleteIfUnchanged, which deletes only while retry_at still matches the dequeued value (verified retry_at is the column an Enqueue upsert resets).
