The artwork worker's drain speed was governed by two hidden Dev flags,
DevArtworkWorkerConcurrency and DevArtworkExternalRPS, both defaulting to 2.
On a large library's one-time backfill the external rate limiter is the real
ceiling: every art-less item waits on it before the (rate-limited) external
lookup, so the drain crawls at ~RPS items/sec while local-art items are
unaffected.
Promote both to documented, supported options: ArtworkWorkerConcurrency
(default 4) sets local-resolution parallelism, ArtworkExternalMaxRPS
(default 2, 0 = unlimited) caps external-agent lookups to stay polite to
Last.fm/Deezer/etc. Operators can now trade first-backfill speed against
external-API rate limits. The old Dev names still map for backward compat.
Backfill enumerated every album, artist, playlist and radio via GetAll
and mapped out just the ID. GetAll materializes full entities (library
joins, participant/stats/tags JSON, annotation, artwork hydration), so on
a large library it loaded tens of thousands of heavy structs only to read
one field each — spiking transient RSS to ~1GB during the one-time
upgrade backfill, a memory risk on small NAS/Pi hardware.
Add GetAllIDs to the album, artist, playlist and radio repositories: it
reuses each repo's base row-set filter (library visibility, artist
content join, playlist userFilter) but projects only id, skipping the
heavy columns and post-processing. A per-repo parity test asserts
GetAllIDs returns exactly the same id set as GetAll.
Verified on a 727MB / 29k-artist production DB copy: peak RSS during
backfill dropped from ~1012MB to ~89MB, file descriptors flat, same
36,138 items enqueued.
decodeTile ran image.Decode on every sampled album's resolved bytes
before processItem's maxImageBytes/maxImagePixels guards applied,
letting an oversized or decompression-bomb tile fully decode
unbounded. Enforce both caps inside decodeTile itself.
TestArtworkBackoffSchedule and TestArtworkWorkerRunNoLeak needed no real
*testing.T (no synctest), so move them into worker_test.go as Ginkgo
specs. TestArtworkBreakerHalfOpen stays plain since testing/synctest
requires a real *testing.T, matching core/scrobbler's precedent.
goleak now ignores notify's nonrecursive-tree goroutines (linux uses
inotify, which spawns dispatch+internal instead of darwin's recursive
dispatch), and the read-only-dir prune spec skips under root, where
permission bits cannot make Remove fail.
The playlist ExternalImageURL step used sources.go's fromURL, which maps any
non-200 to a generic error, so a stale URL returning 404/410 was classified
transient: infinite backoff plus it counted toward the circuit breaker,
blocking valid external work. Add a local fetch in resolve.go that maps
404/410 to model.ErrNotFound (definitive) while keeping other non-200s
transient. sources.go is left untouched.
With CoverArtPriority="external,cover.jpg", a transient external failure
followed by a folder hit dropped the external error: the worker recorded
found and deleted the queue row, so the configured higher-priority external
art was never retried. Carry extError onto the fallback resolution and add
an outcomeFoundStale that persists+serves the art but reschedules via
MarkFailed, giving the external source another chance. When external later
answers definitively-not-found, the hit is not stale and the row is deleted.
A user-editable ExternalImageURL can point at an arbitrarily large endpoint;
a fast server could make the worker buffer hundreds of MB inside the 5s HTTP
timeout. Bound the read to a fixed 20MB cap (no config knob) via io.LimitReader
and fail the item if it is exceeded.
A playlist ExternalImageURL pointing at a local file that fails to open was
routed through extError, causing failed/48h-retry loops that burn a rate
limiter token forever instead of falling through to the generated grid.
Correct the inverted workerDeps.extGate comment, trim over-budget doc comments, and add conf.Server.ArtistImageFolder to the resolution fingerprint so an image-folder change re-resolves artist artwork.
Queue rows for deleted entities failed forever (Get -> ErrNotFound -> failed -> capped retries, unbounded). Add ArtworkQueueRepository.PurgeDangling, called from Prune next to the item_artwork purge. Separately, the found/absent path unconditionally deleted the dequeued row, erasing a concurrent scan re-enqueue; switch to DeleteIfUnchanged, which deletes only while retry_at still matches the dequeued value (verified retry_at is the column an Enqueue upsert resets).
resolvePlaylist only built the generated grid, dropping the uploaded-image, sidecar and ExternalImageURL sources the old reader_playlist.go chain serves. Port the full chain before the grid fallback: uploaded (upload), sidecar (folder), and ExternalImageURL routed through extGate with the same extError semantics as the other external steps. Also rewires the artist external step onto ArtistImageResult.
callGetImage swallowed all agent errors, so an agent outage surfaced as ErrNotFound and the worker settled artist artwork as a definitive absent (and reset the breaker). Add an additive ArtistImageResult path that returns the underlying agent error on transient failure while keeping ArtistImage byte-identical for existing callers; the worker's artist external step uses it via fromArtistExternalResult.
Resolves one queue item end to end: hash/dedup, decode + 128px thumbnail
blurhash, place bytes (store vs source file), and persist found/absent/
failed state for the worker (Task 4) to act on.
Known-absent states carry an empty hash and malformed persisted hashes could panic path sharding or inject separators; Write/Open/Remove now return an error for anything but 16 lowercase hex chars.
Write falls through to a real write when the liveness touch fails, and sweep retention now matches the recorded mime's extension so obsolete variants are reclaimed.
Duplicate ImageStore writes refresh the file mtime and Remove skips files newer than the cutoff, so overlapping acquisitions cannot lose their store files to a concurrent prune.
Sweep now honors an mtime grace window (in-flight acquisitions and temp files), reacquired orphans reset the prune grace window, and the queue mock implements real stale-absent semantics.
* fix(subsonic): surface the reason a transcode decision failed
getTranscodeDecision returned a bare "failed to make transcode decision"
with no clue why, and the probe command ran ffprobe with -v quiet, so even
the server log bottomed out at "exit status 1". A user whose files had been
moved by an external tool only saw the opaque error.
ProbeAudioStream now returns a typed ProbeError that separates the file path
from the reason: ffprobe runs with -v error so its stderr diagnostic is
captured, and a missing or unreadable file is reported as "file not found"
rather than ffprobe's misleading "Invalid data found". The handler logs the
full detail (including the path) and returns the reason to the client with
the server path stripped out.
Reported-by: Tolriq (Symfonium)
* refactor(ffmpeg): use errors.AsType for ExitError match
probeErrorReason used the older var+errors.As form while the rest of the
codebase (and its sibling transcodeFailureReason) uses the generic
errors.AsType. Switch to it for consistency; behavior is unchanged.
* fix(subsonic): return error 70 when the source file is missing
A getTranscodeDecision probe failure was always reported as generic error 0.
When the source file is gone (moved or deleted out from under the DB), that is
a not-found condition, so return the standard Subsonic error 70 ("data not
found") instead — matching what the endpoint already returns for an unknown
mediaId. Files that exist but are corrupt or unreadable stay error 0.
ProbeError now wraps the underlying cause and implements Unwrap, so the handler
detects the case with errors.Is(err, fs.ErrNotExist).
* fix(ffmpeg): keep probe error paths out of client-facing reasons
Addresses review feedback on the ProbeError type: the Reason field doubled as
both the log detail and the client message, so an ffprobe launch failure (a
*os.PathError from fork/exec) could leak the ffprobe binary path to clients,
and an unexpected stat error was reduced to "file not accessible" in the log.
Split the two concerns: Reason now holds only a path-free, client-safe string
(built at construction), while Error() logs the full underlying cause. Launch
failures return a generic "could not read file" instead of the raw exec error.
SafeReason no longer does substring path-stripping (removing the empty-Path
edge case); the stripping happens once, against ffprobe's stderr.
* fix(subsonic): don't report a broken ffprobe as a missing media file
Two issues from review of the previous commit:
Code 70 was selected with errors.Is(err, fs.ErrNotExist), but a launch failure
of a deleted ffprobe binary is an *os.PathError that also wraps fs.ErrNotExist.
A server-side ffprobe problem was therefore reported to clients as a missing
media file. ProbeError now carries an explicit NotFound flag, set only on the
file-access branch, and the handler keys the code off that instead of the chain.
ffprobe can also exit 0 while yielding no audio stream (an audio-suffixed
container holding only video). That parse failure was returned unwrapped, so
clients got "internal error"; it is now wrapped in a ProbeError too.
* feat(scrobbler): add exponential backoff delay helper
* feat(scrobbler): back off retries up to 4m during outages
* test(scrobbler): verify backoff schedule with synctest; clarify backoffDelay doc
Adds a testing/synctest-based test that drives the real run loop against a
failing service and asserts the exact 5s/10s/20s/40s retry schedule and the
drain-on-recovery reset, addressing the review note that the run loop's
behavior was untested. Also clarifies the backoffDelay doc comment: the
argument is a zero-based retry index.
* feat(utils): add ParseDuration/FormatDuration with day and week units
* feat(criteria): add per-playlist refreshDelay to smart playlist rules
* feat(smartplaylist): honor per-playlist refreshDelay in refresh gate
* feat(subsonic): compute smart playlist validUntil from effective refresh delay
* fix(playlists): reset smart playlist evaluation window when rules change via API
* refactor(utils): flatten FormatDuration recursion, single-pass duration regex
* fix(playlists): address PR review feedback
- Reset EvaluatedAt to nil instead of zero-time on rules change and NSP
re-import, so getPlaylist(s) never reports year-1 Changed/validUntil in
the window between an edit and the next owner read
- Parse negative d/w durations so they are rejected with the consistent
"negative duration" error instead of "unknown unit"
- Quote input in the negative-duration error, matching the parse error
- Gate per-playlist RefreshDelay behind IsSmartPlaylist, matching its doc
* fix(scrobbler): tolerate out-of-order playback reports
Clients may fire reportPlayback requests concurrently (Feishin sends
'starting' and 'playing' in parallel, and the previous track's 'stopped'
races the next track's start), so reports can be processed out of order.
Two cases corrupted the now-playing session: a late 'starting' for the
track already playing downgraded the session state, freezing position
estimation at 0:00 until the next report; and a late 'stopped' for the
previous track removed the new track's session and dispatched a playback
report mislabeled with the new track's metadata, causing presence-style
plugins (e.g. Discord Rich Presence) to clear or show stale state.
ReportPlayback now ignores a 'starting' report when the session already
has the same track in playing state, and ignores a 'stopped' report for
a track other than the current session's - skipping both the session
removal and the plugin dispatch, while still counting the play and
dispatching external scrobbles for the stopped track.
Reported in https://github.com/jeffvli/feishin/issues/2131
* fix(scrobbler): serialize session writes to close starting/playing race
The out-of-order 'starting' guard checked the session cache before the
media-file load, leaving a window where a concurrent 'playing' report on a
fresh session could write between the check and the write, and still be
overwritten back to 'starting'. Session check-then-write sections are now
serialized by a mutex, with the guard re-checked after the load. Also
tightens the guard comment to say 'playing session', matching the condition.
Found by Codex review on #5793.
* fix(scrobbler): fully exit ReportPlayback when ignoring out-of-order reports
The out-of-order guards used 'break', which only exits the switch, so the
post-switch NowPlaying block still ran for an ignored 'starting' report and
enqueued a NowPlaying dispatch with the stale report's position - potentially
overwriting a pending correct-position entry, since the queue is keyed by
client. Return nil instead, so ignored reports have no side effects.
Found by Gemini review on #5793.
* refactor(jellyfin): inject core/sonic into the Jellyfin Router
* feat(jellyfin): add AudioMuse /info endpoint
* feat(jellyfin): add AudioMuse /similar_tracks endpoint
* feat(jellyfin): gate AudioMuse endpoints on sonic provider
* feat(jellyfin): add AudioMuse /find_path endpoint
* fix(jellyfin): fix case-insensitive route collision across positions
canonicalRouteSegments keyed canonical case by lower-cased segment name
alone, globally. Two unrelated routes sharing a segment name with
different casing at different tree depths (e.g. "Info" in
/System/Info/Public vs "info" in /AudioMuseAI/info) silently overwrote
each other, 404-ing the loser even for exact-case requests. Replace the
flat map with a position-aware trie mirroring the routing tree.
* test(jellyfin): e2e tests for AudioMuse endpoints
* docs(jellyfin): document AudioMuse compatibility endpoints
* test(jellyfin): harden AudioMuse tests and doc note (final-review follow-ups)
- Comment-lock the []string{} (not nil) contract for /AudioMuseAI/info's
AvailableEndpoints so it keeps serializing as [] rather than null, and
add a raw-body assertion to the existing empty-list test to catch a
regression a struct-only unmarshal can't detect.
- Cover the previously-untested engine-error branch in similar_tracks and
find_path, both of which degrade to an empty result.
- Document that find_path's path/total_distance only reflect hops through
libraries the caller can access in multi-library setups.
* refactor(jellyfin): dedup AudioMuse test request helper, presize dedup map
* refactor(sonic): expose sonic.Engine interface; drop typed-nil guard in jellyfin.New
The Jellyfin Router's sonic field was an interface but New() took the concrete
*sonic.Sonic, so a nil arg became a non-nil typed-nil and needed a guard — the
only injected dependency that did. Move the interface (sonic.Engine) beside its
implementation, take it in New() like every other service, and bind it in wire.
* refactor(jellyfin): case-insensitive routing via lowercased paths
Replace the position-aware route trie with a trivial middleware that lowercases
the request path, and register every route in lowercase. Simpler, and no segment
name can collide across positions. caseInsensitivePaths moves into middlewares.go
alongside normalizeQueryKeys. Relies on the invariant that no Jellyfin path segment
carries case-sensitive data (all ids are lowercase hex via dto.EncodeID).
* feat(jellyfin): add AudioMuse /health endpoint
A liveness probe matching the reference plugin: 200 with an empty body when a
SonicSimilarity provider is loaded, 404 otherwise. /AudioMuseAI/info now
advertises it (list alphabetized like the plugin's OrderBy).
Also trims the AudioMuse and case-insensitive-routing comments to their essential
rationale.
* fix(jellyfin): hex-encode user IDs so lowercased paths stay valid
Address PR review: user IDs were the one id the Jellyfin API emitted raw (base62,
uppercase-capable), so lowercasing request paths could alter a userId segment. Encode
them via dto.EncodeID like every other id, making the 'all boundary ids are lowercase
hex' invariant true — no routing special-casing needed. Also caps user-controlled n /
max_steps, fixes the songAgent test comment, and adds leading slashes to the README
endpoint list.
* fix(jellyfin): stream playlist tracks instead of loading every one
PR #5783 left the playlist paths materializing: all three loaded every track
of a playlist, whatever the client asked for. A playlist can be the whole
library — a smart playlist matching everything — so this is the same OOM class
that PR fixed for the other collections. Measured on a 96k-track smart playlist,
/Items?ParentId=<playlist>&Limit=10 peaked at 1.3GB to return ten tracks.
Playlist tracks now stream from a cursor, like every other collection:
- PlaylistTrackRepository gains GetCursor and CountAll, sharing the select
builder with loadTracks so cursor rows hydrate identically, plus
GetMediaFileIDs for callers that need every id but no track data.
- playlists.Tracks(ctx, id) exposes that repo to the HTTP layer with visibility
enforced, returning ErrNotFound rather than the repo's nil-and-log-a-warning
(which /Items would hit on every album browse, since ParentId is usually not
a playlist).
- /Playlists/{id}/Items now honors StartIndex/Limit, which it silently ignored
before — it always returned the whole playlist. Real Jellyfin pages it.
- getPlaylist runs an id-only query: PlaylistInfo carries every track id so it
can't be paged, but it no longer hydrates rows it discards.
- The route joins the throttled group, as it's now cursor-backed.
Measured against a copy of a 96k-track production DB, peak RSS over idle, with
byte-for-byte identical responses on every endpoint:
/Items?ParentId=<pl>&Limit=10 1275MB -> 1MB 8.8s -> 3.6s (0.27s warm)
/Items?ParentId=<pl> unbounded 1353MB -> 8MB 8.7s -> 3.4s
/Playlists/<pl>/Items 1217MB -> 7MB 8.8s -> 3.4s
/Playlists/<pl> 1178MB -> 33MB 9.1s -> 3.8s
TotalRecordCount now costs a count query where the old path got it from
len(tracks): 6ms on the largest real playlist in that library (2638 tracks),
288ms on the synthetic all-96k one. The old path paid 1.3GB and 4s+ instead.
* fix(jellyfin): bound unbounded /Items searches
The other collections stream, so an unbounded one costs about one item of
memory. Search can't: the repositories' Search returns a slice, so it
materializes every match. PR #5783 left two ways to reach that.
A whitespace-only SearchTerm was the first. " " != "", so it took the search
path, where doSearch trims it back to empty and hits its "empty query, return
everything in natural order" branch — with no LIMIT, since executeTwoPhase only
applies one when Max > 0. The whole library, materialized. Trimming at the two
parse sites makes the `search != ""` checks mean what they look like they mean:
a blank term is not a search, so it takes the unfiltered streaming path, which
still returns everything, exactly as real Jellyfin does for an empty term.
A real search with no Limit was the second, and needs an actual bound. Search
gets a default of 100 when the client sends no Limit — matching the
DefaultSearchLimit in Jellyfin's unreleased SqlSearchProvider — plus a ceiling,
without which Limit=999999 would still materialize the library. The default
alone wouldn't have closed the hole. An explicit Limit under the ceiling is
honored unclamped, as upstream does; truncating a search is safe in a way
truncating /Items is not, since nothing syncs a library through searchTerm.
Note this is upstream's own bug: v10.11's /Search/Hints returns the entire
library for a whitespace-only term, which master fixed by switching to
ThrowIfNullOrWhiteSpace.
* fix(jellyfin): cap the search Limit the client asked for, not the merge window
searchPage sees two different things in opts.Max: the client's Limit for a
single-type query, and mergeTypes' internal offset+limit window for a multi-type
one. Clamping there hit both, so a multi-type search paging past the ceiling
fetched only `ceiling` rows of the first type and the merged page skipped into
the next one — IncludeItemTypes=Audio,MusicAlbum&SearchTerm=song&StartIndex=2000
&Limit=1 returned an album instead of the 2001st song.
The ceiling now applies where the client's Limit is read: queryItems (after the
playlist branch, so a playlist parent's page isn't capped by a stray SearchTerm)
and getArtists, which reads its own. A limit of 0 stays 0, keeping searchPage's
default. What a deep page materializes is then bounded by the client's
StartIndex, as it already was for any multi-type query, search or not.
Also from review: the playlist-track mock reused the previously stored Options
when called without any, so a later no-args call inherited stale paging.
* fix(jellyfin): apply the search default to the client's Limit, not per type
The default lived in searchPage, which runs per type and after mergeTypes has
already picked its branch. So an unbounded multi-type search left q.limit at 0,
mergeTypes took its chained branch, and StartIndex was applied to a list each
type had already truncated to the default — dropping matches rather than paging
them. With 200 matching songs, IncludeItemTypes=Audio,MusicAlbum&SearchTerm=song
&StartIndex=150 returned nothing at all, and without StartIndex it returned the
default per type instead of in total.
clampSearchLimit now applies the default and the ceiling together, to the
client's Limit, at the two places it's read (queryItems and getArtists). A search
therefore always gives mergeTypes a real window, so it pages the merged result
and cuts it once, at the end.
* fix(jellyfin): bound the multi-type search window against StartIndex
mergeTypes asks each type for offset+limit rows before paginating the merged
list, so a search still materialized whatever StartIndex asked for:
IncludeItemTypes=Audio,MusicAlbum&SearchTerm=x&StartIndex=500000&Limit=1 pulled
~500001 matches per type. Bounding the window alone isn't enough — below it the
merged rows are the client's page, but at it a truncated type is followed by the
next one's rows, which is what made a clamped window serve an album where the
2001st song belonged.
So the window is capped at maxSearchLimit and the page is clipped to it: pages
below the ceiling are served in full and unchanged, a page straddling it is cut
at it, and past it the result is empty rather than another type's rows. The
total reports what can actually be paged to, so a client stops instead of asking
for pages that no longer exist.
This is the merge's own limit, not the client's: a single-type search still pages
as deep as it likes, since its offset goes to SQL. Non-search multi-type queries
keep the unbounded offset+limit window, which predates this and wants the same
treatment via CountAll (exact per-type totals let whole types be skipped) rather
than a cap.
* fix(jellyfin): advertise the pageable search total, not the page window
Clipping the multi-type search total to `window` clipped it to StartIndex+Limit
on an ordinary page, so a first page of Limit=10 reported TotalRecordCount 10
however many matches there were, and a client paging on the total stopped after
one page. The cap belongs at the ceiling — what can be paged to overall — not at
the current page.
The tests missed it because the only one asserting a total used
StartIndex=maxSearchLimit, where the window happens to equal the ceiling.
* refactor(jellyfin): fold the search clamp into clampLimit and simplify mergeTypes
Cleanup pass over the branch, no behaviour change:
- clampSearchLimit was clampLimit (similar.go) with different constants, so the
latter takes the default and ceiling as arguments and both call it. Similar's
default moves out of three IntOr calls into defaultSimilarLimit.
- mergeTypes derived a second `limit` and needed an early return for the empty
page, only because paginate reads 0 as "unbounded". Clipping the merged slice
to the window instead lets q.limit be passed straight through: window is
min(offset+limit, ceiling), so clipping there is the same cut.
- playlistTracks returned (result, handled, error) where handled=false always
meant error=nil. Splitting the lookup out gives playlistTracksRepo returning
(repo, ok), and the nilerr suppression goes with it.
- The comment on playlists.Tracks blamed the log warning for its extra Get; the
reason is that PlaylistRepository.Tracks discards the error behind a nil. Also
drops a stale reference to a renamed variable.
