navidrome/persistence
Deluan Quintão dbd26ba2e7
perf(scanner): improve playlist importing on large libraries (#6055)
* perf(persistence): avoid a full media_file scan when resolving playlist paths

FindByPaths built one OR-ed equality term per path. On the real media_file
schema SQLite abandons the path index at just two OR-ed terms and falls back to
SCAN media_file, re-testing every term against every row, so the cost grows with
(rows x terms).

Group the candidates by library and emit one IN list per library instead, which
plans as SEARCH media_file USING INDEX media_file_path_nocase. The NOCASE
collation is kept so ASCII case-insensitive matching still works.

This is the dominant cost of M3U playlist import, which resolves every track on
every scan. Measured with a 1000-track playlist against a migrated DB:

  100k media_file rows:   397 -> 51,414 tracks/sec
  500k media_file rows:  78.5 -> 47,174 tracks/sec

The rate no longer degrades as the table grows, which is the expected shape for
an index lookup. Reported in #6043, where an 8 hour scan of a 2M-song library
spent 7h52m in the playlist phase.

* docs(playlists): correct the stale reason for the M3U lookup chunk size

The expression-tree depth ceiling applied to the old OR-per-path query, which
capped a batch at roughly 500 terms. The IN form is bound by SQLite's 32766
variable limit instead, which the 400 candidates per chunk sit far below.
2026-08-30 22:17:25 -04:00
..
e2e feat(artwork): new artwork pipeline with background resolution and Low Quality Image Placeholders (#5847) 2026-08-09 15:03:27 -04:00
album_repository.go feat: add optional natural sort order for names and titles (#6015) 2026-08-23 14:31:37 -04:00
album_repository_test.go feat: add optional natural sort order for names and titles (#6015) 2026-08-23 14:31:37 -04:00
artist_repository.go feat(artwork): new artwork pipeline with background resolution and Low Quality Image Placeholders (#5847) 2026-08-09 15:03:27 -04:00
artist_repository_test.go feat(artwork): new artwork pipeline with background resolution and Low Quality Image Placeholders (#5847) 2026-08-09 15:03:27 -04:00
artwork_hydration.go feat(artwork): new artwork pipeline with background resolution and Low Quality Image Placeholders (#5847) 2026-08-09 15:03:27 -04:00
artwork_hydration_test.go feat(artwork): new artwork pipeline with background resolution and Low Quality Image Placeholders (#5847) 2026-08-09 15:03:27 -04:00
artwork_queue_repository.go perf(artwork): cap the stale-absent recheck at 100 items per kind per hour (#6007) 2026-08-21 15:23:07 -04:00
artwork_queue_repository_test.go perf(artwork): cap the stale-absent recheck at 100 items per kind per hour (#6007) 2026-08-21 15:23:07 -04:00
artwork_repository.go feat(artwork): store the resolution trace so artwork explain works offline (#5980) 2026-08-21 10:24:01 -04:00
artwork_repository_test.go feat(artwork): store the resolution trace so artwork explain works offline (#5980) 2026-08-21 10:24:01 -04:00
collation_test.go perf(db): index media_file album/artist sort orders (#5706) 2026-07-03 08:58:55 -04:00
criteria_sql.go feat(scrobbler): add per-user scrobble filter (#5964) 2026-08-15 16:10:53 -04:00
criteria_sql_benchmark_test.go feat(scrobbler): add per-user scrobble filter (#5964) 2026-08-15 16:10:53 -04:00
criteria_sql_test.go feat(scrobbler): add per-user scrobble filter (#5964) 2026-08-15 16:10:53 -04:00
export_test.go feat(bfr): Big Refactor: new scanner, lots of new fields and tags, improvements and DB schema changes (#2709) 2025-02-19 20:35:17 -05:00
folder_repository.go fix(artwork): re-resolve artwork when image files change on disk (#5965) 2026-08-16 13:24:07 -04:00
folder_repository_test.go test: unskip path-separator tests on Windows (#5381) (#5916) 2026-08-19 11:50:32 -04:00
genre_repository.go fix(jellyfin): resolve genre id as a MusicGenre item 2026-08-10 21:37:57 -04:00
genre_repository_test.go refactor: run Go modernize (#5002) 2026-02-08 09:57:30 -05:00
helpers.go feat: add optional natural sort order for names and titles (#6015) 2026-08-23 14:31:37 -04:00
helpers_test.go feat: add optional natural sort order for names and titles (#6015) 2026-08-23 14:31:37 -04:00
item_tags_test.go perf(genre): index genre filtering via join tables across all APIs (#5940) 2026-08-11 08:00:50 -04:00
library_repository.go perf(db): keep query planner statistics trustworthy with full ANALYZE (#5740) 2026-07-13 12:04:29 -04:00
library_repository_test.go perf(subsonic): speed up artist search3 deep-offset pagination (#5620) 2026-06-16 21:47:15 -04:00
mediafile_repository.go perf(scanner): improve playlist importing on large libraries (#6055) 2026-08-30 22:17:25 -04:00
mediafile_repository_test.go perf(scanner): improve playlist importing on large libraries (#6055) 2026-08-30 22:17:25 -04:00
persistence.go feat(artwork): new artwork pipeline with background resolution and Low Quality Image Placeholders (#5847) 2026-08-09 15:03:27 -04:00
persistence_suite_test.go fix(jellyfin): stream collection responses to prevent OOM on large libraries (#5783) 2026-07-15 14:07:00 -04:00
persistence_test.go fix(server): play queue should not return empty entries for deleted tracks 2024-09-20 11:22:37 -04:00
player_repository.go fix(share): enforce per-user ownership on share reads 2026-06-05 15:50:59 -04:00
player_repository_test.go fix(share): enforce per-user ownership on share reads 2026-06-05 15:50:59 -04:00
playlist_repository.go feat: add optional natural sort order for names and titles (#6015) 2026-08-23 14:31:37 -04:00
playlist_repository_test.go feat: add optional natural sort order for names and titles (#6015) 2026-08-23 14:31:37 -04:00
playlist_track_repository.go feat: add optional natural sort order for names and titles (#6015) 2026-08-23 14:31:37 -04:00
playlist_track_repository_test.go fix(playlists): chunk track deletes to stay under the SQLite variable limit (#5977) 2026-08-17 15:47:41 -04:00
playqueue_repository.go refactor: multiple syntax updates for Go 1.26 2026-05-19 18:02:36 -03:00
playqueue_repository_test.go fix(server): ensure single record per user by reusing existing playqueue ID 2025-06-11 17:26:13 -04:00
plugin_cleanup.go feat(plugins): New Plugin System with multi-language PDK support (#4833) 2026-01-14 19:22:48 -05:00
plugin_cleanup_test.go feat(plugins): New Plugin System with multi-language PDK support (#4833) 2026-01-14 19:22:48 -05:00
plugin_repository.go fix(plugins): clear plugin errors on startup to allow retrying 2026-03-02 08:56:56 -05:00
plugin_repository_test.go fix(plugins): clear plugin errors on startup to allow retrying 2026-03-02 08:56:56 -05:00
property_repository.go Replace beego/orm with dbx (#2693) 2023-12-09 13:52:17 -05:00
property_repository_test.go revert: separation of write and read DBs 2024-11-19 18:41:50 -05:00
radio_repository.go feat(artwork): new artwork pipeline with background resolution and Low Quality Image Placeholders (#5847) 2026-08-09 15:03:27 -04:00
radio_repository_test.go feat(artwork): new artwork pipeline with background resolution and Low Quality Image Placeholders (#5847) 2026-08-09 15:03:27 -04:00
scrobble_buffer_repository.go fix(plugins): discard buffered scrobbles when a plugin is removed (#5737) 2026-07-08 12:37:17 -04:00
scrobble_buffer_repository_test.go fix(plugins): discard buffered scrobbles when a plugin is removed (#5737) 2026-07-08 12:37:17 -04:00
scrobble_repository.go feat(server): add scrobble history Native API (#5761) 2026-07-13 11:32:03 -04:00
scrobble_repository_test.go feat(server): add scrobble history Native API (#5761) 2026-07-13 11:32:03 -04:00
share_repository.go feat(artist): add Share and Download actions to the Artist detail page (#5944) 2026-08-12 11:59:56 -04:00
share_repository_test.go feat(artist): add Share and Download actions to the Artist detail page (#5944) 2026-08-12 11:59:56 -04:00
smart_playlist_repository.go feat(scrobbler): add per-user scrobble filter (#5964) 2026-08-15 16:10:53 -04:00
smart_playlist_repository_test.go fix(playlist): preserve smart playlist song count on re-import (#5907) (#5908) 2026-08-18 20:58:55 -04:00
sort_index_coverage_test.go perf(db): index media_file album/artist sort orders (#5706) 2026-07-03 08:58:55 -04:00
sql_annotations.go feat(playlists): per-user starred/rating annotations (backend) (#5749) 2026-07-14 07:38:25 -04:00
sql_annotations_test.go perf(db): skip annotation join in CountAll when unused (#5694) 2026-06-30 22:50:43 -04:00
sql_base_repository.go feat: add optional natural sort order for names and titles (#6015) 2026-08-23 14:31:37 -04:00
sql_base_repository_test.go fix(jellyfin): honor the Filters, SortBy and MaxHeight params clients actually send (#5981) 2026-08-19 08:36:44 -04:00
sql_bookmarks.go refactor: run Go modernize (#5002) 2026-02-08 09:57:30 -05:00
sql_bookmarks_test.go revert: separation of write and read DBs 2024-11-19 18:41:50 -05:00
sql_participations.go perf(persistence): use *_artists join tables for artist participant filters (#5930) 2026-08-10 11:42:27 -04:00
sql_restful.go fix(jellyfin): honor the Filters, SortBy and MaxHeight params clients actually send (#5981) 2026-08-19 08:36:44 -04:00
sql_restful_test.go feat(subsonic): sort search3 results by relevance (#5086) 2026-02-23 08:51:54 -05:00
sql_search.go perf(subsonic): speed up artist search3 deep-offset pagination (#5620) 2026-06-16 21:47:15 -04:00
sql_search_fts.go feat(search): rank exact matches above prefix matches (#5704) 2026-07-02 15:51:03 -04:00
sql_search_fts_test.go feat(search): rank exact matches above prefix matches (#5704) 2026-07-02 15:51:03 -04:00
sql_search_like.go chore: go fix 2026-05-28 22:13:05 -03:00
sql_search_like_test.go feat(subsonic): sort search3 results by relevance (#5086) 2026-02-23 08:51:54 -05:00
sql_search_test.go feat(subsonic): sort search3 results by relevance (#5086) 2026-02-23 08:51:54 -05:00
sql_tags.go perf(genre): index genre filtering via join tables across all APIs (#5940) 2026-08-11 08:00:50 -04:00
tag_library_filtering_test.go refactor: run Go modernize (#5002) 2026-02-08 09:57:30 -05:00
tag_repository.go feat(jellyfin): filter items by year and record label (#5817) 2026-07-19 12:39:31 -04:00
tag_repository_test.go refactor: run Go modernize (#5002) 2026-02-08 09:57:30 -05:00
transcoding_repository.go fix: restrict transcoding config reads to admins (#5564) 2026-06-04 23:07:13 -04:00
transcoding_repository_test.go fix: restrict transcoding config reads to admins (#5564) 2026-06-04 23:07:13 -04:00
user_props_repository.go Replace beego/orm with dbx (#2693) 2023-12-09 13:52:17 -05:00
user_repository.go feat(jellyfin): non-expiring, audience-scoped tokens revocable by password change (#6013) 2026-08-22 20:36:24 -04:00
user_repository_test.go feat(jellyfin): non-expiring, audience-scoped tokens revocable by password change (#6013) 2026-08-22 20:36:24 -04:00