mirror of
https://github.com/navidrome/navidrome.git
synced 2026-11-01 16:07:47 +01:00
perf(scanner): improve playlist importing on large libraries (#6055)
* perf(persistence): avoid a full media_file scan when resolving playlist paths FindByPaths built one OR-ed equality term per path. On the real media_file schema SQLite abandons the path index at just two OR-ed terms and falls back to SCAN media_file, re-testing every term against every row, so the cost grows with (rows x terms). Group the candidates by library and emit one IN list per library instead, which plans as SEARCH media_file USING INDEX media_file_path_nocase. The NOCASE collation is kept so ASCII case-insensitive matching still works. This is the dominant cost of M3U playlist import, which resolves every track on every scan. Measured with a 1000-track playlist against a migrated DB: 100k media_file rows: 397 -> 51,414 tracks/sec 500k media_file rows: 78.5 -> 47,174 tracks/sec The rate no longer degrades as the table grows, which is the expected shape for an index lookup. Reported in #6043, where an 8 hour scan of a 2M-song library spent 7h52m in the playlist phase. * docs(playlists): correct the stale reason for the M3U lookup chunk size The expression-tree depth ceiling applied to the old OR-per-path query, which capped a batch at roughly 500 terms. The IN form is bound by SQLite's 32766 variable limit instead, which the 400 candidates per chunk sit far below.
This commit is contained in:
parent
9ff0058620
commit
dbd26ba2e7
3 changed files with 42 additions and 9 deletions
|
|
@ -25,8 +25,8 @@ func (s *playlists) parseM3U(ctx context.Context, pls *model.Playlist, folder *m
|
|||
return err
|
||||
}
|
||||
var mfs model.MediaFiles
|
||||
// Chunk size of 100 lines, as each line can generate up to 4 lookup candidates
|
||||
// (NFC/NFD × raw/lowercase), and SQLite has a max expression tree depth of 1000.
|
||||
// Chunked so a huge playlist is not held in memory at once. Each line yields up to
|
||||
// 4 lookup candidates (NFC/NFD × raw/lowercase), far below SQLite's 32766 variables.
|
||||
for lines := range slice.CollectChunks(slice.LinesFrom(reader), 100) {
|
||||
filteredLines := make([]string, 0, len(lines))
|
||||
for _, line := range lines {
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue