mirror of
https://github.com/navidrome/navidrome.git
synced 2026-11-15 14:48:41 +01:00
5ec6e6a8d4* fix(subsonic): make search3 empty-query pagination fast at large offsets Empty-query search3 (used by clients like Symfonium to sync the whole library) degraded linearly with songOffset: the offset optimization in optimizePagination keeps the original query's LEFT JOINs (annotation, bookmark, library) inside its rowid NOT IN subquery, making it as slow as plain OFFSET (~5s per page at offset 900K on a 920K-track library). Rewrite the empty-query branch of doSearch to use the same two-phase approach as the FTS search: Phase 1 paginates rowids on the bare main table, which SQLite satisfies with a covering index at any offset; Phase 2 hydrates only the page's rows with all JOINs. The Phase 2 hydration logic is extracted into hydrateRowidPage, now shared with ftsSearch.execute. Also replace the media_file_missing index with a composite covering index on (missing, library_id), so Phase 1 stays covering for non-admin users, whose queries include a library_id filter. The composite serves all missing-only lookups via its prefix. With a 920K-track / 85K-album test library, search3 empty-query responses are now flat (~0.1s) at every offset, for both admin and non-admin users (previously 3-5s at offsets above 600K). * refactor(persistence): share search Phase 1 contract and dedup junction fan-out Extract the Phase 1 query assembly that was duplicated between the FTS search and the empty-query search into executeTwoPhase: both paths now supply only their strategy-specific FROM/JOINs and ORDER BY, while the shared contract (missing filter, library access, options.Filters, and Max/Offset semantics) lives in one place. Also fix a pagination integrity bug: the artist library filter joins the library_artist junction table, so an artist present in multiple libraries produced duplicate rowids in Phase 1, corrupting offset-based pagination (short pages and repeated artists during full-library syncs). Phase 1 now applies DISTINCT whenever a junction-based LibraryFilter is set. DISTINCT is used instead of GROUP BY because bm25() cannot be evaluated in a grouped query; plain-filter tables (media_file, album) skip the dedup so their Phase 1 keeps the streaming covering-index plan. This also fixes the same duplication in the pre-existing FTS search path. * fix(persistence): pin artist search Phase 1 join order with CROSS JOIN search3 always filters artists by library (library_artist.library_id IN ...), and with the junction JOIN in the search Phase 1 rowid query SQLite chose to drive from library_artist, sorting every junction row with a temp b-tree on each page — a flat ~200ms penalty per request at 405K artists, even at offset 0 (the previous code avoided this by accident: its GROUP BY artist.id pinned an artist-driven plan). Use CROSS JOIN (SQLite's explicit join-order override) in a search-only variant of the artist library filter, keeping artist as the outer table so Phase 1 streams rowids in artist.id order from the primary key index and LIMIT/OFFSET short-circuits. The DISTINCT dedup stays and costs nothing under the streaming plan. Other artist queries keep the planner's freedom. With 405K artists, empty-query artist search is now 0.07s at offset 0 and 0.25s at offset 399K end-to-end (was 0.31s/0.34s before this fix, and up to 1.2s on master at deep offsets). Artist FTS text search is unaffected. |
||
|---|---|---|
| .. | ||
| 20200130083147_create_schema.go | ||
| 20200131183653_standardize_item_type.go | ||
| 20200208222418_add_defaults_to_annotations.go | ||
| 20200220143731_change_duration_to_float.go | ||
| 20200310171621_enable_search_by_albumartist.go | ||
| 20200310181627_add_transcoding_and_player_tables.go | ||
| 20200319211049_merge_search_into_main_tables.go | ||
| 20200325185135_add_album_artist_id.go | ||
| 20200326090707_fix_album_artists_importing.go | ||
| 20200327193744_add_year_range_to_album.go | ||
| 20200404214704_add_indexes.go | ||
| 20200409002249_enable_search_by_tracks_artists.go | ||
| 20200411164603_add_created_and_updated_fields_to_playlists.go | ||
| 20200418110522_reindex_to_fix_album_years.go | ||
| 20200419222708_reindex_to_change_full_text_search.go | ||
| 20200423204116_add_sort_fields.go | ||
| 20200508093059_add_artist_song_count.go | ||
| 20200512104202_add_disc_subtitle.go | ||
| 20200516140647_add_playlist_tracks_table.go | ||
| 20200608153717_referential_integrity.go | ||
| 20200706231659_add_default_transcodings.go | ||
| 20200710211442_add_playlist_path.go | ||
| 20200731095603_create_play_queues_table.go | ||
| 20200801101355_create_bookmark_table.go | ||
| 20200819111809_drop_email_unique_constraint.go | ||
| 20201003111749_add_starred_at_index.go | ||
| 20201010162350_add_album_size.go | ||
| 20201012210022_add_artist_playlist_size.go | ||
| 20201021085410_add_mbids.go | ||
| 20201021093209_add_media_file_indexes.go | ||
| 20201021135455_add_media_file_artist_index.go | ||
| 20201030162009_add_artist_info_table.go | ||
| 20201110205344_add_comments_and_lyrics.go | ||
| 20201128100726_add_real-path_option.go | ||
| 20201213124814_add_all_artist_ids_to_album.go | ||
| 20210322132848_add_timestamp_indexes.go | ||
| 20210418232815_fix_album_comments.go | ||
| 20210430212322_add_bpm_metadata.go | ||
| 20210530121921_create_shares_table.go | ||
| 20210601231734_update_share_fieldnames.go | ||
| 20210616150710_encrypt_all_passwords.go | ||
| 20210619231716_drop_player_name_unique_constraint.go | ||
| 20210623155401_add_user_prefs_player_scrobbler_enabled.go | ||
| 20210625223901_add_referential_integrity_to_user_props.go | ||
| 20210626213026_add_scrobble_buffer.go | ||
| 20210715151153_add_genre_tables.go | ||
| 20210821212604_add_mediafile_channels.go | ||
| 20211008205505_add_smart_playlist.go | ||
| 20211023184825_add_order_title_to_media_file.go | ||
| 20211026191915_unescape_lyrics_and_comments.go | ||
| 20211029213200_add_userid_to_playlist.go | ||
| 20211102215414_add_alphabetical_by_artist_index.go | ||
| 20211105162746_remove_invalid_artist_ids.go | ||
| 20220724231849_add_musicbrainz_release_track_id.go | ||
| 20221219112733_add_album_image_paths.go | ||
| 20221219140528_remove_cover_art_id.go | ||
| 20230112111457_add_album_paths.go | ||
| 20230114121537_touch_playlists.go | ||
| 20230115103212_create_internet_radio.go | ||
| 20230117155559_add_replaygain_metadata.go | ||
| 20230117180400_add_album_info.go | ||
| 20230119152657_recreate_share_table.go | ||
| 20230202143713_change_path_list_separator.go | ||
| 20230209181414_change_image_files_list_separator.go | ||
| 20230310222612_add_download_to_share.go | ||
| 20230404104309_empty_sql_migration.sql | ||
| 20230515184510_add_release_date.go | ||
| 20230616214944_rename_musicbrainz_recording_id.go | ||
| 20231208182311_add_discs_to_album.go | ||
| 20231209211223_alter_lyric_column.go | ||
| 20240122223340_add_default_values_to_null_columns.go.go | ||
| 20240426202913_add_id_to_scrobble_buffer.go | ||
| 20240511210036_add_sample_rate.go | ||
| 20240511220020_add_library_table.go | ||
| 20240629152843_remove_annotation_id.go | ||
| 20240802044339_player_use_user_id_over_username.go | ||
| 20241020003138_add_sort_tags_index.sql | ||
| 20241024125533_fix_sort_tags_collation_and_index.sql | ||
| 20241026183640_support_new_scanner.go | ||
| 20250522013904_share_user_id_on_delete.sql | ||
| 20250611010101_playqueue_current_to_index.go | ||
| 20250701010101_add_folder_hash.go | ||
| 20250701010102_add_annotation_user_foreign_key.sql | ||
| 20250701010103_add_library_stats.go | ||
| 20250701010104_make_replaygain_fields_nullable.go | ||
| 20250701010105_remove_dangling_items.sql | ||
| 20250701010106_add_participant_stats_to_all_artists.sql | ||
| 20250701010107_add_mbid_indexes.sql | ||
| 20250701010108_add_multi_library_support.go | ||
| 20250823142158_make_playqueue_position_int.sql | ||
| 20251109010105_add_annotation_rating_date.sql | ||
| 20251206013022_create_scrobbles_table.sql | ||
| 20260104203627_playlist_case_insensitive_name.sql | ||
| 20260106000620_create_plugin_table.sql | ||
| 20260117201522_add_avg_rating_column.sql | ||
| 20260220173400_add_fts5_search.go | ||
| 20260228020813_add_plugin_allow_write_access.sql | ||
| 20260228172956_add_playlist_image_file.go | ||
| 20260302021413_rename_playlist_image_fields.go | ||
| 20260307175815_add_codec_and_update_transcodings.go | ||
| 20260309120007_fix_probe_data_null.go | ||
| 20260309203355_ensure_default_transcodings.go | ||
| 20260310113858_fix_aac_transcode_command.go | ||
| 20260315233131_add_artist_uploaded_image.go | ||
| 20260316000000_normalize_timestamps.sql | ||
| 20260318182414_add_radio_uploaded_image.go | ||
| 20260405124200_fix_schema_inconsistencies.sql | ||
| 20260410201914_fix_zero_album_created_at.sql | ||
| 20260513173954_move_ss_before_input.go | ||
| 20260520211813_add_media_file_artists_composite_index.sql | ||
| 20260612171826_add_media_file_missing_library_id_index.sql | ||
| migration.go | ||