mirror of
https://github.com/navidrome/navidrome.git
synced 2026-10-10 19:37:08 +02:00
cc315dcc8c* perf(db): keep query planner statistics trustworthy with full ANALYZE PRAGMA optimize's internal ANALYZE runs with a limited analysis budget (~2000 rows) that writes wrong sqlite_stat1 entries for low-cardinality indexes: on a 96K-track library it claimed (missing, library_id) narrows to ~2000 rows when it matches the whole table. The planner then prefers that index over the sort index and falls back to a full-table temp B-tree sort per request, turning paginated song listings into multi-second queries (reproduced at 5.5s on real hardware; ~90x slower than with correct stats). Every index-creating migration re-triggered the poisoning via the post-migration optimize, and the daily optimizer could re-trigger it on large library changes. Setting analysis_limit on the connection does not help: optimize ignores it. Run a plain full ANALYZE instead: after migrations with schema changes, and in db.Optimize (daily schedule and scan-end). Stats are stored in the database file, so one connection suffices and the per-connection pool loop is gone. The Optimize call at shutdown is removed: stats are maintained at migration/scan/daily points, and an ANALYZE during shutdown only delays it and races container stop timeouts. * perf(db): drop startup PRAGMA optimize that re-poisons planner stats The startup PRAGMA optimize=0x10002 runs SQLite's budget-limited internal ANALYZE (bit 0x02), which writes truncated sqlite_stat1 rows for low-cardinality indexes -- the exact statistics-poisoning this PR set out to eliminate. Because DevOptimizeDB defaults to true, a restart with no pending migrations would re-poison the planner until the next scan or daily Optimize. Remove it: statistics are already refreshed with a full ANALYZE after schema-changing migrations (Init) and via Optimize at scan-end and on the daily schedule, so nothing on the startup path needs to touch them. Also clarify that Optimize is a no-op unless DevOptimizeDB is enabled. * chore(db): remove the DevOptimizeDB flag and skip Optimize on quick scans The flag only gated the optimize/ANALYZE maintenance calls and there is no reason to leave planner statistics unmaintained; the guards are gone along with the flag. The scan-end Optimize now runs only after full scans — quick scans barely move the statistics, and the daily schedule covers drift. * style(scanner): drop redundant comment in runOptimize * chore(persistence): drop the no-op PRAGMA optimize from ScanEnd Mask 0x10000 only selects candidate tables by size change; without the 0x02 action bit optimize does nothing (verified: sqlite_stat1 stays stale after a 100x table growth). The scan-end statistics refresh is db.Optimize's full ANALYZE, and the expression-collation-index concern the old comment guarded against no longer applies. * fix(scanner): run the post-scan ANALYZE in the server process With the external scanner (the default), the scan pipeline runs in a subprocess, so its ANALYZE was invisible to the server: SQLite loads sqlite_stat1 into the process's shared schema cache, and an ANALYZE from another process does not refresh it — verified with the production DSN that even brand-new pool connections keep planning with the old statistics until the server restarts. An in-process ANALYZE, by contrast, is immediately visible to every pooled connection through the same shared cache. Move the full-scan Optimize from the scanner pipeline to the scan controller, which always runs in the server process. * fix(scanner): honor promoted full scans in the optimize gate A quick scan resuming an interrupted full scan is promoted inside the scanner (possibly in a subprocess); mirror the promotion in the controller so the post-scan ANALYZE isn't skipped. * refactor: apply cleanup review findings - drop forceFullRescan's inline ANALYZE: Init already runs a full ANALYZE after any migration batch with schema changes, so upgrades including a full-rescan migration analyzed the whole DB twice - resumingFullScan uses a filtered CountAll instead of fetching and scanning all libraries - document why CallScan (CLI) deliberately skips the post-scan Optimize * perf(db): make planner analysis maintenance resilient Check analysis freshness every 30 minutes and refresh statistics when the last successful run is over 24 hours old or a scan marked them pending. Persist successful analysis state, retry skipped or failed maintenance, coordinate checks with scans, and cover standalone CLI full scans. * perf(db): avoid analyzing routine quick-scan changes Reserve pending analysis for full scans, unscanned libraries, and retry state. Incremental quick scans now rely on the 24-hour freshness window instead of triggering a full ANALYZE at the next maintenance check. * fix(scan): analyze resumed full scans in CLI * fix(db): back off failed analysis retries * feat(db): allow disabling scheduled analysis * test(db): remove redundant analysis coverage * refactor(db): split ANALYZE maintenance into optimize.go and dedupe call sites - move query-planner statistics code from db.go to its own optimize.go (and matching optimize_test.go) - log ANALYZE elapsed time inside Optimize/OptimizeIfNeeded instead of repeating the timing block at every call site - drop the LastDBAnalyzeAttemptAt write on success: it is only read while failures >= 1, and every failure rewrites it first - extract runPostScanAnalysis (cmd) and anyIncludedLibrary (scanner) helpers |
||
|---|---|---|
| .. | ||
| 20200130083147_create_schema.go | ||
| 20200131183653_standardize_item_type.go | ||
| 20200208222418_add_defaults_to_annotations.go | ||
| 20200220143731_change_duration_to_float.go | ||
| 20200310171621_enable_search_by_albumartist.go | ||
| 20200310181627_add_transcoding_and_player_tables.go | ||
| 20200319211049_merge_search_into_main_tables.go | ||
| 20200325185135_add_album_artist_id.go | ||
| 20200326090707_fix_album_artists_importing.go | ||
| 20200327193744_add_year_range_to_album.go | ||
| 20200404214704_add_indexes.go | ||
| 20200409002249_enable_search_by_tracks_artists.go | ||
| 20200411164603_add_created_and_updated_fields_to_playlists.go | ||
| 20200418110522_reindex_to_fix_album_years.go | ||
| 20200419222708_reindex_to_change_full_text_search.go | ||
| 20200423204116_add_sort_fields.go | ||
| 20200508093059_add_artist_song_count.go | ||
| 20200512104202_add_disc_subtitle.go | ||
| 20200516140647_add_playlist_tracks_table.go | ||
| 20200608153717_referential_integrity.go | ||
| 20200706231659_add_default_transcodings.go | ||
| 20200710211442_add_playlist_path.go | ||
| 20200731095603_create_play_queues_table.go | ||
| 20200801101355_create_bookmark_table.go | ||
| 20200819111809_drop_email_unique_constraint.go | ||
| 20201003111749_add_starred_at_index.go | ||
| 20201010162350_add_album_size.go | ||
| 20201012210022_add_artist_playlist_size.go | ||
| 20201021085410_add_mbids.go | ||
| 20201021093209_add_media_file_indexes.go | ||
| 20201021135455_add_media_file_artist_index.go | ||
| 20201030162009_add_artist_info_table.go | ||
| 20201110205344_add_comments_and_lyrics.go | ||
| 20201128100726_add_real-path_option.go | ||
| 20201213124814_add_all_artist_ids_to_album.go | ||
| 20210322132848_add_timestamp_indexes.go | ||
| 20210418232815_fix_album_comments.go | ||
| 20210430212322_add_bpm_metadata.go | ||
| 20210530121921_create_shares_table.go | ||
| 20210601231734_update_share_fieldnames.go | ||
| 20210616150710_encrypt_all_passwords.go | ||
| 20210619231716_drop_player_name_unique_constraint.go | ||
| 20210623155401_add_user_prefs_player_scrobbler_enabled.go | ||
| 20210625223901_add_referential_integrity_to_user_props.go | ||
| 20210626213026_add_scrobble_buffer.go | ||
| 20210715151153_add_genre_tables.go | ||
| 20210821212604_add_mediafile_channels.go | ||
| 20211008205505_add_smart_playlist.go | ||
| 20211023184825_add_order_title_to_media_file.go | ||
| 20211026191915_unescape_lyrics_and_comments.go | ||
| 20211029213200_add_userid_to_playlist.go | ||
| 20211102215414_add_alphabetical_by_artist_index.go | ||
| 20211105162746_remove_invalid_artist_ids.go | ||
| 20220724231849_add_musicbrainz_release_track_id.go | ||
| 20221219112733_add_album_image_paths.go | ||
| 20221219140528_remove_cover_art_id.go | ||
| 20230112111457_add_album_paths.go | ||
| 20230114121537_touch_playlists.go | ||
| 20230115103212_create_internet_radio.go | ||
| 20230117155559_add_replaygain_metadata.go | ||
| 20230117180400_add_album_info.go | ||
| 20230119152657_recreate_share_table.go | ||
| 20230202143713_change_path_list_separator.go | ||
| 20230209181414_change_image_files_list_separator.go | ||
| 20230310222612_add_download_to_share.go | ||
| 20230404104309_empty_sql_migration.sql | ||
| 20230515184510_add_release_date.go | ||
| 20230616214944_rename_musicbrainz_recording_id.go | ||
| 20231208182311_add_discs_to_album.go | ||
| 20231209211223_alter_lyric_column.go | ||
| 20240122223340_add_default_values_to_null_columns.go.go | ||
| 20240426202913_add_id_to_scrobble_buffer.go | ||
| 20240511210036_add_sample_rate.go | ||
| 20240511220020_add_library_table.go | ||
| 20240629152843_remove_annotation_id.go | ||
| 20240802044339_player_use_user_id_over_username.go | ||
| 20241020003138_add_sort_tags_index.sql | ||
| 20241024125533_fix_sort_tags_collation_and_index.sql | ||
| 20241026183640_support_new_scanner.go | ||
| 20250522013904_share_user_id_on_delete.sql | ||
| 20250611010101_playqueue_current_to_index.go | ||
| 20250701010101_add_folder_hash.go | ||
| 20250701010102_add_annotation_user_foreign_key.sql | ||
| 20250701010103_add_library_stats.go | ||
| 20250701010104_make_replaygain_fields_nullable.go | ||
| 20250701010105_remove_dangling_items.sql | ||
| 20250701010106_add_participant_stats_to_all_artists.sql | ||
| 20250701010107_add_mbid_indexes.sql | ||
| 20250701010108_add_multi_library_support.go | ||
| 20250823142158_make_playqueue_position_int.sql | ||
| 20251109010105_add_annotation_rating_date.sql | ||
| 20251206013022_create_scrobbles_table.sql | ||
| 20260104203627_playlist_case_insensitive_name.sql | ||
| 20260106000620_create_plugin_table.sql | ||
| 20260117201522_add_avg_rating_column.sql | ||
| 20260220173400_add_fts5_search.go | ||
| 20260228020813_add_plugin_allow_write_access.sql | ||
| 20260228172956_add_playlist_image_file.go | ||
| 20260302021413_rename_playlist_image_fields.go | ||
| 20260307175815_add_codec_and_update_transcodings.go | ||
| 20260309120007_fix_probe_data_null.go | ||
| 20260309203355_ensure_default_transcodings.go | ||
| 20260310113858_fix_aac_transcode_command.go | ||
| 20260315233131_add_artist_uploaded_image.go | ||
| 20260316000000_normalize_timestamps.sql | ||
| 20260318182414_add_radio_uploaded_image.go | ||
| 20260405124200_fix_schema_inconsistencies.sql | ||
| 20260410201914_fix_zero_album_created_at.sql | ||
| 20260513173954_move_ss_before_input.go | ||
| 20260520211813_add_media_file_artists_composite_index.sql | ||
| 20260612171826_add_media_file_missing_library_id_index.sql | ||
| 20260612222838_make_bpm_bitdepth_nullable.sql | ||
| 20260618120509_add_metadata_to_default_transcodings.go | ||
| 20260629123100_recently_added_plain_indexes.sql | ||
| 20260702152457_backfill_artist_search_normalized.go | ||
| 20260703013908_optimize_media_file_sort_indexes.sql | ||
| 20260712211040_add_primary_key_and_update_index_for_scrobbles.sql | ||
| migration.go | ||