remote/navidrome
1
0
Fork 0
mirror of https://github.com/navidrome/navidrome.git synced 2026-10-08 02:17:25 +02:00
Code Issues Projects Releases Packages Wiki Activity

feat(cli): add missing file list and remap subcommands (#5928)

Browse source 
* feat(cli): add missing file list and remap subcommands

Signed-off-by: zerovox <933064+zerovox@users.noreply.github.com>

* fix: prevent remapping from dropping participants on target track

* fix: after remapping, refresh stats synchronously

* fix: only move album annotations if moving a track would empty the old album

* fix(persistence): keep the new item's annotation when reassigning onto an item the user already annotated

ReassignAnnotation was a plain UPDATE; the annotation table is unique on
(user_id, item_id, item_type), so when a user had annotated both items the
statement aborted and none of the rows moved. In the scanner that surfaced as
a warning; in the missing-file remap it rolled back the whole operation.
UPDATE OR IGNORE moves what it can and leaves the conflicting rows for GC.

* fix(core): keep the target track's history when remapping a missing file onto it

The remap discards the target's row, and GC then dropped its play counts,
stars, ratings, bookmarks and every playlist entry pointing at it. That is
harmless in the scanner, whose target was imported seconds earlier, but the
CLI lets the user pick any existing track. Move those references onto the
surviving id first; where a user already has a row for both, theirs on the
missing file wins.

* fix(persistence): stop FindByPaths dropping plain paths that contain a colon

Any colon was taken as the libraryID separator, and a non-numeric prefix
made the whole path vanish from the lookup. 'missing fix' then rejected the
very paths 'missing list' printed, and M3U imports silently skipped such
tracks. Only a numeric prefix qualifies a path now.

* perf(cli): stream 'missing list' instead of loading every missing file into memory

GetAll materialised the whole result set before a single row was written;
on a library with 97k missing files that peaked at 1.28 GB of RSS. Iterate
the repository cursor and write rows as they arrive.

* refactor(core): tidy the missing-file remap

Drop the log lines copied from deleteMissing that still said 'after deleting
missing files', the debug-on-success branches, and the what-comments; build
the affected album list without slice helpers.

* fix(cli): move path to the last column of 'missing list'

Path is the only variable-width field, so leading with it misaligns every
row that follows. Applies to both csv and json.

* fix(persistence): also try a numeric colon prefix as a plain path

'1999: A Different Life/01.mp3' parsed as library 1999 plus a truncated path
and matched nothing. The prefix is ambiguous, so search both ways.

Also buffer the json branch of 'missing list', which wrote a syscall per row.

* fix(persistence): move scrobbles and buffered scrobbles off a discarded media file

Both tables carry ON DELETE CASCADE on media_file_id, so 'missing fix'
deleting the target erased its play history and dropped scrobbles still
waiting on an external service. scrobble_buffer needs OR IGNORE for its
unique (user_id, service, media_file_id, play_time).

* fix(persistence): recompute the cached average rating after merging annotations

Merging the discarded row's annotations grows the rating population of the
surviving track, so media_file.average_rating no longer matched what the
annotation rows say. Only reachable since the remap started merging those
rows instead of deleting them.

* fix(persistence): recompute the cached average rating inside ReassignAnnotation

Moving annotation rows always changes the new item's rating population, so
the recompute belongs with the move rather than at each call site. Covers
the album reassign in the remap and the two scanner sites, and replaces the
explicit call ReassignReferences was making.

Album was the worse case: rate an album, move its files, and 'missing fix'
handed the rating to an album still caching an average of 0.

* fix(cli): let libraryID:path win over a file literally named like one

FindByPaths searches a numeric-prefixed reference both ways, so a top-level
file named '1:foo.mp3' can tie with library 1's 'foo.mp3'. The CLI then
rejected the reference as ambiguous while advising the exact syntax the
caller had used. Also disambiguates the same path in two libraries, which
is what the qualified form is for.

---------

Signed-off-by: zerovox <933064+zerovox@users.noreply.github.com>
Co-authored-by: Deluan Quintão <deluan@navidrome.org>
This commit is contained in:
ts 2026-09-12 10:08:25 -06:00 • committed by GitHub
commit c6732e1fdf
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
12 changed files with 756 additions and 29 deletions
Download patch file Download diff file Expand all files Collapse all files

171
cmd/missing.go Normal file
View file

@ -0,0 +1,171 @@
package cmd
import (
"bufio"
"context"
"encoding/csv"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"strconv"
"strings"
"github.com/Masterminds/squirrel"
"github.com/navidrome/navidrome/core"
"github.com/navidrome/navidrome/log"
"github.com/navidrome/navidrome/model"
"github.com/navidrome/navidrome/utils/slice"
"github.com/spf13/cobra"
)
var missingListFormat string
func init() {
missingListCmd.Flags().StringVarP(&missingListFormat, "format", "f", "csv", "output format [supported values: csv, json]")
missingCmd.AddCommand(missingListCmd)
missingCmd.AddCommand(missingFixCmd)
rootCmd.AddCommand(missingCmd)
}
var (
missingCmd = &cobra.Command{
Use: "missing",
Short: "Manage missing files",
Long: "List files marked as missing and remap them onto existing files",
}
missingListCmd = &cobra.Command{
Use: "list",
Short: "List missing files",
Run: func(cmd *cobra.Command, _ []string) {
runMissingList(cmd.Context())
},
}
missingFixCmd = &cobra.Command{
Use: "fix <missing path|id> <target path|id>",
Short: "Remap a missing file onto an existing file",
Long: "Remap a file marked as missing onto an existing (non-missing) file, the same way\n" +
"the scanner reconciles moved or renamed files. Each argument may be a media file ID,\n" +
"a library-relative path, or a libraryID:path pair.",
Args: cobra.ExactArgs(2),
Run: func(cmd *cobra.Command, args []string) {
runMissingFix(cmd.Context(), args[0], args[1])
},
}
)
type displayMissingFile struct {
ID string `json:"id"`
LibraryID int `json:"libraryId"`
Title string `json:"title"`
Album string `json:"album"`
Artist string `json:"artist"`
Path string `json:"path"`
}
func runMissingList(ctx context.Context) {
if missingListFormat != "csv" && missingListFormat != "json" {
log.Fatal("Invalid output format. Must be one of csv, json", "format", missingListFormat)
}
ds, ctx := getAdminContext(ctx)
mfs, err := ds.MediaFile(ctx).GetCursor(model.QueryOptions{
Filters: squirrel.Eq{"missing": true},
Sort: "path",
})
if err == nil {
err = writeMissingList(os.Stdout, missingListFormat, mfs)
}
if err != nil {
log.Fatal(ctx, "Failed to retrieve missing files", err)
}
}
// writeMissingList streams the cursor so a library with many missing files doesn't get loaded into memory
func writeMissingList(w io.Writer, format string, mfs model.MediaFileCursor) error {
if format == "json" {
bw := bufio.NewWriter(w)
_, _ = io.WriteString(bw, "[")
sep := ""
for mf, err := range mfs {
if err != nil {
return err
}
j, _ := json.Marshal(displayMissingFile{ID: mf.ID, LibraryID: mf.LibraryID, Title: mf.Title, Album: mf.Album, Artist: mf.Artist, Path: mf.Path})
_, _ = fmt.Fprintf(bw, "%s%s", sep, j)
sep = ","
}
_, _ = io.WriteString(bw, "]\n")
return bw.Flush()
}
cw := csv.NewWriter(w)
_ = cw.Write([]string{"id", "library id", "title", "album", "artist", "path"})
for mf, err := range mfs {
if err != nil {
return err
}
_ = cw.Write([]string{mf.ID, strconv.Itoa(mf.LibraryID), mf.Title, mf.Album, mf.Artist, mf.Path})
}
cw.Flush()
return cw.Error()
}
func runMissingFix(ctx context.Context, missingRef, targetRef string) {
ds, ctx := getAdminContext(ctx)
missing := resolveMediaFile(ctx, ds, missingRef)
target := resolveMediaFile(ctx, ds, targetRef)
if err := core.NewMaintenance(ds).RemapMissingFile(ctx, missing.ID, target.ID); err != nil {
log.Fatal(ctx, "Failed to remap missing file", "missing", missing.Path, "target", target.Path, err)
}
fmt.Printf("Remapped %q onto %q\n", missing.Path, target.Path)
}
// resolveMediaFile looks up a media file by ID first, then by path (optionally libraryID:path).
func resolveMediaFile(ctx context.Context, ds model.DataStore, ref string) *model.MediaFile {
mf, err := ds.MediaFile(ctx).Get(ref)
if err == nil {
return mf
}
if !errors.Is(err, model.ErrNotFound) {
log.Fatal(ctx, "Error looking up media file", "ref", ref, err)
}
mfs, err := ds.MediaFile(ctx).FindByPaths([]string{ref})
if err != nil {
log.Fatal(ctx, "Error looking up media file by path", "ref", ref, err)
}
if len(mfs) == 0 {
log.Fatal(ctx, "No media file found", "ref", ref)
}
mfs = preferQualified(ref, mfs)
if len(mfs) > 1 {
log.Fatal(ctx, "Path matches multiple files; disambiguate with an ID or libraryID:path", "ref", ref, "matches", len(mfs))
}
return &mfs[0]
}
// preferQualified resolves the ambiguity FindByPaths creates by searching a "libraryID:path"
// reference both ways: an explicit library wins over a file literally named like one.
func preferQualified(ref string, mfs model.MediaFiles) model.MediaFiles {
id, path, ok := strings.Cut(ref, ":")
if !ok {
return mfs
}
libraryID, err := strconv.Atoi(id)
if err != nil {
return mfs
}
qualified := slice.Filter(mfs, func(mf model.MediaFile) bool {
return mf.LibraryID == libraryID && strings.EqualFold(mf.Path, path)
})
if len(qualified) == 0 {
return mfs
}
return qualified
}

81
cmd/missing_test.go Normal file
View file

@ -0,0 +1,81 @@
package cmd
import (
"errors"
"strings"
"github.com/navidrome/navidrome/model"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
)
var _ = Describe("writeMissingList", func() {
cursor := func(err error, mfs ...model.MediaFile) model.MediaFileCursor {
return func(yield func(model.MediaFile, error) bool) {
for _, mf := range mfs {
if !yield(mf, nil) {
return
}
}
if err != nil {
yield(model.MediaFile{}, err)
}
}
}
song := model.MediaFile{ID: "1", LibraryID: 1, Path: "Bach: Goldberg/01.mp3", Title: "Aria", Album: "Goldberg", Artist: "Bach"}
It("writes csv with a header, quoting as needed", func() {
var out strings.Builder
Expect(writeMissingList(&out, "csv", cursor(nil, song))).To(Succeed())
Expect(out.String()).To(Equal("id,library id,title,album,artist,path\n1,1,Aria,Goldberg,Bach,Bach: Goldberg/01.mp3\n"))
})
It("writes a json array", func() {
var out strings.Builder
Expect(writeMissingList(&out, "json", cursor(nil, song, song))).To(Succeed())
Expect(out.String()).To(MatchJSON(`[
{"id":"1","libraryId":1,"path":"Bach: Goldberg/01.mp3","title":"Aria","album":"Goldberg","artist":"Bach"},
{"id":"1","libraryId":1,"path":"Bach: Goldberg/01.mp3","title":"Aria","album":"Goldberg","artist":"Bach"}
]`))
})
It("writes an empty json array when nothing is missing", func() {
var out strings.Builder
Expect(writeMissingList(&out, "json", cursor(nil))).To(Succeed())
Expect(out.String()).To(MatchJSON(`[]`))
})
It("returns the cursor's error", func() {
var out strings.Builder
Expect(writeMissingList(&out, "csv", cursor(errors.New("boom"), song))).To(MatchError("boom"))
})
})
var _ = Describe("preferQualified", func() {
target := model.MediaFile{ID: "want", LibraryID: 1, Path: "foo.mp3"}
decoy := model.MediaFile{ID: "decoy", LibraryID: 1, Path: "1:foo.mp3"}
It("picks the library-qualified match over a literal path that looks like one", func() {
Expect(preferQualified("1:foo.mp3", model.MediaFiles{target, decoy})).To(Equal(model.MediaFiles{target}))
})
It("picks the named library when the same path exists in two", func() {
other := model.MediaFile{ID: "other", LibraryID: 2, Path: "foo.mp3"}
Expect(preferQualified("1:foo.mp3", model.MediaFiles{target, other})).To(Equal(model.MediaFiles{target}))
})
It("leaves an unqualified reference ambiguous", func() {
both := model.MediaFiles{target, {ID: "other", LibraryID: 2, Path: "foo.mp3"}}
Expect(preferQualified("foo.mp3", both)).To(Equal(both))
})
It("leaves it alone when the prefix is not a library id", func() {
both := model.MediaFiles{decoy, {ID: "other", LibraryID: 2, Path: "1:foo.mp3"}}
Expect(preferQualified("x:foo.mp3", both)).To(Equal(both))
})
It("leaves it alone when no candidate matches the qualified form", func() {
both := model.MediaFiles{decoy, {ID: "other", LibraryID: 2, Path: "1:foo.mp3"}}
Expect(preferQualified("9:nope.mp3", both)).To(Equal(both))
})
})

97
core/maintenance.go
View file

@ -2,6 +2,7 @@ package core
import (
"context"
"errors"
"fmt"
"slices"
"sync"
@ -14,11 +15,23 @@ import (
"github.com/navidrome/navidrome/utils/slice"
)
var (
// ErrNotMissing is returned when a remap is attempted from a file not marked as missing.
ErrNotMissing = errors.New("file is not marked as missing")
// ErrTargetMissing is returned when the remap target is itself a missing file.
ErrTargetMissing = errors.New("target file is missing")
// ErrSameFile is returned when the remap source and target are the same file.
ErrSameFile = errors.New("missing and target are the same file")
)
type Maintenance interface {
// DeleteMissingFiles deletes specific missing files by their IDs
DeleteMissingFiles(ctx context.Context, ids []string) error
// DeleteAllMissingFiles deletes all files marked as missing
DeleteAllMissingFiles(ctx context.Context) error
// RemapMissingFile moves a missing file's identity onto an existing file, the manual
// counterpart to the scanner's move detection (phaseMissingTracks.moveMatched).
RemapMissingFile(ctx context.Context, missingID, targetID string) error
}
type maintenanceService struct {
@ -40,6 +53,87 @@ func (s *maintenanceService) DeleteAllMissingFiles(ctx context.Context) error {
return s.deleteMissing(ctx, nil)
}
func (s *maintenanceService) RemapMissingFile(ctx context.Context, missingID, targetID string) error {
if missingID == targetID {
return fmt.Errorf("%w: %q", ErrSameFile, missingID)
}
missing, err := s.ds.MediaFile(ctx).Get(missingID)
if err != nil {
return fmt.Errorf("loading missing file %q: %w", missingID, err)
}
if !missing.Missing {
return fmt.Errorf("%w: %q", ErrNotMissing, missingID)
}
target, err := s.ds.MediaFile(ctx).GetWithParticipants(targetID)
if err != nil {
return fmt.Errorf("loading target file %q: %w", targetID, err)
}
if target.Missing {
return fmt.Errorf("%w: %q", ErrTargetMissing, targetID)
}
oldAlbumID, newAlbumID := missing.AlbumID, target.AlbumID
err = s.ds.WithTx(func(tx model.DataStore) error {
discardedID := target.ID
// Preserve the original created_at so the remapped track doesn't resurface in "Recently Added"
target.CreatedAt = missing.CreatedAt
target.ID = missing.ID
if err := tx.MediaFile(ctx).Put(target); err != nil {
return fmt.Errorf("update matched track: %w", err)
}
// Unlike the scanner's freshly-imported target, this one may carry history of its own
if err := tx.MediaFile(ctx).ReassignReferences(discardedID, missing.ID); err != nil {
return fmt.Errorf("reassign target references: %w", err)
}
if err := tx.MediaFile(ctx).Delete(discardedID); err != nil {
return fmt.Errorf("delete discarded track: %w", err)
}
if oldAlbumID != newAlbumID {
oldAlbumTracks, err := tx.MediaFile(ctx).CountAll(model.QueryOptions{Filters: squirrel.Eq{"album_id": oldAlbumID}})
if err != nil {
return fmt.Errorf("get old album tracks: %w", err)
}
if oldAlbumTracks == 0 {
if err := tx.Album(ctx).ReassignAnnotation(oldAlbumID, newAlbumID); err != nil {
return fmt.Errorf("reassign album annotations: %w", err)
}
if err := tx.Album(ctx).CopyAttributes(oldAlbumID, newAlbumID, "created_at"); err != nil && !errors.Is(err, model.ErrNotFound) {
return fmt.Errorf("copy album attributes: %w", err)
}
}
}
return nil
})
if err != nil {
log.Error(ctx, "Error remapping missing file", "missing", missing.Path, "target", target.Path, err)
return err
}
if err := s.ds.GC(ctx); err != nil {
log.Error(ctx, "Error running GC after remapping missing file", err)
return err
}
// Stats are refreshed synchronously, unlike deleteMissing, so the CLI sees them before it exits.
// album/artist play count aggregates are not recalculated here; they are refreshed by the next scan.
if _, err := s.ds.Artist(ctx).RefreshStats(true); err != nil {
log.Error(ctx, "Error refreshing artist stats after remapping missing file", err)
}
affectedAlbumIDs := []string{newAlbumID}
if oldAlbumID != newAlbumID {
affectedAlbumIDs = append(affectedAlbumIDs, oldAlbumID)
}
if err := s.refreshAlbums(ctx, affectedAlbumIDs); err != nil {
log.Error(ctx, "Error refreshing album stats after remapping missing file", err)
}
return nil
}
// deleteMissing handles the deletion of missing files and triggers necessary cleanup operations
func (s *maintenanceService) deleteMissing(ctx context.Context, ids []string) error {
// Track affected album IDs before deletion for refresh
@ -68,7 +162,8 @@ func (s *maintenanceService) deleteMissing(ctx context.Context, ids []string) er
return err
}
// Refresh statistics in background
// Refresh statistics in background. album/artist play count aggregates are not recalculated
// here; they are refreshed by the next scan.
s.refreshStatsAsync(ctx, affectedAlbumIDs)
return nil

162
core/maintenance_test.go
View file

@ -4,6 +4,7 @@ import (
"context"
"errors"
"sync"
"time"
"github.com/navidrome/navidrome/model"
"github.com/navidrome/navidrome/model/request"
@ -250,6 +251,167 @@ var _ = Describe("Maintenance", func() {
})
})
})
Describe("RemapMissingFile", func() {
It("relocates the missing file's identity onto the target and runs GC", func() {
created := time.Now().Add(-30 * 24 * time.Hour)
mfRepo.SetData(model.MediaFiles{
{ID: "m1", Path: "old/song.mp3", AlbumID: "album1", CreatedAt: created, Missing: true},
{ID: "t1", Path: "new/song.mp3", AlbumID: "album1", Missing: false},
})
Expect(service.RemapMissingFile(ctx, "m1", "t1")).To(Succeed())
got, err := mfRepo.Get("m1")
Expect(err).ToNot(HaveOccurred())
Expect(got.Path).To(Equal("new/song.mp3")) // moved to target's location
Expect(got.Missing).To(BeFalse())
Expect(got.CreatedAt).To(BeTemporally("==", created)) // created_at preserved
exists, _ := mfRepo.Exists("t1")
Expect(exists).To(BeFalse()) // discarded row removed
Expect(ds.GCCalled).To(BeTrue())
})
It("moves the target's annotations, bookmarks and playlist entries onto the surviving id", func() {
mfRepo.SetData(model.MediaFiles{
{ID: "m1", AlbumID: "album1", Missing: true},
{ID: "t1", AlbumID: "album1", Missing: false},
})
Expect(service.RemapMissingFile(ctx, "m1", "t1")).To(Succeed())
Expect(mfRepo.ReassignReferencesCalls).To(HaveKeyWithValue("t1", "m1"))
})
It("reassigns album annotations when the old album is emptied", func() {
albumRepo := ds.MockedAlbum.(*extendedAlbumRepo)
mfRepo.SetData(model.MediaFiles{
{ID: "m1", AlbumID: "album1", Missing: true},
{ID: "t1", AlbumID: "album2", Missing: false},
})
mfRepo.SetCountAll(0)
Expect(service.RemapMissingFile(ctx, "m1", "t1")).To(Succeed())
Expect(albumRepo.ReassignAnnotationCalls).To(HaveKeyWithValue("album1", "album2"))
})
It("does not reassign album annotations when the old album is not emptied", func() {
albumRepo := ds.MockedAlbum.(*extendedAlbumRepo)
mfRepo.SetData(model.MediaFiles{
{ID: "m1", AlbumID: "album1", Missing: true},
{ID: "m2", AlbumID: "album1", Missing: false},
{ID: "t1", AlbumID: "album2", Missing: false},
})
Expect(service.RemapMissingFile(ctx, "m1", "t1")).To(Succeed())
Expect(albumRepo.ReassignAnnotationCalls).To(BeEmpty())
})
It("does not reassign annotations when the album is unchanged", func() {
albumRepo := ds.MockedAlbum.(*extendedAlbumRepo)
mfRepo.SetData(model.MediaFiles{
{ID: "m1", AlbumID: "album1", Missing: true},
{ID: "t1", AlbumID: "album1", Missing: false},
})
Expect(service.RemapMissingFile(ctx, "m1", "t1")).To(Succeed())
Expect(albumRepo.ReassignAnnotationCalls).To(BeEmpty())
})
It("returns ErrNotFound when the missing file does not exist", func() {
mfRepo.SetData(model.MediaFiles{{ID: "t1", Missing: false}})
Expect(service.RemapMissingFile(ctx, "nope", "t1")).To(MatchError(model.ErrNotFound))
})
It("refuses to remap from a file that is not missing", func() {
mfRepo.SetData(model.MediaFiles{
{ID: "m1", Missing: false},
{ID: "t1", Missing: false},
})
Expect(service.RemapMissingFile(ctx, "m1", "t1")).To(MatchError(ErrNotMissing))
})
It("refuses to remap a file onto itself", func() {
mfRepo.SetData(model.MediaFiles{{ID: "m1", Missing: true}})
Expect(service.RemapMissingFile(ctx, "m1", "m1")).To(MatchError(ErrSameFile))
})
It("refuses to remap onto a target that is itself missing", func() {
mfRepo.SetData(model.MediaFiles{
{ID: "m1", Missing: true},
{ID: "t1", Missing: true},
})
Expect(service.RemapMissingFile(ctx, "m1", "t1")).To(MatchError(ErrTargetMissing))
})
It("refreshes artist and album stats right after the remap", func() {
artistRepo := ds.MockedArtist.(*extendedArtistRepo)
albumRepo := ds.MockedAlbum.(*extendedAlbumRepo)
albumRepo.SetData(model.Albums{
{ID: "album1", Name: "Old Album", SongCount: 2, Size: 1100, Duration: 110},
{ID: "album2", Name: "New Album", SongCount: 1, Size: 2000, Duration: 200},
})
mfRepo.SetData(model.MediaFiles{
{ID: "m1", Path: "old/1.mp3", Album: "Old Album", AlbumID: "album1", Missing: true, Size: 100, Duration: 10},
{ID: "k1", Path: "old/2.mp3", Album: "Old Album", AlbumID: "album1", Missing: false, Size: 1000, Duration: 100},
{ID: "t1", Path: "new/1.mp3", Album: "New Album", AlbumID: "album2", Missing: false, Size: 2000, Duration: 200},
})
Expect(service.RemapMissingFile(ctx, "m1", "t1")).To(Succeed())
Expect(artistRepo.IsRefreshStatsCalled()).To(BeTrue(), "Artist stats should be refreshed")
// The old album lost the remapped track, so its stats are recalculated from the remaining one
oldAlbum, err := albumRepo.Get("album1")
Expect(err).ToNot(HaveOccurred())
Expect(oldAlbum.SongCount).To(Equal(1))
Expect(oldAlbum.Size).To(Equal(int64(1000)))
Expect(oldAlbum.Duration).To(BeNumerically("==", 100))
// The target album keeps the track, now under the missing file's ID
newAlbum, err := albumRepo.Get("album2")
Expect(err).ToNot(HaveOccurred())
Expect(newAlbum.SongCount).To(Equal(1))
Expect(newAlbum.Size).To(Equal(int64(2000)))
Expect(newAlbum.Duration).To(BeNumerically("==", 200))
})
It("returns an error if GC fails", func() {
mfRepo.SetData(model.MediaFiles{
{ID: "m1", AlbumID: "album1", Missing: true},
{ID: "t1", AlbumID: "album1", Missing: false},
})
ds.GCError = errors.New("gc failed")
Expect(service.RemapMissingFile(ctx, "m1", "t1")).To(MatchError(ContainSubstring("gc failed")))
})
It("preserves the target's participants on the remapped track", func() {
participant := model.Participant{
Artist: model.Artist{ID: "a1", Name: "Artist", OrderArtistName: "artist", MbzArtistID: "mbz-artist"},
}
mfRepo.SetData(model.MediaFiles{
{ID: "m1", AlbumID: "album1", Missing: true},
{ID: "t1", AlbumID: "album2", Missing: false, Participants: model.Participants{
model.RoleArtist: model.ParticipantList{participant},
}},
})
Expect(service.RemapMissingFile(ctx, "m1", "t1")).To(Succeed())
// The surviving row is the missing file's ID, holding the target's data
got, err := mfRepo.GetWithParticipants("m1")
Expect(err).ToNot(HaveOccurred())
Expect(got.Participants).To(HaveKeyWithValue(model.RoleArtist, model.ParticipantList{participant}))
})
})
})
// Test helper to create a mock DataStore with controllable behavior

3
model/mediafile.go
View file

@ -563,6 +563,9 @@ type MediaFileRepository interface {
DeleteMissing(ids []string) error
DeleteAllMissing() (int64, error)
FindByPaths(paths []string) (MediaFiles, error)
// ReassignReferences moves annotations, bookmarks and playlist entries from prevID to newID,
// keeping newID's own row wherever a user has both.
ReassignReferences(prevID, newID string) error
// The following methods are used exclusively by the scanner:
MarkMissing(bool, ...*MediaFile) error

40
persistence/mediafile_repository.go
View file

@ -362,19 +362,14 @@ func (r *mediaFileRepository) FindByPaths(paths []string) (model.MediaFiles, err
var unqualified []string
for _, path := range paths {
parts := strings.SplitN(path, ":", 2)
if len(parts) == 2 {
// Library-qualified path: "libraryID:path"
libraryID, err := strconv.Atoi(parts[0])
if err != nil {
// Invalid format, skip
continue
// A numeric prefix is ambiguous: "1:foo.mp3" qualifies a library, but "1999: A Life/01.mp3"
// is a plain path. Search both ways rather than guessing.
if id, rest, ok := strings.Cut(path, ":"); ok {
if libraryID, err := strconv.Atoi(id); err == nil {
byLibrary[libraryID] = append(byLibrary[libraryID], rest)
}
byLibrary[libraryID] = append(byLibrary[libraryID], parts[1])
} else {
// Unqualified path: search across all libraries
unqualified = append(unqualified, path)
}
unqualified = append(unqualified, path)
}
query := Or{}
@ -405,6 +400,29 @@ func (r *mediaFileRepository) Delete(id string) error {
return r.delete(Eq{"id": id})
}
func (r *mediaFileRepository) ReassignReferences(prevID, newID string) error {
if err := r.ReassignAnnotation(prevID, newID); err != nil {
return fmt.Errorf("reassigning annotations: %w", err)
}
if err := r.reassignBookmark(prevID, newID); err != nil {
return fmt.Errorf("reassigning bookmarks: %w", err)
}
upd := Update("playlist_tracks").Set("media_file_id", newID).Where(Eq{"media_file_id": prevID})
if _, err := r.executeSQL(upd); err != nil {
return fmt.Errorf("reassigning playlist tracks: %w", err)
}
upd = Update("scrobbles").Set("media_file_id", newID).Where(Eq{"media_file_id": prevID})
if _, err := r.executeSQL(upd); err != nil {
return fmt.Errorf("reassigning scrobbles: %w", err)
}
// OR IGNORE: scrobble_buffer is unique on (user_id, service, media_file_id, play_time)
buf := Expr("update or ignore scrobble_buffer set media_file_id = ? where media_file_id = ?", newID, prevID)
if _, err := r.executeSQL(buf); err != nil {
return fmt.Errorf("reassigning buffered scrobbles: %w", err)
}
return nil
}
func (r *mediaFileRepository) DeleteAllMissing() (int64, error) {
user := loggedUser(r.ctx)
if !user.IsAdmin {

123
persistence/mediafile_repository_test.go
View file

@ -16,6 +16,7 @@ import (
"github.com/navidrome/navidrome/model/criteria"
"github.com/navidrome/navidrome/model/id"
"github.com/navidrome/navidrome/model/request"
"github.com/navidrome/navidrome/utils/slice"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
"github.com/pocketbase/dbx"
@ -918,6 +919,105 @@ var _ = Describe("MediaRepository", func() {
})
})
Describe("ReassignReferences", func() {
var prev, next model.MediaFile
var pr model.PlaylistRepository
var pls model.Playlist
BeforeEach(func() {
ctx := request.WithUser(log.NewContext(context.TODO()), model.User{ID: "userid"})
pr = NewPlaylistRepository(ctx, GetDBXBuilder())
prev = model.MediaFile{ID: "reassign-prev", LibraryID: 1, Path: "reassign/prev.mp3", Title: "Prev"}
next = model.MediaFile{ID: "reassign-next", LibraryID: 1, Path: "reassign/next.mp3", Title: "Next"}
Expect(mr.Put(&prev)).To(Succeed())
Expect(mr.Put(&next)).To(Succeed())
pls = model.Playlist{Name: "Reassign", OwnerID: "userid"}
pls.AddMediaFilesByID([]string{prev.ID})
Expect(pr.Put(&pls)).To(Succeed())
})
AfterEach(func() {
_ = pr.Delete(pls.ID)
_ = mr.Delete(prev.ID)
_ = mr.Delete(next.ID)
_, _ = mr.(*mediaFileRepository).executeSQL(squirrel.Delete("annotation").Where(squirrel.Eq{"item_id": []string{prev.ID, next.ID}}))
_, _ = mr.(*mediaFileRepository).executeSQL(squirrel.Delete("bookmark").Where(squirrel.Eq{"item_id": []string{prev.ID, next.ID}}))
_, _ = mr.(*mediaFileRepository).executeSQL(squirrel.Delete("scrobbles").Where(squirrel.Eq{"media_file_id": []string{prev.ID, next.ID}}))
_, _ = mr.(*mediaFileRepository).executeSQL(squirrel.Delete("scrobble_buffer").Where(squirrel.Eq{"media_file_id": []string{prev.ID, next.ID}}))
})
It("moves annotations, bookmarks and playlist entries onto the new id", func() {
Expect(mr.SetRating(5, prev.ID)).To(Succeed())
Expect(mr.AddBookmark(prev.ID, "here", 42)).To(Succeed())
Expect(mr.ReassignReferences(prev.ID, next.ID)).To(Succeed())
got, err := mr.Get(next.ID)
Expect(err).ToNot(HaveOccurred())
Expect(got.Rating).To(Equal(5))
bookmarks, err := mr.GetBookmarks()
Expect(err).ToNot(HaveOccurred())
Expect(bookmarks).To(ContainElement(HaveField("Item.ID", next.ID)))
withTracks, err := pr.GetWithTracks(pls.ID, false, false)
Expect(err).ToNot(HaveOccurred())
Expect(withTracks.Tracks).To(HaveLen(1))
Expect(withTracks.Tracks[0].MediaFileID).To(Equal(next.ID))
})
It("moves scrobbles and buffered scrobbles onto the new id", func() {
ctx := request.WithUser(log.NewContext(context.TODO()), model.User{ID: "userid"})
scrobbles := NewScrobbleRepository(ctx, GetDBXBuilder())
buffer := NewScrobbleBufferRepository(ctx, GetDBXBuilder())
Expect(scrobbles.RecordScrobble(prev.ID, time.Now())).To(Succeed())
Expect(buffer.Enqueue("lastfm", "userid", prev.ID, time.Now())).To(Succeed())
Expect(mr.ReassignReferences(prev.ID, next.ID)).To(Succeed())
Expect(mr.Delete(prev.ID)).To(Succeed())
all, err := scrobbles.GetAll()
Expect(err).ToNot(HaveOccurred())
mine := slice.Map(slice.Filter(all, func(sc model.Scrobble) bool {
return sc.MediaFileID == prev.ID || sc.MediaFileID == next.ID
}), func(sc model.Scrobble) string { return sc.MediaFileID })
Expect(mine).To(ConsistOf(next.ID))
entry, err := buffer.Next("lastfm", "userid")
Expect(err).ToNot(HaveOccurred())
Expect(entry).ToNot(BeNil())
Expect(entry.MediaFile.ID).To(Equal(next.ID))
})
It("recomputes the average rating after merging another user's annotation", func() {
other := NewMediaFileRepository(request.WithUser(log.NewContext(context.TODO()), model.User{ID: "2222"}), GetDBXBuilder())
Expect(mr.SetRating(5, next.ID)).To(Succeed())
Expect(other.SetRating(3, prev.ID)).To(Succeed())
Expect(mr.ReassignReferences(prev.ID, next.ID)).To(Succeed())
got, err := mr.Get(next.ID)
Expect(err).ToNot(HaveOccurred())
Expect(got.AverageRating).To(Equal(4.0))
})
It("keeps the new id's own annotation and bookmark when both exist", func() {
Expect(mr.SetRating(5, prev.ID)).To(Succeed())
Expect(mr.SetRating(1, next.ID)).To(Succeed())
Expect(mr.AddBookmark(prev.ID, "prev", 42)).To(Succeed())
Expect(mr.AddBookmark(next.ID, "next", 7)).To(Succeed())
Expect(mr.ReassignReferences(prev.ID, next.ID)).To(Succeed())
got, err := mr.Get(next.ID)
Expect(err).ToNot(HaveOccurred())
Expect(got.Rating).To(Equal(1))
bookmarks, err := mr.GetBookmarks()
Expect(err).ToNot(HaveOccurred())
Expect(bookmarks).To(ContainElement(SatisfyAll(HaveField("Item.ID", next.ID), HaveField("Comment", "next"))))
})
})
Describe("FindByPaths", func() {
// Test fixtures for Unicode and case-sensitivity tests
var testFiles []model.MediaFile
@ -930,6 +1030,8 @@ var _ = Describe("MediaRepository", func() {
{ID: "findpath-3", LibraryID: 1, Path: "plex/02 - ＡＣＲＯＳＳ.flac", Title: "Fullwidth"},
// French diacritic: è (U+00E8, can decompose to e + combining grave)
{ID: "findpath-4", LibraryID: 1, Path: "artist/Michèle/song.mp3", Title: "French"},
{ID: "findpath-5", LibraryID: 1, Path: "Bach: Goldberg Variations/01.mp3", Title: "Colon"},
{ID: "findpath-6", LibraryID: 1, Path: "1999: A Different Life/01.mp3", Title: "Numeric colon"},
}
for _, mf := range testFiles {
Expect(mr.Put(&mf)).To(Succeed())
@ -942,6 +1044,27 @@ var _ = Describe("MediaRepository", func() {
}
})
It("treats a path whose prefix is not a library id as unqualified", func() {
results, err := mr.FindByPaths([]string{"Bach: Goldberg Variations/01.mp3"})
Expect(err).ToNot(HaveOccurred())
Expect(results).To(HaveLen(1))
Expect(results[0].ID).To(Equal("findpath-5"))
})
It("finds a plain path whose colon prefix looks like a library id", func() {
results, err := mr.FindByPaths([]string{"1999: A Different Life/01.mp3"})
Expect(err).ToNot(HaveOccurred())
Expect(results).To(HaveLen(1))
Expect(results[0].ID).To(Equal("findpath-6"))
})
It("splits only the first colon of a library-qualified path", func() {
results, err := mr.FindByPaths([]string{"1:Bach: Goldberg Variations/01.mp3"})
Expect(err).ToNot(HaveOccurred())
Expect(results).To(HaveLen(1))
Expect(results[0].ID).To(Equal("findpath-5"))
})
It("finds files by exact path", func() {
results, err := mr.FindByPaths([]string{"1:artist/Album/track.mp3"})
Expect(err).ToNot(HaveOccurred())

14
persistence/sql_annotations.go
View file

@ -185,12 +185,14 @@ func (r sqlRepository) ReassignAnnotation(prevID string, newID string) error {
if prevID == newID || prevID == "" || newID == "" {
return nil
}
upd := Update(annotationTable).Where(And{
Eq{annotationTable + ".item_type": r.tableName},
Eq{annotationTable + ".item_id": prevID},
}).Set("item_id", newID)
_, err := r.executeSQL(upd)
return err
// OR IGNORE keeps newID's own row where a user annotated both, instead of aborting the whole statement
upd := Expr("update or ignore "+annotationTable+" set item_id = ? where item_type = ? and item_id = ?",
newID, r.tableName, prevID)
if _, err := r.executeSQL(upd); err != nil {
return err
}
// The moved rows change newID's rating population, so its cached average no longer matches
return r.updateAvgRating(newID)
}
func (r sqlRepository) cleanAnnotations() error {

47
persistence/sql_annotations_test.go
View file

@ -30,6 +30,53 @@ var _ = Describe("Annotation Filters", func() {
_, _ = albumRepo.executeSQL(squirrel.Delete("album").Where(squirrel.Eq{"id": albumWithoutAnnotation.ID}))
})
Describe("ReassignAnnotation", func() {
var prev, next model.Album
BeforeEach(func() {
prev = model.Album{ID: "reassign-prev", Name: "Prev", LibraryID: 1}
next = model.Album{ID: "reassign-next", Name: "Next", LibraryID: 1}
Expect(albumRepo.Put(&prev)).To(Succeed())
Expect(albumRepo.Put(&next)).To(Succeed())
})
AfterEach(func() {
_, _ = albumRepo.executeSQL(squirrel.Delete("annotation").Where(squirrel.Eq{"item_id": []string{prev.ID, next.ID}}))
_, _ = albumRepo.executeSQL(squirrel.Delete("album").Where(squirrel.Eq{"id": []string{prev.ID, next.ID}}))
})
It("moves the annotation when the new item has none", func() {
Expect(albumRepo.SetRating(4, prev.ID)).To(Succeed())
Expect(albumRepo.ReassignAnnotation(prev.ID, next.ID)).To(Succeed())
got, err := albumRepo.Get(next.ID)
Expect(err).ToNot(HaveOccurred())
Expect(got.Rating).To(Equal(4))
})
It("recomputes the new item's cached average rating", func() {
Expect(albumRepo.SetRating(4, prev.ID)).To(Succeed())
Expect(albumRepo.ReassignAnnotation(prev.ID, next.ID)).To(Succeed())
got, err := albumRepo.Get(next.ID)
Expect(err).ToNot(HaveOccurred())
Expect(got.AverageRating).To(Equal(4.0))
})
It("keeps the new item's annotation when both exist", func() {
Expect(albumRepo.SetRating(4, prev.ID)).To(Succeed())
Expect(albumRepo.SetRating(2, next.ID)).To(Succeed())
Expect(albumRepo.ReassignAnnotation(prev.ID, next.ID)).To(Succeed())
got, err := albumRepo.Get(next.ID)
Expect(err).ToNot(HaveOccurred())
Expect(got.Rating).To(Equal(2))
})
})
Describe("annotationBoolFilter", func() {
DescribeTable("creates correct SQL expressions",
func(field, value string, expectedSQL string, expectedArgs []any) {

7
persistence/sql_bookmarks.go
View file

@ -144,6 +144,13 @@ func (r sqlRepository) GetBookmarks() (model.Bookmarks, error) {
return resp, nil
}
func (r sqlRepository) reassignBookmark(prevID, newID string) error {
upd := Expr("update or ignore "+bookmarkTable+" set item_id = ? where item_type = ? and item_id = ?",
newID, r.tableName, prevID)
_, err := r.executeSQL(upd)
return err
}
func (r sqlRepository) cleanBookmarks() error {
del := Delete(bookmarkTable).Where(Eq{"item_type": r.tableName}).Where("item_id not in (select id from " + r.tableName + ")")
c, err := r.executeSQL(del)

6
scanner/phase_2_missing_tracks_test.go
View file

@ -311,7 +311,7 @@ var _ = Describe("phaseMissingTracks", func() {
When("PurgeMissing is 'always'", func() {
BeforeEach(func() {
conf.Server.Scanner.PurgeMissing = consts.PurgeMissingAlways
mr.CountAllValue = 3
mr.SetCountAll(3)
mr.DeleteAllMissingValue = 3
})
It("should purge missing files", func() {
@ -325,7 +325,7 @@ var _ = Describe("phaseMissingTracks", func() {
When("PurgeMissing is 'full'", func() {
BeforeEach(func() {
conf.Server.Scanner.PurgeMissing = consts.PurgeMissingFull
mr.CountAllValue = 2
mr.SetCountAll(2)
mr.DeleteAllMissingValue = 2
})
It("should not purge missing files if not a full scan", func() {
@ -346,7 +346,7 @@ var _ = Describe("phaseMissingTracks", func() {
When("PurgeMissing is 'never'", func() {
BeforeEach(func() {
conf.Server.Scanner.PurgeMissing = consts.PurgeMissingNever
mr.CountAllValue = 1
mr.SetCountAll(1)
mr.DeleteAllMissingValue = 1
})
It("should not purge missing files", func() {

34
tests/mock_mediafile_repo.go
View file

@ -24,11 +24,14 @@ type MockMediaFileRepo struct {
model.MediaFileRepository
Data map[string]*model.MediaFile
Err bool
// Add fields and methods for controlling CountAll and DeleteAllMissing in tests
CountAllValue int64
// Add fields and methods for controlling CountAll and DeleteAllMissing in tests.
// A nil CountAllValue is unset, and CountAll falls back to counting rows in Data.
CountAllValue *int64
CountAllOptions model.QueryOptions
DeleteAllMissingValue int64
Options model.QueryOptions
// ReassignReferencesCalls records prevID -> newID
ReassignReferencesCalls map[string]string
Options model.QueryOptions
// Add fields for cross-library move detection tests
FindRecentFilesByMBZTrackIDFunc func(missing model.MediaFile, since time.Time) (model.MediaFiles, error)
FindRecentFilesByPropertiesFunc func(missing model.MediaFile, since time.Time) (model.MediaFiles, error)
@ -40,6 +43,10 @@ func (m *MockMediaFileRepo) SetError(err bool) {
m.Err = err
}
func (m *MockMediaFileRepo) SetCountAll(count int64) {
m.CountAllValue = &count
}
func (m *MockMediaFileRepo) SetData(mfs model.MediaFiles) {
m.Data = make(map[string]*model.MediaFile)
for i, mf := range mfs {
@ -163,6 +170,17 @@ func (m *MockMediaFileRepo) Delete(id string) error {
return nil
}
func (m *MockMediaFileRepo) ReassignReferences(prevID, newID string) error {
if m.Err {
return errors.New("error")
}
if m.ReassignReferencesCalls == nil {
m.ReassignReferencesCalls = make(map[string]string)
}
m.ReassignReferencesCalls[prevID] = newID
return nil
}
func (m *MockMediaFileRepo) IncPlayCount(id string, timestamp time.Time) error {
if m.Err {
return errors.New("error")
@ -251,11 +269,11 @@ func (m *MockMediaFileRepo) CountAll(opts ...model.QueryOptions) (int64, error)
if m.Err {
return 0, errors.New("error")
}
if m.CountAllValue != 0 {
if len(opts) > 0 {
m.CountAllOptions = opts[0]
}
return m.CountAllValue, nil
if len(opts) > 0 {
m.CountAllOptions = opts[0]
}
if m.CountAllValue != nil {
return *m.CountAllValue, nil
}
return int64(len(m.Data)), nil
}