mirror of
https://github.com/navidrome/navidrome.git
synced 2026-10-08 10:27:08 +02:00
* fix(artwork): pause the artwork worker while a scan is running The artwork worker added in 0.64 writes to the database continuously, including while a scan runs. On slow storage the scanner holds the write lock for many seconds per folder, so the two writers keep timing each other out: artwork writes fail with "database is locked", and a single busy timeout on the scanner side aborts the whole scan. The worker now stops dispatching queue items while scanner.IsScanning reports true, including mid-batch, and resumes on the next poll after the scan ends. Artwork requests are unaffected, since they serve local art without the worker. * fix(db): run ANALYZE one index at a time so writers are not starved A full ANALYZE is a single write transaction, so every other write waits for it to finish and fails after the 15s busy timeout. On slow NAS storage it was measured taking over 26 minutes. The analysis now runs ANALYZE per index (per table for unindexed and WITHOUT ROWID tables), which produces the same sqlite_stat1 rows as a full ANALYZE, and pauses briefly between steps (up to 150ms, just above SQLite's longest busy-handler sleep) so waiting writers get the lock. * fix(scanner): ignore Synology @eaDir metadata folders Synology creates an @eaDir folder next to media files, holding one subfolder per file with generated thumbnails. The scanner and watcher treated them as regular folders, which on one reported library added tens of thousands of extra folders to every scan. * fix(db): analyze tables with only partial indexes as a whole A partial index does not record the table's row count, so a table whose only indexes are partial needs a table-level ANALYZE to get the sqlite_stat1 row a full ANALYZE would write. Navidrome's schema has no such table today, but the stepped analysis should match a full ANALYZE for any schema a future migration creates. * fix(scanner): retry busy folder saves and stop phase 1 on a fatal error On slow storage, a single SQLITE_BUSY while saving a folder aborted the whole scan, even when another writer held the lock only briefly. The folder save now runs as a retryable unit: on a busy error it waits (5s, 10s, 15s) and reruns the transaction, up to three times, before failing. Side effects that do not survive a rollback (the album ID map consumed by persistAlbum, the artwork queue items, the image-change record) are rebuilt per attempt or recorded only after a successful commit. When a folder save does fail, phase 1 used to keep walking the library and reading tags for every remaining folder, discarding the results, before reporting the error; a reporter saw 40 silent minutes. The walk now stops as soon as the save fails, and the walker honors cancellation instead of blocking on its channel. Because an early stop leaves folders unvisited, phase 1 no longer marks unvisited folders missing when the phase failed; the resumed scan handles them. * refactor(persistence): move busy retry into DataStore.WithTxRetry The scanner retried its folder save itself, which meant it had to know SQLite error codes. WithTxRetry now owns that policy: it reruns the block in a fresh transaction on SQLITE_BUSY, up to three times with growing delays, and runs it only once when already inside a transaction, since the outer transaction would still hold the lock. The block receives the context to use, and attempts that will be retried carry a marker so a busy statement in them is logged as a warning; only the final attempt logs errors. The scanner's inner error logs are folded into wrapped errors, so a recovered retry no longer prints error-level lines, and the folder path travels in the log context. * fix(persistence): join the enclosing transaction in a nested WithTxRetry Called on a store that is already inside a transaction, WithTxRetry went through WithTx, which opens a second, independent transaction on another connection. That transaction waits on the lock the outer one holds and fails with SQLITE_BUSY, and if it does succeed the outer transaction cannot roll it back. It now runs the block on the enclosing transaction, which owns the lock, the commit and the rollback. Found by a Codex (gpt-6-sol) review. * fix(scanner): retry the remaining scan writes on a busy database Every write step after phase 1 still aborted the whole scan on a single SQLITE_BUSY: phase 1 finalize, phase 2 moves and purge, phase 3 album saves and play count refreshes, the deferred playlist import flag, library ScanBegin, GC, the missing-artwork enqueue, tag counts, and the final library update. They now go through WithTxRetry. The phase 2 move had to be made rerun-safe first: it changed the target track's ID inside the transaction, so a rerun would have deleted the moved track itself, and it marked album annotations as handled even when the transaction rolled back. It now works on a copy per attempt and records the annotation reassignment only after a commit. Artist.RefreshStats is left alone: it updates artists in batches outside a transaction, and one transaction around all of them would hold the write lock for the whole refresh on slow storage. Phase 4 playlist imports go through the playlist service and are left for a follow-up. * fix(scanner): claim the album before moving its annotations The rerun-safe moveMatched checked processedAlbumAnnotations before its transaction and marked the album only after the commit. Phase 2 runs same-library and cross-library moves in separate pipeline stages, so two moves into one album could both pass the check; the second would reassign annotations again and overwrite the album's created_at. The album is now claimed under the lock before the transaction, as the old code effectively did, and the claim is released if the move fails so a later move can still reassign. Found by a Codex (gpt-6-sol) review. * fix(artwork): keep artwork housekeeping from writing during scans The artwork worker already pauses while a scan runs, but its housekeeping jobs did not: the hourly missing-artwork recheck (a bulk INSERT ... SELECT over albums and artists), the startup run of the same recheck, and the daily prune all kept competing with the scanner for the write lock. They now run through LockForMaintenance, like the scheduled DB analysis: they skip while a scan is running and keep a scan from starting until they finish. Skipping the recheck loses nothing, since each scan with changes queues missing artwork at its end. * refactor(scanner): log retried step errors once, from the caller Blocks passed to WithTxRetry still logged their own errors at error level on every attempt, so a busy error that a retry absorbed printed several error lines (GC printed three). They now return wrapped errors and the callers, which already log them, report the final outcome once. Also: drop a leftover variable in phase 1 finalize, check the walk context once, stop repeating the folder field that is already in the log context, stop shadowing finalize's err in phase 3, and format the WithTxRetry scope the same way as WithTx. * test(scanner): make the scanner suite's temp DB cleanup best effort Which DB file the process-wide DB handle opens depends on which spec touches it first. When the Scanner container wins the random order, its temp DB stays open until db.Close after RunSpecs, and on Windows removing the temp dir fails with 'being used by another process'. Ginkgo pins that on the container's last spec, which is now one of the busy-database specs. The sibling suites skip Windows for the same reason; this one now removes its temp dir on a best-effort basis instead, so it keeps running there.
780 lines
25 KiB
Go
780 lines
25 KiB
Go
package scanner
|
|
|
|
import (
|
|
"context"
|
|
"fmt"
|
|
"io/fs"
|
|
"os"
|
|
"path/filepath"
|
|
"testing/fstest"
|
|
|
|
"github.com/navidrome/navidrome/conf"
|
|
"github.com/navidrome/navidrome/conf/configtest"
|
|
"github.com/navidrome/navidrome/core/storage"
|
|
"github.com/navidrome/navidrome/model"
|
|
"github.com/navidrome/navidrome/tests"
|
|
. "github.com/onsi/ginkgo/v2"
|
|
. "github.com/onsi/gomega"
|
|
"golang.org/x/sync/errgroup"
|
|
)
|
|
|
|
var _ = Describe("walk_dir_tree", func() {
|
|
Describe("walkDirTree", func() {
|
|
var (
|
|
fsys storage.MusicFS
|
|
job *scanJob
|
|
ctx context.Context
|
|
)
|
|
|
|
Context("full library", func() {
|
|
BeforeEach(func() {
|
|
DeferCleanup(configtest.SetupConfig())
|
|
ctx = GinkgoT().Context()
|
|
fsys = &mockMusicFS{
|
|
FS: fstest.MapFS{
|
|
"root/a/.ndignore": {Data: []byte("ignored/*")},
|
|
"root/a/f1.mp3": {},
|
|
"root/a/f2.mp3": {},
|
|
"root/a/ignored/bad.mp3": {},
|
|
"root/b/cover.jpg": {},
|
|
"root/c/f3": {},
|
|
"root/d": {},
|
|
"root/d/.ndignore": {},
|
|
"root/d/f1.mp3": {},
|
|
"root/d/f2.mp3": {},
|
|
"root/d/f3.mp3": {},
|
|
"root/e/original/f1.mp3": {},
|
|
"root/e/symlink": {Mode: fs.ModeSymlink, Data: []byte("original")},
|
|
"root/f/realsong.mp3": {Data: []byte("AUDIO")},
|
|
"root/f/legit.mp3": {Mode: fs.ModeSymlink, Data: []byte("realsong.mp3")},
|
|
"root/f/secret": {Data: []byte("TOPSECRET")},
|
|
"root/f/evil.mp3": {Mode: fs.ModeSymlink, Data: []byte("secret")},
|
|
"root/g/.Hack Sign Original Soundtrack/track.mp3": {},
|
|
"root/h/.hidden.mp3": {},
|
|
"root/i/.git/config": {},
|
|
"root/i/.streams/stream.mp3": {},
|
|
},
|
|
}
|
|
job = &scanJob{
|
|
fs: fsys,
|
|
lib: model.Library{Path: "/music"},
|
|
}
|
|
})
|
|
|
|
// Helper function to call walkDirTree and collect folders from the results channel
|
|
getFolders := func() map[string]*folderEntry {
|
|
results, err := walkDirTree(ctx, job)
|
|
Expect(err).ToNot(HaveOccurred())
|
|
|
|
folders := map[string]*folderEntry{}
|
|
g := errgroup.Group{}
|
|
g.Go(func() error {
|
|
for folder := range results {
|
|
folders[folder.path] = folder
|
|
}
|
|
return nil
|
|
})
|
|
_ = g.Wait()
|
|
return folders
|
|
}
|
|
|
|
DescribeTable("symlink handling",
|
|
func(followSymlinks bool, expectedFolderCount int) {
|
|
conf.Server.Scanner.FollowSymlinks = followSymlinks
|
|
folders := getFolders()
|
|
|
|
Expect(folders).To(HaveLen(expectedFolderCount + 2)) // +2 for `.` and `root`
|
|
|
|
// Basic folder structure checks
|
|
Expect(folders["root/a"].audioFiles).To(SatisfyAll(
|
|
HaveLen(2),
|
|
HaveKey("f1.mp3"),
|
|
HaveKey("f2.mp3"),
|
|
))
|
|
Expect(folders["root/a"].imageFiles).To(BeEmpty())
|
|
Expect(folders["root/b"].audioFiles).To(BeEmpty())
|
|
Expect(folders["root/b"].imageFiles).To(SatisfyAll(
|
|
HaveLen(1),
|
|
HaveKey("cover.jpg"),
|
|
))
|
|
Expect(folders["root/c"].audioFiles).To(BeEmpty())
|
|
Expect(folders["root/c"].imageFiles).To(BeEmpty())
|
|
Expect(folders).ToNot(HaveKey("root/d"))
|
|
|
|
// By default (Scanner.IgnoreDotFolders == true), dot-prefixed
|
|
// folders are skipped, dot-prefixed files are not indexed, and
|
|
// the special ignoredDirs (.git, .streams) are never traversed.
|
|
Expect(folders).ToNot(HaveKey("root/g/.Hack Sign Original Soundtrack"))
|
|
Expect(folders["root/h"].audioFiles).To(BeEmpty())
|
|
Expect(folders).ToNot(HaveKey("root/i/.git"))
|
|
Expect(folders).ToNot(HaveKey("root/i/.streams"))
|
|
|
|
// Symlink specific checks
|
|
if followSymlinks {
|
|
Expect(folders["root/e/symlink"].audioFiles).To(HaveLen(1))
|
|
Expect(folders["root/f"].audioFiles).To(HaveKey("legit.mp3"))
|
|
Expect(folders["root/f"].audioFiles).To(HaveKey("realsong.mp3"))
|
|
Expect(folders["root/f"].audioFiles).ToNot(HaveKey("evil.mp3"))
|
|
} else {
|
|
Expect(folders).ToNot(HaveKey("root/e/symlink"))
|
|
Expect(folders["root/f"].audioFiles).To(HaveKey("realsong.mp3"))
|
|
Expect(folders["root/f"].audioFiles).ToNot(HaveKey("legit.mp3"))
|
|
Expect(folders["root/f"].audioFiles).ToNot(HaveKey("evil.mp3"))
|
|
}
|
|
},
|
|
Entry("with symlinks enabled", true, 11),
|
|
Entry("with symlinks disabled", false, 10),
|
|
)
|
|
|
|
DescribeTable("dot-prefixed folders with IgnoreDotFolders disabled",
|
|
func(followSymlinks bool) {
|
|
conf.Server.Scanner.FollowSymlinks = followSymlinks
|
|
conf.Server.Scanner.IgnoreDotFolders = false
|
|
folders := getFolders()
|
|
|
|
// Dot-prefixed album folders are now traversed and indexed
|
|
Expect(folders["root/g/.Hack Sign Original Soundtrack"].audioFiles).To(SatisfyAll(
|
|
HaveLen(1),
|
|
HaveKey("track.mp3"),
|
|
))
|
|
|
|
// Dot-prefixed files are still ignored, even with the flag off
|
|
Expect(folders["root/h"].audioFiles).To(BeEmpty())
|
|
|
|
// Special ignoredDirs remain blocked regardless of the flag
|
|
Expect(folders).ToNot(HaveKey("root/i/.git"))
|
|
Expect(folders).ToNot(HaveKey("root/i/.streams"))
|
|
},
|
|
Entry("with symlinks enabled", true),
|
|
Entry("with symlinks disabled", false),
|
|
)
|
|
})
|
|
|
|
Context("with target folders", func() {
|
|
BeforeEach(func() {
|
|
DeferCleanup(configtest.SetupConfig())
|
|
ctx = GinkgoT().Context()
|
|
fsys = &mockMusicFS{
|
|
FS: fstest.MapFS{
|
|
"Artist/Album1/track1.mp3": {},
|
|
"Artist/Album1/track2.mp3": {},
|
|
"Artist/Album2/track1.mp3": {},
|
|
"Artist/Album2/track2.mp3": {},
|
|
"Artist/Album2/Sub/track3.mp3": {},
|
|
"OtherArtist/Album3/track1.mp3": {},
|
|
},
|
|
}
|
|
job = &scanJob{
|
|
fs: fsys,
|
|
lib: model.Library{Path: "/music"},
|
|
}
|
|
})
|
|
|
|
It("should recursively walk all subdirectories of target folders", func() {
|
|
results, err := walkDirTree(ctx, job, "Artist")
|
|
Expect(err).ToNot(HaveOccurred())
|
|
|
|
folders := map[string]*folderEntry{}
|
|
g := errgroup.Group{}
|
|
g.Go(func() error {
|
|
for folder := range results {
|
|
folders[folder.path] = folder
|
|
}
|
|
return nil
|
|
})
|
|
_ = g.Wait()
|
|
|
|
// Should include the target folder and all its descendants
|
|
Expect(folders).To(SatisfyAll(
|
|
HaveKey("Artist"),
|
|
HaveKey("Artist/Album1"),
|
|
HaveKey("Artist/Album2"),
|
|
HaveKey("Artist/Album2/Sub"),
|
|
))
|
|
|
|
// Should not include folders outside the target
|
|
Expect(folders).ToNot(HaveKey("OtherArtist"))
|
|
Expect(folders).ToNot(HaveKey("OtherArtist/Album3"))
|
|
|
|
// Verify audio files are present
|
|
Expect(folders["Artist/Album1"].audioFiles).To(HaveLen(2))
|
|
Expect(folders["Artist/Album2"].audioFiles).To(HaveLen(2))
|
|
Expect(folders["Artist/Album2/Sub"].audioFiles).To(HaveLen(1))
|
|
})
|
|
|
|
It("should handle multiple target folders", func() {
|
|
results, err := walkDirTree(ctx, job, "Artist/Album1", "OtherArtist")
|
|
Expect(err).ToNot(HaveOccurred())
|
|
|
|
folders := map[string]*folderEntry{}
|
|
g := errgroup.Group{}
|
|
g.Go(func() error {
|
|
for folder := range results {
|
|
folders[folder.path] = folder
|
|
}
|
|
return nil
|
|
})
|
|
_ = g.Wait()
|
|
|
|
// Should include both target folders and their descendants
|
|
Expect(folders).To(SatisfyAll(
|
|
HaveKey("Artist/Album1"),
|
|
HaveKey("OtherArtist"),
|
|
HaveKey("OtherArtist/Album3"),
|
|
))
|
|
|
|
// Should not include other folders
|
|
Expect(folders).ToNot(HaveKey("Artist"))
|
|
Expect(folders).ToNot(HaveKey("Artist/Album2"))
|
|
Expect(folders).ToNot(HaveKey("Artist/Album2/Sub"))
|
|
})
|
|
|
|
It("should skip non-existent target folders and preserve them in lastUpdates", func() {
|
|
// Setup job with lastUpdates for both existing and non-existing folders
|
|
job.lastUpdates = map[string]model.FolderUpdateInfo{
|
|
model.FolderID(job.lib, "Artist/Album1"): {},
|
|
model.FolderID(job.lib, "NonExistent/DeletedFolder"): {},
|
|
model.FolderID(job.lib, "OtherArtist/Album3"): {},
|
|
}
|
|
|
|
// Try to scan existing folder and non-existing folder
|
|
results, err := walkDirTree(ctx, job, "Artist/Album1", "NonExistent/DeletedFolder")
|
|
Expect(err).ToNot(HaveOccurred())
|
|
|
|
// Collect results
|
|
folders := map[string]struct{}{}
|
|
for folder := range results {
|
|
folders[folder.path] = struct{}{}
|
|
}
|
|
|
|
// Should only include the existing folder
|
|
Expect(folders).To(HaveKey("Artist/Album1"))
|
|
Expect(folders).ToNot(HaveKey("NonExistent/DeletedFolder"))
|
|
|
|
// The non-existent folder should still be in lastUpdates (not removed by popLastUpdate)
|
|
Expect(job.lastUpdates).To(HaveKey(model.FolderID(job.lib, "NonExistent/DeletedFolder")))
|
|
|
|
// The existing folder should have been removed from lastUpdates
|
|
Expect(job.lastUpdates).ToNot(HaveKey(model.FolderID(job.lib, "Artist/Album1")))
|
|
|
|
// Folders not in targets should remain in lastUpdates
|
|
Expect(job.lastUpdates).To(HaveKey(model.FolderID(job.lib, "OtherArtist/Album3")))
|
|
})
|
|
})
|
|
})
|
|
|
|
Describe("helper functions", func() {
|
|
dir, _ := os.Getwd()
|
|
fsys := os.DirFS(dir)
|
|
baseDir := filepath.Join("tests", "fixtures")
|
|
|
|
Describe("isDirOrSymlinkToDir", func() {
|
|
BeforeEach(func() {
|
|
DeferCleanup(configtest.SetupConfig())
|
|
})
|
|
|
|
Context("with symlinks enabled", func() {
|
|
BeforeEach(func() {
|
|
tests.SkipOnWindows("symlink semantics")
|
|
conf.Server.Scanner.FollowSymlinks = true
|
|
})
|
|
|
|
DescribeTable("returns expected result",
|
|
func(dirName string, expected bool) {
|
|
dirEntry := getDirEntry("tests/fixtures", dirName)
|
|
Expect(isDirOrSymlinkToDir(fsys, baseDir, dirEntry)).To(Equal(expected))
|
|
},
|
|
Entry("normal dir", "empty_folder", true),
|
|
Entry("symlink to dir", "symlink2dir", true),
|
|
Entry("regular file", "test.mp3", false),
|
|
Entry("symlink to file", "symlink", false),
|
|
)
|
|
})
|
|
|
|
Context("with symlinks disabled", func() {
|
|
BeforeEach(func() {
|
|
conf.Server.Scanner.FollowSymlinks = false
|
|
})
|
|
|
|
DescribeTable("returns expected result",
|
|
func(dirName string, expected bool) {
|
|
dirEntry := getDirEntry("tests/fixtures", dirName)
|
|
Expect(isDirOrSymlinkToDir(fsys, baseDir, dirEntry)).To(Equal(expected))
|
|
},
|
|
Entry("normal dir", "empty_folder", true),
|
|
Entry("symlink to dir", "symlink2dir", false),
|
|
Entry("regular file", "test.mp3", false),
|
|
Entry("symlink to file", "symlink", false),
|
|
)
|
|
})
|
|
})
|
|
|
|
Describe("resolveEntryName", func() {
|
|
var fsys fs.FS
|
|
BeforeEach(func() {
|
|
DeferCleanup(configtest.SetupConfig())
|
|
fsys = fstest.MapFS{
|
|
"dir/real.mp3": {Data: []byte("AUDIO")},
|
|
"dir/mid.mp3": {Mode: fs.ModeSymlink, Data: []byte("real.mp3")},
|
|
"dir/chain.mp3": {Mode: fs.ModeSymlink, Data: []byte("mid.mp3")},
|
|
"dir/audio.mp3": {Mode: fs.ModeSymlink, Data: []byte("real.mp3")},
|
|
"dir/evil.mp3": {Mode: fs.ModeSymlink, Data: []byte("../outside/passwd")},
|
|
"dir/loop1.mp3": {Mode: fs.ModeSymlink, Data: []byte("loop2.mp3")},
|
|
"dir/loop2.mp3": {Mode: fs.ModeSymlink, Data: []byte("loop1.mp3")},
|
|
"dir/dangle.mp3": {Mode: fs.ModeSymlink, Data: []byte("missing.mp3")},
|
|
}
|
|
})
|
|
|
|
resolve := func(name string) (string, bool) {
|
|
entries, err := fs.ReadDir(fsys, "dir")
|
|
Expect(err).ToNot(HaveOccurred())
|
|
for _, e := range entries {
|
|
if e.Name() == name {
|
|
return resolveEntryName(GinkgoT().Context(), fsys, "dir", e)
|
|
}
|
|
}
|
|
Fail("entry not found: " + name)
|
|
return "", false
|
|
}
|
|
|
|
Context("with symlinks enabled", func() {
|
|
BeforeEach(func() { conf.Server.Scanner.FollowSymlinks = true })
|
|
|
|
It("returns the entry name for a plain file", func() {
|
|
name, ok := resolve("real.mp3")
|
|
Expect(ok).To(BeTrue())
|
|
Expect(name).To(Equal("real.mp3"))
|
|
})
|
|
It("resolves a direct symlink to its audio target name", func() {
|
|
name, ok := resolve("audio.mp3")
|
|
Expect(ok).To(BeTrue())
|
|
Expect(name).To(Equal("real.mp3"))
|
|
})
|
|
It("resolves a symlink CHAIN to the final target name", func() {
|
|
name, ok := resolve("chain.mp3")
|
|
Expect(ok).To(BeTrue())
|
|
Expect(name).To(Equal("real.mp3"))
|
|
})
|
|
It("resolves a symlink to a non-audio target name (so caller can reject it)", func() {
|
|
name, ok := resolve("evil.mp3")
|
|
Expect(ok).To(BeTrue())
|
|
Expect(name).To(Equal("passwd"))
|
|
})
|
|
It("rejects a symlink loop", func() {
|
|
_, ok := resolve("loop1.mp3")
|
|
Expect(ok).To(BeFalse())
|
|
})
|
|
})
|
|
|
|
Context("with symlinks disabled", func() {
|
|
BeforeEach(func() { conf.Server.Scanner.FollowSymlinks = false })
|
|
|
|
It("returns the entry name for a plain file", func() {
|
|
name, ok := resolve("real.mp3")
|
|
Expect(ok).To(BeTrue())
|
|
Expect(name).To(Equal("real.mp3"))
|
|
})
|
|
It("skips any file symlink", func() {
|
|
_, ok := resolve("audio.mp3")
|
|
Expect(ok).To(BeFalse())
|
|
})
|
|
})
|
|
})
|
|
|
|
Describe("symlink chain (real fs)", func() {
|
|
BeforeEach(func() {
|
|
tests.SkipOnWindows("symlink semantics")
|
|
DeferCleanup(configtest.SetupConfig())
|
|
})
|
|
|
|
classify := func(fsys fs.FS, dirPath, name string) (string, bool) {
|
|
entries, err := fs.ReadDir(fsys, dirPath)
|
|
Expect(err).ToNot(HaveOccurred())
|
|
for _, e := range entries {
|
|
if e.Name() == name {
|
|
return resolveEntryName(GinkgoT().Context(), fsys, dirPath, e)
|
|
}
|
|
}
|
|
Fail("entry not found: " + name)
|
|
return "", false
|
|
}
|
|
|
|
Context("committed 3-level fixtures", func() {
|
|
// tests.Init chdirs to the repo root, so the committed fixtures are at "tests/fixtures".
|
|
var fsys fs.FS
|
|
BeforeEach(func() {
|
|
conf.Server.Scanner.FollowSymlinks = true
|
|
wd, err := os.Getwd()
|
|
Expect(err).ToNot(HaveOccurred())
|
|
fsys = os.DirFS(wd)
|
|
})
|
|
|
|
It("keeps a 3-level chain that resolves to real audio", func() {
|
|
name, ok := classify(fsys, "tests/fixtures/symlink_chain", "level3.mp3")
|
|
Expect(ok).To(BeTrue())
|
|
Expect(name).To(Equal("test.mp3"))
|
|
Expect(model.IsAudioFile(name)).To(BeTrue())
|
|
})
|
|
|
|
It("rejects a 3-level chain that resolves to a non-audio file", func() {
|
|
name, ok := classify(fsys, "tests/fixtures/symlink_chain", "evil3.mp3")
|
|
Expect(ok).To(BeTrue())
|
|
Expect(name).To(Equal("index.html"))
|
|
Expect(model.IsAudioFile(name)).To(BeFalse())
|
|
})
|
|
|
|
It("skips the chain entirely when FollowSymlinks is disabled", func() {
|
|
conf.Server.Scanner.FollowSymlinks = false
|
|
_, ok := classify(fsys, "tests/fixtures/symlink_chain", "level3.mp3")
|
|
Expect(ok).To(BeFalse())
|
|
_, ok = classify(fsys, "tests/fixtures/symlink_chain", "evil3.mp3")
|
|
Expect(ok).To(BeFalse())
|
|
})
|
|
})
|
|
|
|
// Regression for #5752: the production localFS must resolve file symlinks.
|
|
// It wraps os.DirFS behind the fs.FS interface, so fs.ReadLink-based
|
|
// resolution is not available and full OS-level resolution is required.
|
|
Context("production local storage FS", func() {
|
|
var libRoot string
|
|
var musicFS storage.MusicFS
|
|
|
|
BeforeEach(func() {
|
|
conf.Server.Scanner.FollowSymlinks = true
|
|
|
|
// Reproduces the reported layout: a "pool" with the real files and a
|
|
// library containing only symlinks into the pool.
|
|
base := GinkgoT().TempDir()
|
|
pool := filepath.Join(base, "pool")
|
|
libRoot = filepath.Join(base, "userlib")
|
|
Expect(os.MkdirAll(pool, 0755)).To(Succeed())
|
|
Expect(os.MkdirAll(libRoot, 0755)).To(Succeed())
|
|
Expect(os.WriteFile(filepath.Join(pool, "real.mp3"), []byte("AUDIO"), 0600)).To(Succeed())
|
|
Expect(os.WriteFile(filepath.Join(pool, "secrets.txt"), []byte("TOPSECRET"), 0600)).To(Succeed())
|
|
// mid.wav lives OUTSIDE the library and has an audio name, but points at a
|
|
// non-audio file. A chain through it must be classified by the FINAL target.
|
|
Expect(os.Symlink(filepath.Join(pool, "secrets.txt"), filepath.Join(pool, "mid.wav"))).To(Succeed())
|
|
|
|
Expect(os.Symlink("../pool/real.mp3", filepath.Join(libRoot, "relative.mp3"))).To(Succeed())
|
|
Expect(os.Symlink(filepath.Join(pool, "real.mp3"), filepath.Join(libRoot, "absolute.mp3"))).To(Succeed())
|
|
Expect(os.Symlink(filepath.Join(pool, "mid.wav"), filepath.Join(libRoot, "evil.wav"))).To(Succeed())
|
|
Expect(os.Symlink(filepath.Join(pool, "missing.mp3"), filepath.Join(libRoot, "broken.mp3"))).To(Succeed())
|
|
|
|
u, err := storage.LocalPathToURL(libRoot)
|
|
Expect(err).ToNot(HaveOccurred())
|
|
s, err := storage.For(u.String())
|
|
Expect(err).ToNot(HaveOccurred())
|
|
musicFS, err = s.FS()
|
|
Expect(err).ToNot(HaveOccurred())
|
|
})
|
|
|
|
walkRoot := func() *folderEntry {
|
|
job := &scanJob{fs: musicFS, lib: model.Library{Path: libRoot}}
|
|
results, err := walkDirTree(GinkgoT().Context(), job)
|
|
Expect(err).ToNot(HaveOccurred())
|
|
var root *folderEntry
|
|
for folder := range results {
|
|
if folder.path == "." {
|
|
root = folder
|
|
}
|
|
}
|
|
Expect(root).ToNot(BeNil())
|
|
return root
|
|
}
|
|
|
|
It("imports symlinks to out-of-library audio files", func() {
|
|
root := walkRoot()
|
|
Expect(root.audioFiles).To(HaveKey("relative.mp3"))
|
|
Expect(root.audioFiles).To(HaveKey("absolute.mp3"))
|
|
})
|
|
|
|
It("rejects a chain that ends in a non-audio file, even through an audio-named intermediate", func() {
|
|
root := walkRoot()
|
|
Expect(root.audioFiles).ToNot(HaveKey("evil.wav"))
|
|
})
|
|
|
|
It("skips broken symlinks", func() {
|
|
root := walkRoot()
|
|
Expect(root.audioFiles).ToNot(HaveKey("broken.mp3"))
|
|
})
|
|
|
|
It("skips all file symlinks when FollowSymlinks is disabled", func() {
|
|
conf.Server.Scanner.FollowSymlinks = false
|
|
root := walkRoot()
|
|
Expect(root.audioFiles).To(BeEmpty())
|
|
})
|
|
})
|
|
|
|
Context("out-of-tree escape (temp dir)", func() {
|
|
var root string
|
|
BeforeEach(func() {
|
|
conf.Server.Scanner.FollowSymlinks = true
|
|
root = GinkgoT().TempDir()
|
|
outside := GinkgoT().TempDir()
|
|
Expect(os.WriteFile(filepath.Join(outside, "passwd"), []byte("TOPSECRET"), 0600)).To(Succeed())
|
|
Expect(os.WriteFile(filepath.Join(outside, "real.flac"), []byte("AUDIO"), 0600)).To(Succeed())
|
|
Expect(os.WriteFile(filepath.Join(root, "song.mp3"), []byte("AUDIO"), 0600)).To(Succeed())
|
|
// evil.mp3 escapes to a non-audio target; legit.flac is a valid out-of-tree audio symlink.
|
|
Expect(os.Symlink(filepath.Join(outside, "passwd"), filepath.Join(root, "evil.mp3"))).To(Succeed())
|
|
Expect(os.Symlink(filepath.Join(outside, "real.flac"), filepath.Join(root, "legit.flac"))).To(Succeed())
|
|
})
|
|
|
|
It("rejects the absolute-path escape but keeps legit out-of-tree audio", func() {
|
|
fsys := os.DirFS(root)
|
|
|
|
name, ok := classify(fsys, ".", "song.mp3")
|
|
Expect(ok).To(BeTrue())
|
|
Expect(model.IsAudioFile(name)).To(BeTrue())
|
|
|
|
name, ok = classify(fsys, ".", "legit.flac")
|
|
Expect(ok).To(BeTrue())
|
|
Expect(model.IsAudioFile(name)).To(BeTrue())
|
|
|
|
name, ok = classify(fsys, ".", "evil.mp3")
|
|
Expect(ok).To(BeTrue())
|
|
Expect(model.IsAudioFile(name)).To(BeFalse())
|
|
})
|
|
|
|
It("skips all file symlinks when FollowSymlinks is disabled", func() {
|
|
conf.Server.Scanner.FollowSymlinks = false
|
|
fsys := os.DirFS(root)
|
|
entries, err := fs.ReadDir(fsys, ".")
|
|
Expect(err).ToNot(HaveOccurred())
|
|
for _, e := range entries {
|
|
_, ok := resolveEntryName(GinkgoT().Context(), fsys, ".", e)
|
|
if e.Type()&fs.ModeSymlink != 0 {
|
|
Expect(ok).To(BeFalse(), e.Name())
|
|
} else {
|
|
Expect(ok).To(BeTrue(), e.Name())
|
|
}
|
|
}
|
|
})
|
|
})
|
|
})
|
|
|
|
Describe("isDirIgnored", func() {
|
|
DescribeTable("returns expected result",
|
|
func(dirName string, expected bool) {
|
|
Expect(isDirIgnored(dirName)).To(Equal(expected))
|
|
},
|
|
Entry("normal dir", "empty_folder", false),
|
|
Entry("dot-prefixed album dir", ".Hack Sign Original Soundtrack", false),
|
|
Entry("git dir", ".git", true),
|
|
Entry("streams dir", ".streams", true),
|
|
Entry("dir starting with ellipsis", "...unhidden_folder", false),
|
|
Entry("recycle bin", "$Recycle.Bin", true),
|
|
Entry("snapshot dir", "#snapshot", true),
|
|
Entry("synology metadata dir", "@eaDir", true),
|
|
)
|
|
})
|
|
|
|
Describe("isIgnoredEntry", func() {
|
|
BeforeEach(func() {
|
|
DeferCleanup(configtest.SetupConfig())
|
|
})
|
|
|
|
DescribeTable("with IgnoreDotFolders enabled (default)",
|
|
func(name string, isDir, expected bool) {
|
|
conf.Server.Scanner.IgnoreDotFolders = true
|
|
Expect(isIgnoredEntry(name, isDir)).To(Equal(expected))
|
|
},
|
|
Entry("normal dir", "Album", true, false),
|
|
Entry("normal file", "track.mp3", false, false),
|
|
Entry("dot folder", ".Hack Sign Original Soundtrack", true, true),
|
|
Entry("dot file", ".hidden.mp3", false, true),
|
|
Entry("blocklisted dir", ".git", true, true),
|
|
Entry("ellipsis dir", "...unhidden", true, false),
|
|
)
|
|
|
|
DescribeTable("with IgnoreDotFolders disabled",
|
|
func(name string, isDir, expected bool) {
|
|
conf.Server.Scanner.IgnoreDotFolders = false
|
|
Expect(isIgnoredEntry(name, isDir)).To(Equal(expected))
|
|
},
|
|
Entry("normal dir", "Album", true, false),
|
|
Entry("normal file", "track.mp3", false, false),
|
|
Entry("dot folder is allowed", ".Hack Sign Original Soundtrack", true, false),
|
|
Entry("dot file is still ignored", ".hidden.mp3", false, true),
|
|
Entry("blocklisted dir still ignored", ".git", true, true),
|
|
)
|
|
})
|
|
|
|
Describe("isDotEntry", func() {
|
|
DescribeTable("returns expected result",
|
|
func(name string, expected bool) {
|
|
Expect(isDotEntry(name)).To(Equal(expected))
|
|
},
|
|
Entry("dot folder", ".Hidden", true),
|
|
Entry("dot file", ".hidden.mp3", true),
|
|
Entry("current dir", ".", false),
|
|
Entry("parent dir", "..", false),
|
|
Entry("two leading dots", "..foo", false),
|
|
Entry("ellipsis", "...unhidden", false),
|
|
Entry("normal name", "Album", false),
|
|
)
|
|
})
|
|
|
|
Describe("fullReadDir", func() {
|
|
var (
|
|
fsys fakeFS
|
|
ctx context.Context
|
|
)
|
|
|
|
BeforeEach(func() {
|
|
ctx = GinkgoT().Context()
|
|
fsys = fakeFS{MapFS: fstest.MapFS{
|
|
"root/a/f1": {},
|
|
"root/b/f2": {},
|
|
"root/c/f3": {},
|
|
}}
|
|
})
|
|
|
|
DescribeTable("reading directory entries",
|
|
func(failOn string, expectedErr error, expectedNames []string) {
|
|
fsys.failOn = failOn
|
|
fsys.err = expectedErr
|
|
dir, _ := fsys.Open("root")
|
|
entries := fullReadDir(ctx, dir.(fs.ReadDirFile))
|
|
Expect(entries).To(HaveLen(len(expectedNames)))
|
|
for i, name := range expectedNames {
|
|
Expect(entries[i].Name()).To(Equal(name))
|
|
}
|
|
},
|
|
Entry("reads all entries", "", nil, []string{"a", "b", "c"}),
|
|
Entry("skips entries with permission error", "b", nil, []string{"a", "c"}),
|
|
Entry("aborts on fs.ErrNotExist", "", fs.ErrNotExist, []string{}),
|
|
)
|
|
})
|
|
})
|
|
})
|
|
|
|
type fakeFS struct {
|
|
fstest.MapFS
|
|
failOn string
|
|
err error
|
|
}
|
|
|
|
func (f *fakeFS) Open(name string) (fs.File, error) {
|
|
dir, err := f.MapFS.Open(name)
|
|
return &fakeDirFile{File: dir, fail: f.failOn, err: f.err}, err
|
|
}
|
|
|
|
type fakeDirFile struct {
|
|
fs.File
|
|
entries []fs.DirEntry
|
|
pos int
|
|
fail string
|
|
err error
|
|
}
|
|
|
|
// Only works with n == -1
|
|
func (fd *fakeDirFile) ReadDir(int) ([]fs.DirEntry, error) {
|
|
if fd.err != nil {
|
|
return nil, fd.err
|
|
}
|
|
if fd.entries == nil {
|
|
fd.entries, _ = fd.File.(fs.ReadDirFile).ReadDir(-1)
|
|
}
|
|
var dirs []fs.DirEntry
|
|
for {
|
|
if fd.pos >= len(fd.entries) {
|
|
break
|
|
}
|
|
e := fd.entries[fd.pos]
|
|
fd.pos++
|
|
if e.Name() == fd.fail {
|
|
return dirs, &fs.PathError{Op: "lstat", Path: e.Name(), Err: fs.ErrPermission}
|
|
}
|
|
dirs = append(dirs, e)
|
|
}
|
|
return dirs, nil
|
|
}
|
|
|
|
func getDirEntry(baseDir, name string) os.DirEntry {
|
|
dirEntries, _ := os.ReadDir(baseDir)
|
|
for _, entry := range dirEntries {
|
|
if entry.Name() == name {
|
|
return entry
|
|
}
|
|
}
|
|
panic(fmt.Sprintf("Could not find %s in %s", name, baseDir))
|
|
}
|
|
|
|
// mockMusicFS is a mock implementation of the MusicFS interface that supports symlinks
|
|
type mockMusicFS struct {
|
|
storage.MusicFS
|
|
fs.FS
|
|
}
|
|
|
|
// Open resolves symlinks
|
|
func (m *mockMusicFS) Open(name string) (fs.File, error) {
|
|
f, err := m.FS.Open(name)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
|
|
info, err := f.Stat()
|
|
if err != nil {
|
|
f.Close()
|
|
return nil, err
|
|
}
|
|
|
|
if info.Mode()&fs.ModeSymlink != 0 {
|
|
// For symlinks, read the target path from the Data field
|
|
target := string(m.FS.(fstest.MapFS)[name].Data)
|
|
f.Close()
|
|
return m.FS.Open(target)
|
|
}
|
|
|
|
return f, nil
|
|
}
|
|
|
|
// Stat uses Open to resolve symlinks
|
|
func (m *mockMusicFS) Stat(name string) (fs.FileInfo, error) {
|
|
f, err := m.Open(name)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
defer f.Close()
|
|
return f.Stat()
|
|
}
|
|
|
|
// ReadDir uses Open to resolve symlinks
|
|
func (m *mockMusicFS) ReadDir(name string) ([]fs.DirEntry, error) {
|
|
f, err := m.Open(name)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
defer f.Close()
|
|
if dirFile, ok := f.(fs.ReadDirFile); ok {
|
|
return dirFile.ReadDir(-1)
|
|
}
|
|
return nil, fmt.Errorf("not a directory")
|
|
}
|
|
|
|
// ReadLink returns the target of the named symbolic link (implements fs.ReadLinkFS).
|
|
func (m *mockMusicFS) ReadLink(name string) (string, error) {
|
|
mapFS := m.FS.(fstest.MapFS)
|
|
entry, ok := mapFS[name]
|
|
if !ok {
|
|
return "", &fs.PathError{Op: "readlink", Path: name, Err: fs.ErrNotExist}
|
|
}
|
|
if entry.Mode&fs.ModeSymlink == 0 {
|
|
return "", &fs.PathError{Op: "readlink", Path: name, Err: fmt.Errorf("not a symlink")}
|
|
}
|
|
return string(entry.Data), nil
|
|
}
|
|
|
|
// Lstat returns FileInfo for the named file without following symlinks (implements fs.ReadLinkFS).
|
|
func (m *mockMusicFS) Lstat(name string) (fs.FileInfo, error) {
|
|
mapFS := m.FS.(fstest.MapFS)
|
|
if _, ok := mapFS[name]; !ok {
|
|
return nil, &fs.PathError{Op: "lstat", Path: name, Err: fs.ErrNotExist}
|
|
}
|
|
f, err := m.FS.Open(name)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
defer f.Close()
|
|
return f.Stat()
|
|
}
|