This document provides a comprehensive technical explanation of Navidrome's music library scanner system.
## Architecture Overview
The Navidrome scanner is built on a multi-phase pipeline architecture designed for efficient processing of music files. It systematically traverses file system directories, processes metadata, and maintains a database representation of the music library. A key performance feature is that some phases run sequentially while others execute in parallel.
The execution flow shows that Phases 1 and 2 run sequentially, while Phases 3 and 4 execute in parallel to maximize performance before the final processing steps.
## Core Components
### Scanner Controller (`controller.go`)
This is the entry point for all scanning operations. It provides:
- Public API for initiating scans and checking scan status
- Event broadcasting to notify clients about scan progress
- Serialization of scan operations (prevents concurrent scans)
- Progress tracking and monitoring
- Error collection and reporting
```go
type Scanner interface {
// ScanAll starts a full scan of the music library. This is a blocking operation.
The primary implementation that orchestrates the four-phase scanning pipeline. Each phase follows the Phase interface pattern:
```go
type phase[T any] interface {
producer() ppl.Producer[T]
stages() []ppl.Stage[T]
finalize(error) error
description() string
}
```
This design enables:
- Type-safe pipeline construction with generics
- Modular phase implementation
- Separation of concerns
- Easy measurement of performance
### External Scanner (`external.go`)
The External Scanner is a specialized implementation that offloads the scanning process to a separate subprocess. This is specifically designed to address memory management challenges in long-running Navidrome instances.
```go
// scannerExternal is a scanner that runs an external process to do the scanning. It is used to avoid
// memory leaks or retention in the main process, as the scanner can consume a lot of memory. The
// external process will be spawned with the same executable as the current process, and will run
// the "scan" command with the "--subprocess" flag.
//
// The external process will send progress updates to the main process through its STDOUT, and the main
// process will forward them to the caller.
```
```mermaid
sequenceDiagram
participant MP as Main Process
participant ES as External Scanner
participant SP as Subprocess (navidrome scan --subprocess)
participant FS as File System
participant DB as Database
Note over MP: DevExternalScanner=true
MP->>ES: ScanAll(ctx, fullScan)
activate ES
ES->>ES: Locate executable path
ES->>SP: Start subprocess with args:<br>scan --subprocess --configfile ... etc.
activate SP
Note over ES,SP: Create pipe for communication
par Subprocess executes scan
SP->>FS: Read files & metadata
SP->>DB: Update database
and Main process monitors progress
loop For each progress update
SP->>ES: Send encoded progress info via stdout pipe
ES->>MP: Forward progress info
end
end
SP-->>ES: Subprocess completes (success/error)
deactivate SP
ES-->>MP: Return aggregated warnings/errors
deactivate ES
```
Technical details:
1.**Process Isolation**
- Spawns a separate process using the same executable
- Uses the `--subprocess` flag to indicate it's running as a child process
- Preserves configuration by passing required flags (`--configfile`, `--datafolder`, etc.)
2.**Inter-Process Communication**
- Uses a pipe for bidirectional communication
- Encodes/decodes progress updates using Go's `gob` encoding for efficient binary transfer
- Properly handles process termination and error propagation
3.**Memory Management Benefits**
- Scanning operations can be memory-intensive, especially with large music libraries
- Memory leaks or excessive allocations are automatically cleaned up when the process terminates
- Main Navidrome process remains stable even if scanner encounters memory-related issues
4.**Error Handling**
- Detects non-zero exit codes from the subprocess
- Propagates error messages back to the main process
- Ensures resources are properly cleaned up, even in error conditions
## Scanning Process Flow
### Phase 1: Folder Scan (`phase_1_folders.go`)
This phase handles the initial traversal and media file processing.
```mermaid
flowchart TD
A[Start Phase 1] --> B{Full Scan?}
B -- Yes --> C[Scan All Folders]
B -- No --> D[Scan Modified Folders]
C --> E[Read File Metadata]
D --> E
E --> F[Create Artists]
E --> G[Create Albums]
F --> H[Save to Database]
G --> H
H --> I[Mark Missing Folders]
I --> J[End Phase 1]
```
**Technical implementation details:**
1.**Folder Traversal**
- Uses `walkDirTree` to traverse the directory structure
- Handles symbolic links and hidden files
- Processes `.ndignore` files for exclusions
- Maps files to appropriate types (audio, image, playlist)
2.**Metadata Extraction**
- Processes files in batches (defined by `filesBatchSize = 200`)
- Extracts metadata using the configured storage backend
- Converts raw metadata to `MediaFile` objects
- Collects and normalizes tag information
3.**Album and Artist Creation**
- Groups tracks by album ID
- Creates album records from track metadata
- Handles album ID changes by tracking previous IDs
| `PID.Track` | Format for track persistent IDs (critical for tracking moved files) | "musicbrainz_trackid\|albumid,discnumber,tracknumber,title" |
| `PID.Album` | Format for album persistent IDs (affects album grouping) | "musicbrainz_albumid\|albumartistid,album,albumversion,releasedate" |
These options can be set in the Navidrome configuration file (e.g., `navidrome.toml`) or via environment variables with the `ND_` prefix (e.g., `ND_SCANNER_ENABLED=false`). For environment variables, dots in option names are replaced with underscores.
## Conclusion
The Navidrome scanner represents a sophisticated system for efficiently managing music libraries. Its phase-based pipeline architecture, careful handling of edge cases, and performance optimizations allow it to handle libraries of significant size while maintaining data integrity and providing a responsive user experience.
