hans/audio-library-manager
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

documentation and small bug fixes

Browse Source
This commit is contained in:
Hans Dijkema
2026-06-28 18:55:40 +02:00
parent b1b593896a
commit e538e8b180
7 changed files with 362 additions and 223 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+3
View File
@@ -17,3 +17,6 @@ compiled/
/private/*.bak
/*.bak
scribblings/*.html
scribblings/*.css
scribblings/*.js
README.md
+187 -105
View File
@@ -1,134 +1,216 @@
# audio-library-manager
Tools for maintaining audio library trees.
`audio-library-manager` maintains a music library in two forms:
The package currently contains two managers:
1. a source FLAC tree, optionally normalised so FLAC files do not exceed a
configured sample rate; and
2. a mirrored Opus tree, where FLAC files are converted to `.opus` and
non-FLAC sidecar files are copied unchanged.
- `flac-48khz-manager.rkt`: keeps a FLAC tree at a maximum sample rate of 48 kHz.
- `flac2opus-manager.rkt`: mirrors a FLAC source tree to an Opus target tree.
The current entry point is `audio-manager.rkt`, exported as the function
`audio-manager` from `main.rkt`.
Both managers use the same administration files in the source tree:
The implementation is intentionally simple. It is a command-line batch process:
scan files, compare with the saved state, run a small ordered set of processing
steps, write the state, send a report. It does not use places, channels,
threads or a worker protocol.
- `.music-info.db`: keystore state database
- `.flac-48khz-manager.ini`: configuration
- `.flac-48khz-manager.log`: log file
## Requirements
The `flac2opus` state uses its own key prefix in `.music-info.db`, so it can share
the same database with the 48 kHz manager without mixing state entries. State
keys use only normalized paths relative to the configured source/base directory:
absolute mount points are not stored, and path separators are always `/`.
Racket packages:
## FLAC 48 kHz manager
- `racket-audio`
- `simple-ini`
- `simple-log`
- `net-lib`
The command keeps a FLAC directory tree at a maximum sample rate of 48 kHz.
Files above the configured threshold are converted in place by a direct call to
`racket-audio/audio-encoder`. The conversion code is deliberately synchronous:
no places, channels or worker protocol are used.
Native libraries are provided or loaded by `racket-audio`. For the current audio
path this normally means FLAC, Opus/Opusenc, TagLib and SoXR. FFmpeg is not part
of the intended FLAC-to-Opus path; it is only relevant for `racket-audio` features
that need FFmpeg-based decoding.
Run:
## Usage
```sh
racket flac-48khz-manager.rkt /path/to/flac-tree
```
When a FLAC file has no embedded picture and the same directory contains one of
`cover.jpg`, `folder.jpg`, `cover.png` or `folder.png`, the manager embeds that
image as front-cover picture before fingerprinting/conversion. This keeps FLAC
artwork consistent before a later Opus mirror is made.
The `flac-48khz-manager` scanner only processes `.flac` files. Other files such
as `.mp3`, booklets and cover images are ignored by this manager.
## FLAC to Opus mirror manager
The command mirrors a source directory to a target directory. FLAC files are
converted to Ogg Opus files with extension `.opus`; all other regular files are
copied unchanged, preserving their relative path and modification time. Examples
include `booklet.pdf`, `cover.jpg`, `folder.png`, cue sheets, text files and
other sidecar files.
Run:
```sh
racket flac2opus-manager.rkt /path/to/flac-tree /path/to/opus-tree
racket flac2opus-manager.rkt --kbps 192 /path/to/flac-tree /path/to/opus-tree
```
The default Opus bitrate is 224 kbps. The library API exposes the same setting
as keyword argument:
From a Racket REPL or a small wrapper script:
```racket
(manage-flac2opus-tree source-directory target-directory #:kbps 224)
#lang racket/base
(require audio-library-manager)
(audio-manager "/mnt/music" "/mnt/music-opus")
```
Metadata is copied through `racket-audio/taglib` and `racket-audio/audio-encoder`:
ordinary TagLib properties are transferred, embedded pictures are transferred,
and an additional `FLAC2OPUS` comment is written to mark the conversion. The
batch converter itself remains a simple synchronous function call.
The first argument is the source music tree. The second argument is the target
Opus mirror tree.
The manager removes target files that belonged to source files which disappeared
since the previous run. It deliberately does not mirror its own root-level
administration files: `.music-info.db`, `.flac-48khz-manager.ini` and
`.flac-48khz-manager.log`.
The optional log level defaults to `debug`:
Paths are handled as Racket paths rather than by splitting on `/`, so Windows
absolute paths and UNC paths such as `\\panderleou\music` are preserved by the
platform path implementation.
```racket
(audio-manager "/mnt/music" "/mnt/music-opus" #:log-level 'info)
```
## Files created in the source tree
The manager stores its administration files in the source directory:
- `.audio-manager.ini` — configuration, read with `simple-ini`.
- `.audio-manager.db` — serialized state hash, keyed by normalized relative
paths.
- `.audio-manager.log` — log file.
State keys are normalized relative paths from the source tree. Absolute mount
points are not stored, and path separators are normalized to `/`. This makes the
state less dependent on the local mount point used on a particular machine.
## Configuration
Important configuration keys are created automatically in `.flac-48khz-manager.ini`.
The same file is used by both managers. The implementation now reads this file as a simple-ini value directly; there is no separate configuration struct layered on top of the ini model.
The manager expects a configuration file at:
```ini
[manager]
max-sample-rate=48000
hash-algorithm="sha256"
change-detection="flac-taglib"
dry-run=#f
display-log=#t
log-file=".flac-48khz-manager.log"
compression-level=5
[opus-manager]
log-file=".flac2opus-manager.log"
[mail]
enabled=#f
send-on-success=#f
send-on-error=#t
host=""
port=25
tls=#f
username=""
password=""
from=""
to=""
cc=""
bcc=""
subject-prefix="[flac-48khz-manager]"
```text
<source-tree>/.audio-manager.ini
```
SMTP reports are HTML and summarize counters and errors.
Example:
## Change detection
```ini
[flac]
max-khz=48000
The default change detection is `flac-taglib`:
[opus]
kbps=224
1. unchanged files are skipped with a cheap `size` + `mtime` comparison;
2. new or visibly changed files get a semantic fingerprint;
3. for FLAC files, the semantic fingerprint consists of FLAC STREAMINFO data,
including the FLAC audio MD5 signature, plus TagLib properties and embedded
picture metadata/content hash;
4. for non-FLAC files in the Opus mirror, the default is `mtime` + `size`;
5. full-file hashing is available by setting `change-detection="hash"`.
[mail]
from=Audio Manager <audio@example.org>
to=Hans <hans@example.org>
server=smtp.example.org
port=25
user=
passwd=
```
This avoids reading every complete audio file during a normal run, which matters
on Windows and network shares.
The mail section is currently required. If the mail configuration is missing or
incomplete, the manager fails at startup with a configuration error. This is
intentional: a batch run should not silently skip reporting.
## FLAC files with ID3v2 prefixes
## Processing model
Version 0.1.4 accepts native FLAC files that start directly with `fLaC`, and FLAC
files with an ID3v2 tag before the `fLaC` marker. The latter occurs in some
libraries and is accepted by players such as foobar2000/libFLAC. The fast
STREAMINFO reader skips the ID3v2 prefix before reading the FLAC metadata.
The manager uses a file walker with an ordered processing pipeline. Each step
receives:
```racket
base-path path info
```
and returns the same three values, with `info` possibly enriched. The `info` hash
contains the relative path, type, extension, state and any metadata discovered by
earlier steps.
The important file states are:
- `new` — the relative path was not present in the saved state.
- `changed` — the path existed before, but type, size or modification time
changed.
- `unchanged` — the path exists and its basic filesystem state is unchanged.
- `deleted` — the path existed in the saved state but no longer exists in the
source tree.
For unchanged entries, the previous enriched `info` hash is reused. For changed
entries, the previous `info` hash is reused but marked as `changed` and updated
with the current type, size and modification time. Stages must therefore treat
`changed` as the signal to re-inspect content-dependent data.
The pipeline is grouped conceptually as:
1. enrichment: normalize/identify known extensions, sniff audio files, detect
FLAC-with-ID3, and read FLAC sample rate/bit depth when needed;
2. conversion/actions: downsample FLAC files above the configured rate, embed a
sidecar cover image when a FLAC has no embedded picture, convert FLAC to
Opus, and copy non-FLAC sidecar files;
3. cleanup/logging: remove target files for deleted source entries and log
progress.
## FLAC handling
FLAC files are detected by extension and confirmed by `racket-audio` sniffing.
A file with extension `.flac` but an ID3 prefix is reported as a FLAC-with-ID3
case. Such files are not treated as ordinary sidecar files.
For valid FLAC files:
- sample rate and bit depth are read through the FLAC decoder;
- files with a sample rate greater than `[flac] max-khz` are converted in place
through `racket-audio/audio-encoder`;
- conversion is synchronous and direct;
- if the file has no embedded picture, `cover.jpg`, `cover.jpeg`, `folder.jpg`,
`folder.jpeg`, `cover.png` or `folder.png` in the same directory is embedded
as a front-cover picture.
Temporary FLAC files produced by interrupted conversions should be ignored by
the scanner. The implementation uses manager-specific temporary names for this
purpose.
## Opus mirror handling
For every processable FLAC source file, the manager writes a corresponding Opus
file under the target tree:
```text
source: album/track.flac
target: album/track.opus
```
The default bitrate is 224 kbps and can be changed with `[opus] kbps`.
Non-FLAC files are copied unchanged to the target tree when they are new,
changed, or missing from the target. Hidden files are ignored. Typical copied
sidecar files include cover images, PDFs, cue sheets and logs.
When a source entry is deleted, the corresponding target entry is removed. For a
deleted FLAC source, the target extension is changed from `.flac` to `.opus`.
Renames are deliberately represented as `deleted` plus `new`; no rename detection
is attempted.
## Logging and report
The manager logs to:
```text
<source-tree>/.audio-manager.log
```
The report contains sections for:
- FLAC files with ID3 tags;
- FLAC conversions and cover updates;
- Opus conversions;
- file counters for processed, new, changed, unchanged and deleted files;
- failed conversions and failed copies.
At the end of a successful scan the serialized state is written to
`.audio-manager.db` and the report is sent by mail.
## API
```racket
(audio-manager music-path opus-path #:log-level [log-level 'debug])
```
Runs the complete batch process. `music-path` is the source tree and `opus-path`
is the target Opus mirror tree. The function returns `'done` after a completed
run.
## Design notes
The package intentionally keeps responsibilities separated:
- `audio-manager.rkt` orchestrates the batch run and owns the pipeline;
- `private/file-walker.rkt` scans the filesystem and maintains the state model;
- `private/flac-handling.rkt` contains FLAC inspection and FLAC downsampling;
- `private/opus-handling.rkt` contains FLAC-to-Opus conversion;
- `private/mail.rkt` sends reports;
- `private/log.rkt` defines the logger;
- `private/util.rkt` contains small path/date/cover helpers.
The manager does not implement an audio pipeline itself. Audio decoding,
encoding, metadata and resampling are delegated to `racket-audio`.
audio-manager.rkt
+1 -1
View File
@@ -256,7 +256,7 @@
;; Kopieer bestanden rechtstreeks indien nodig en geen flac
(define (copy-other base-path path info)
(when (and (needs-copying? path info)
(not (flac? path info)))
(not (presumed-flac? info)))
(with-handlers ([exn? (λ (e)
(set! failed-copies (+ failed-copies 1))
(err-am "Copy file ~a: ~a" path e))])
info.rkt
+1 -1
View File
@@ -10,5 +10,5 @@
(define scribblings '(("scribblings/audio-library-manager.scrbl" ())))
(define build-deps '("rackunit-lib" "scribble-lib" "racket-doc"))
(define pkg-desc "Audio library maintenance tools for FLAC and Opus trees")
(define version "0.2.6")
(define version "0.2.9")
(define pkg-authors '(hans-dijkema))
private/file-walker.rkt
+3 -1
View File
@@ -91,9 +91,11 @@
(if (file-info-equal? info in-db)
(hash-set! in-db 'file-db 'unchanged)
(begin
(hash-set! in-db 'type (hash-ref info 'type))
(hash-set! in-db 'file-db 'changed)
(when (eq? (hash-ref info 'type #f) 'file)
(hash-set! in-db 'size (hash-ref info 'size))
(hash-set! in-db 'mtime (hash-ref info 'mtime))))
(hash-set! in-db 'mtime (hash-ref info 'mtime)))))
(hash-set! db-hash normalized-sub-path in-db)
(values base-path path in-db)
)
private/flac-handling.rkt
+6 -1
View File
@@ -33,7 +33,12 @@
(encoder-settings (hash 'target-sample-rate rate
'compression-level 8))
)
(with-handlers ([exn? (λ (e)
(with-handlers ([exn:break? (λ (e)
(err-am "Ctrl-c used")
(when (file-exists? tmp-file)
(delete-file tmp-file))
(raise e))]
[exn? (λ (e)
(err-am (format "~a" e))
(when (file-exists? tmp-file)
(delete-file tmp-file))
scribblings/audio-library-manager.scrbl
+160 -113
View File
@@ -1,149 +1,196 @@
#lang scribble/manual
@(require (for-label racket/base
(file "../main.rkt")))
"../main.rkt"))
@title{audio-library-manager}
@author{Hans Dijkema}
The @racketmodname[audio-library-manager] package contains command-line tools for
maintaining audio library trees.
@defmodule[audio-library-manager]
The @racketmodname[audio-library-manager] package contains a simple batch manager
for maintaining a FLAC source library and an Opus mirror tree.
The current public entry point is @racket[audio-manager]. The manager scans a
source tree, compares the scan with a saved state hash, enriches file information
through a small ordered pipeline, performs the necessary FLAC and Opus actions,
writes the updated state, logs progress, and sends a mail report.
The design is deliberately synchronous. It does not use places, channels,
threads or a worker protocol. Conversion is performed by ordinary function calls
to @racketmodname[racket-audio].
@section{Administration files}
The managers use these files below the source directory:
The manager stores these files in the source directory:
@itemlist[
@item{@filepath{.music-info.db}: keystore database with file state.}
@item{@filepath{.flac-48khz-manager.ini}: configuration.}
@item{@filepath{.flac-48khz-manager.log}: log file.}]
@item{@filepath{.audio-manager.ini}: configuration read through
@racketmodname[simple-ini].}
@item{@filepath{.audio-manager.db}: serialized state hash, keyed by normalized
relative paths.}
@item{@filepath{.audio-manager.log}: log file written through
@racketmodname[simple-log].}]
The FLAC-to-Opus manager uses a separate key prefix in @filepath{.music-info.db},
so it can share the same database with the 48 kHz manager.
State keys are relative to the configured source tree. Absolute mount points are
not stored, and path separators are normalized to @tt{/}. This makes the state
usable across machines where the same library may be mounted at different base
paths.
@section{FLAC 48 kHz manager}
@section{Usage}
The tool @filepath{flac-48khz-manager.rkt} keeps a FLAC tree at a maximum sample
rate of 48 kHz. Files with a higher sample rate are converted in place by a
direct call to @racketmodname[racket-audio/audio-encoder]. The batch conversion
path is deliberately synchronous; it does not use places, channels or a worker
protocol.
From a Racket program or REPL:
Run the manager as:
@racketblock[
(require audio-library-manager)
@verbatim{racket flac-48khz-manager.rkt <base-directory>}
(audio-manager "/mnt/music" "/mnt/music-opus")
]
When a FLAC file has no embedded picture and its directory contains
@filepath{cover.jpg}, @filepath{folder.jpg}, @filepath{cover.png} or
@filepath{folder.png}, the manager embeds that image as front-cover picture
before fingerprinting and conversion.
The optional log level defaults to @racket['debug]:
The manager only processes @filepath{.flac} files; @filepath{.mp3} files and
other sidecar files are ignored by this command.
@section{FLAC to Opus mirror manager}
The tool @filepath{flac2opus-manager.rkt} mirrors a source directory to a target
directory. FLAC files are converted to Ogg Opus files with extension
@filepath{.opus}. Other regular files are copied unchanged, preserving their
relative path and modification time. This includes sidecar files such as
@filepath{booklet.pdf}, @filepath{cover.jpg}, cue sheets and text files.
Run the manager as:
@verbatim{racket flac2opus-manager.rkt <source-directory> <target-directory>}
The default Opus bitrate is 224 kbps. A different bitrate can be selected with:
@verbatim{racket flac2opus-manager.rkt --kbps 192 <source-directory> <target-directory>}
Metadata is copied through @racketmodname[racket-audio/taglib] and
@racketmodname[racket-audio/audio-encoder]. TagLib properties and embedded
pictures are transferred to the Opus file. The manager also writes a
@tt{FLAC2OPUS} comment indicating that the file was converted by the manager.
The conversion function is a normal synchronous function call.
When a source file disappears, the corresponding target file is removed on the
next run. The manager does not mirror its own root-level administration files.
Paths are handled as Racket paths instead of by splitting on @litchar{/}, so
Windows absolute paths and UNC paths are left to the platform path
implementation.
@racketblock[
(audio-manager "/mnt/music" "/mnt/music-opus" #:log-level 'info)
]
@section{Configuration}
The configuration file is created automatically when it does not exist. The managers read it directly as sections and keys through @racketmodname[simple-ini]; there is no separate configuration struct layered on top of the ini model. The main settings are:
The manager expects @filepath{.audio-manager.ini} in the source tree. A minimal
configuration looks like this:
@verbatim{
[manager]
max-sample-rate=48000
hash-algorithm="sha256"
change-detection="flac-taglib"
dry-run=#f
display-log=#t
log-file=".flac-48khz-manager.log"
compression-level=5
@verbatim|{
[flac]
max-khz=48000
[opus-manager]
log-file=".flac2opus-manager.log"
[opus]
kbps=224
[mail]
enabled=#f
send-on-success=#f
send-on-error=#t
host=""
from=Audio Manager <audio@example.org>
to=Hans <hans@example.org>
server=smtp.example.org
port=25
tls=#f
username=""
password=""
from=""
to=""
cc=""
bcc=""
subject-prefix="[flac-48khz-manager]"
}
user=
passwd=
}|
When mail is enabled, the report is sent as HTML. The message contains the
summary counters and the error table; it does not dump the full log by default.
The mail configuration is required in the current implementation. If it is
missing or incomplete, the manager fails at startup. This is intentional for a
batch process where reporting should not silently disappear.
@section{Change detection}
@section{Processing model}
The default @tt{change-detection} mode is @tt{flac-taglib}. Unchanged files are
first skipped by comparing @tt{size} and @tt{mtime} from the keystore state. New
or visibly changed files get a semantic fingerprint made from FLAC STREAMINFO
and TagLib metadata. The FLAC part includes the STREAMINFO audio MD5 signature;
the TagLib part includes properties and an embedded-picture content hash.
The file walker produces entries with an @racket[info] hash. The manager then
runs an ordered list of steps. Each step receives three values:
For non-FLAC files mirrored by @filepath{flac2opus-manager.rkt}, the default
signature is @tt{mtime} plus @tt{size}. A full-file hash remains available by
setting @tt{change-detection="hash"}.
@racketblock[
base-path
path
info
]
@section{Library API}
and returns the same three values, possibly with @racket[info] enriched. The
important states in @racket[info] are:
@defproc[(manage-flac-tree
[base-directory path-string?]
[#:inspect-flac-proc inspect-flac-proc procedure? inspect-flac-sample-rate]
[#:fingerprint-proc fingerprint-proc procedure? flac-taglib-fingerprint]
[#:convert-proc convert-proc procedure? convert-flac-to-target-in-place])
list?]{
Scans @racket[base-directory], updates the keystore state, converts FLAC files
above the configured maximum sample rate, sends the optional HTML mail report,
and returns a summary association list.
@itemlist[
@item{@racket['new]: the relative path is not present in the saved state.}
@item{@racket['changed]: the path exists, but type, size or modification time
changed.}
@item{@racket['unchanged]: the path exists and its basic filesystem state is
unchanged.}
@item{@racket['deleted]: the path existed in the saved state but no longer
exists in the source tree.}]
The keyword arguments are intended for tests and dry integration work. In normal
use, the default inspector, fingerprint function and converter are used.}
For @racket['unchanged] entries, the previous enriched @racket[info] hash is
reused. For @racket['changed] entries, the previous @racket[info] hash is reused
but marked as changed and updated with the current filesystem data. Content-aware
steps should therefore use @racket['changed] as the signal to re-inspect data
such as sample rate, bit depth and sniffed audio format.
@defproc[(manage-flac2opus-tree
[source-directory path-string?]
[target-directory path-string?]
[#:kbps kbps exact-positive-integer? 224]
[#:convert-proc convert-proc procedure? convert-flac-to-opus])
list?]{
Mirrors @racket[source-directory] to @racket[target-directory]. FLAC files are
converted to Opus at @racket[kbps] kbps; all other regular files are copied.
The state is stored in the source directory's @filepath{.music-info.db} using a
separate @tt{flac2opus} prefix. Keystore keys and stored target paths use
normalized paths relative to the configured source or target base directory;
absolute mount points are not stored and separators are always @tt{/}.}
Conceptually, the pipeline has three groups:
@defproc[(summary->lines [summary list?]) (listof string?)]{
Formats the summary association list as display lines.}
@itemlist[
@item{Enrichment: extension normalization, audio sniffing, FLAC-with-ID3
detection, and FLAC sample-rate/bit-depth inspection.}
@item{Actions: FLAC downsampling, embedding sidecar cover art, FLAC-to-Opus
conversion, and copying non-FLAC sidecar files.}
@item{Cleanup and logging: removal of target files for deleted source entries
and progress logging.}]
@section{FLAC handling}
FLAC source files are presumed by extension and then confirmed by audio sniffing.
A file with extension @filepath{.flac} but an ID3 prefix is reported as a
FLAC-with-ID3 case. Such a file is not copied to the Opus tree as an ordinary
sidecar file.
For valid FLAC files, the manager reads sample rate and bit depth through the
FLAC decoder. Files above the configured @tt{[flac] max-khz} value are converted
in place by @racketmodname[racket-audio/audio-encoder]. The conversion is a
normal synchronous function call.
If a FLAC file has no embedded picture, the manager looks in the same directory
for @filepath{cover.jpg}, @filepath{cover.jpeg}, @filepath{folder.jpg},
@filepath{folder.jpeg}, @filepath{cover.png}, or @filepath{folder.png}. When one
is found, it is embedded as front-cover artwork.
@section{Opus mirror handling}
For every processable FLAC source file, the manager writes an Opus target file
with the same relative path and extension @filepath{.opus}:
@verbatim|{
source: album/track.flac
target: album/track.opus
}|
The default bitrate is 224 kbps and can be changed with @tt{[opus] kbps}.
Non-FLAC files are copied unchanged to the target tree when they are new,
changed, or missing from the target. Hidden files are ignored. This is intended
for sidecar files such as cover images, PDFs, cue sheets and logs.
When a source entry disappears, the corresponding target entry is removed. For a
deleted FLAC source, the target extension is changed from @filepath{.flac} to
@filepath{.opus}. Renames are deliberately represented as a deleted old path and
a new path; no rename detection is attempted.
@section{Logging and report}
Logging goes to @filepath{.audio-manager.log} in the source tree and is also
shown on the console according to the selected log level. At the end of a
completed run the manager writes the serialized state to @filepath{.audio-manager.db}
and sends a report by mail.
The report contains sections for FLAC-with-ID3 cases, FLAC actions, Opus actions,
file counters, failed conversions and failed copies.
@section{API}
@defproc[(audio-manager
[music-path path-string?]
[opus-path path-string?]
[#:log-level log-level symbol? 'debug])
symbol?]{
Runs the complete batch process. @racket[music-path] is the source music tree.
@racket[opus-path] is the target Opus mirror tree. The function returns
@racket['done] after a completed run.}
@section{Module responsibilities}
The package keeps the responsibilities separated:
@itemlist[
@item{@filepath{audio-manager.rkt}: orchestration, counters, report text and
the ordered processing pipeline.}
@item{@filepath{private/file-walker.rkt}: filesystem scan and state transition
model.}
@item{@filepath{private/flac-handling.rkt}: FLAC metadata inspection and FLAC
downsampling.}
@item{@filepath{private/opus-handling.rkt}: FLAC-to-Opus conversion.}
@item{@filepath{private/mail.rkt}: SMTP report sending.}
@item{@filepath{private/log.rkt}: logger definition.}
@item{@filepath{private/util.rkt}: small path, date and cover-art helpers.}]
The manager does not implement audio codecs itself. Decoding, encoding,
metadata, cover handling and resampling are delegated to @racketmodname[racket-audio].