Fret Range & Cleanup

TECHNICAL_GUIDE.md

@@ -31,11 +31,11 @@ The AP setup pins NetworkManager to 2.4 GHz channel 6 with WPA2/RSN and CCMP cip
 
 On Raspberry Pi OS Lite, the installed display service owns `tty1`, conflicts with `getty@tty1.service`, and starts the Zero container through `product-vamp-start-zero`. The wrapper switches to the display VT when `chvt` is available and runs `vamp-zero` inside the container through `setsid --wait --ctty` with `/dev/tty1` as stdin, giving raylib's DRM backend a real virtual terminal while keeping Docker attached to the long-running display process. SSH remains the normal admin path and `tty2` is enabled as the fallback local console.
 
-At startup the Zero validates the packaged `catalogue.db` application identifier and schema version, opens it read-only through Noto, and loads menu metadata. It never creates or migrates a database, scans score JSON, or repairs metadata. Runtime navigation and rendering use that loaded Noto-backed state instead of rescanning asset folders. The UI supports a top-level mode menu, Performance song/part/score menus, and Practice type/lesson/exercise/key/score menus. Melody and chord practice both offer key selection by choosing a pre-generated score variant. The score view displays build-time engraved PNG pages. Musical interpretation, spacing, beaming, ties, accidentals, and pagination happen on the build host, not on the Pi. The view retains at most two page textures.
+At startup the Zero validates the packaged `catalogue.db` application identifier and schema version, opens it read-only through Noto, and loads menu metadata. It never creates or migrates a database, scans score JSON, or repairs metadata. Runtime navigation and rendering use that loaded Noto-backed state instead of rescanning asset folders. The UI supports a top-level mode menu, Performance song/part/score menus, and Practice exercise/key/score menus. Every practice exercise offers twelve pre-generated key variants; performance songs have no key-selection menu. The score view displays build-time engraved PNG pages. Musical interpretation, spacing, beaming, ties, accidentals, and pagination happen on the build host, not on the Pi. The view retains at most two page textures.
 
-Runtime catalogues contain immutable metadata shared by navigation, the HTTP service, and the UI. Song groups and practice lessons reference canonical records by index. SQLite retains complete bundles, while cached measure/page counts let menu queries avoid reading musical JSON. Python constructs the complete database before deployment; metadata and embedded JSON are published together. There are no custom rows, runtime imports, metadata-repair triggers, or persisted page-state tables. Selecting a score loads only that bundle asynchronously, discarding stale results after navigation; the view keeps at most two page textures.
+Runtime catalogues contain immutable metadata shared by navigation, the HTTP service, and the UI. Song groups reference canonical records by index; practice is a flat, numerically ordered exercise list. SQLite retains complete bundles, while cached measure/page counts let menu queries avoid reading musical JSON. Python constructs the complete database before deployment; metadata and embedded JSON are published together. There are no custom rows, runtime imports, metadata-repair triggers, or persisted page-state tables. Selecting a score loads only that bundle asynchronously, discarding stale results after navigation; the view keeps at most two page textures.
 
-Practice IDs retain their existing hyphen-joined format. Generation and catalogue compilation reject distinct paths that produce the same exercise or variant ID, reporting both paths before publishing the generated bundle. For example, `chord/one-two/three/c.json` and `chord/one/two-three/c.json` cannot coexist; rename one lesson or exercise to give it a distinct ID.
+Practice exercise IDs are the three-digit positive source stems, such as `001`. Variant IDs append the canonical key stem, such as `001-db`. Duplicate titles and identical placeholder music remain independent exercises. Catalogue schema 2 has no practice-type or lesson-group columns, and control responses expose one flat exercise list without nested type/lesson copies. Older catalogues must be rebuilt or reinstalled, not migrated.
 
 `--once` (also `runtime.exit_after_startup`) is a headless readiness check: it checks database integrity and validates the catalogue and referenced page assets, checks runtime-directory writes and the configured server bind, and exits without opening audio/display devices or publishing a readiness heartbeat. Page checks include containment, checksums, CPU-only PNG decoding, and exact 976x484 dimensions. The configured heartbeat destination must be absent or a regular file, never a directory, symlink, or special file. Runtime writes and readiness probes use exclusive sibling temporary files; readiness leaves an existing heartbeat unchanged and removes its probes. `--health-check` instead probes the running service's read-only `/api/health` endpoint using the configured timeout. A stopped/unresponsive service or an invalid response fails the check; it does not create or repair files. `--check-config` remains configuration-only.
 
@@ -45,9 +45,9 @@ Zero and Pico share one gesture state machine: a second release within 220 ms of
 
 ## Score Generation
 
-The build runs `python3 -m generator` against `data/` and writes build-only assets under the CMake binary tree. Performance source songs become acoustic, electric, and vocal arrangements. Melody and chord practice sources are emitted in all 12 keys at build time. Generated JSON keeps notation metadata while removing generator-only optional chord markers. The catalogue produces three bundles per song and twelve per practice exercise; source counts are discovered, not fixed in code.
+The build runs `python3 -m generator` against `data/` and writes build-only assets under the CMake binary tree. Performance source songs become acoustic, electric, and vocal arrangements. Numbered practice sources are emitted in all 12 keys at build time. Generated JSON keeps notation metadata while removing generator-only optional chord markers. The catalogue produces three bundles per song and twelve per practice exercise; source counts are discovered, not fixed in code.
 
-Practice sources live at `data/practice/<chord|melody>/<lesson_group>/<exercise>.xml`. For example, `chord/001_three_note_shell_voicings/001-cycles.xml` generates `practice/chord/001_three_note_shell_voicings/001-cycles/{c,db,d,eb,e,f,gb,g,ab,a,bb,b}.json` and corresponding page directories. Keep numeric prefixes zero-padded to the same width for menu ordering. Lesson folders determine lesson labels; MusicXML work/movement titles determine exercise labels, falling back to the filename stem when absent. Repeated titles do not merge exercises: identity includes the practice type, lesson folder, and normalized filename stem. Invalid nesting, unknown practice types, and colliding normalized filenames fail generation. Adding or renaming a source needs no registry edit; the next build discovers it, replaces obsolete generated paths, and builds a fresh database containing only the current sources.
+Practice sources live directly under `data/` with positive three-digit filenames, currently `001.xml` through `366.xml`. For example, `data/001.xml` generates `practice/001/{c,db,d,eb,e,f,gb,g,ab,a,bb,b}.json` and corresponding page directories. Performance sources stay in `data/songs/`. The filename number determines identity and ordering; MusicXML work/movement titles determine labels, falling back to the number when absent. Other root XML names and nested practice sources are rejected. Adding, renaming, or removing a numbered source requires no registry edit; the next build replaces obsolete generated assets and rebuilds the database. The current catalogue has 366 exercises, 4,392 practice variants, and six performance parts.
 
 The pipeline is MuseScore -> uncompressed `.xml` -> musical model -> automatic arranger/transposer -> semantic JSON plus engraved pages. Export MusicXML 3.1 or 4.0, `score-partwise`, with one part and one staff. Independent rhythmic voices remain separate; simultaneous chord tones retain their own spelling, written duration, and tie links. The packaged music describes each transformed part or practice variant, not an archival copy of the source XML.
 
@@ -83,7 +83,7 @@ In MuseScore's score properties, set `vamp:style` to a supported player style. I
 </miscellaneous>
 ```
 
-The default is Medium Swing; `Med.`, `Med. Swing`, and `Medium` remain aliases. A keyless melody or chord exercise needs `vamp:tonic` (for example, `C`) as its transposition reference. Modes and local key changes are preserved when generating practice variants. Keyless variants remain unkeyed in the music and show their selected transposition reference in menus, for example `D (no key)`.
+The default is Medium Swing; `Med.`, `Med. Swing`, and `Medium` remain aliases. A keyless exercise needs `vamp:tonic` (for example, `C`) as its transposition reference. Modes and local key changes are preserved when generating practice variants. Keyless variants remain unkeyed in the music and show their selected transposition reference in menus, for example `D (no key)`.
 
 Place staff text `vamp:optional-chord` at the same onset as exactly one harmony to mark a turnaround omitted from the final arrangement section. The text may be hidden. Parenthesizing an ordinary chord does not make it optional. Orphaned/duplicate markers and unknown `vamp:` properties are errors; consumed directives never enter runtime JSON.
 
@@ -108,19 +108,19 @@ Practice metadata includes optional `authored_key_signature`, captured before tr
 
 Measure durations, event onsets/durations, harmony offsets, span times, and pitch alterations use `{ "numerator": N, "denominator": D }`. Times are in quarter-note units and alterations in semitones. There is no tick grid, display-duration surrogate, or runtime flattening into one voice. Written note types, dots, and time modifications remain distinct from metric duration. Every event owns a voice; every chord tone has its own ID, written duration, duration, and reciprocal tie links. Tuplets reference events and belong to one voice.
 
-The arranger normalizes instrument transposition to concert pitch and preserves all source voices in source-copy sections. Generated guitar rhythms occupy one lane. The seven-section form, optional turnaround handling, and twelve melody/chord practice keys remain automatic. Generated guitar voicings and chord-practice validation share standard six-string tuning (E2 A2 D3 G3 B3 E4), frets 0-15 inclusive, and a maximum four-fret difference between fretted positions; open strings do not count toward that span. The arranger searches all candidates in that range and retains its existing deterministic scoring and voicing policy. Unplayable performance harmonies still produce a warned rhythmic-slash fallback without changing the structured chord meaning or quantizing microtones.
+The arranger normalizes instrument transposition to concert pitch and preserves all source voices in source-copy sections. Generated guitar rhythms occupy one lane. The seven-section form, optional turnaround handling, and twelve practice keys remain automatic. Generated guitar voicings and practice validation share standard six-string tuning (E2 A2 D3 G3 B3 E4), frets 0-20 inclusive, and a maximum four-fret difference between fretted positions; open strings do not count toward that span. The arranger searches all candidates in that range and retains its existing deterministic scoring and voicing policy. Unplayable performance harmonies still produce a warned rhythmic-slash fallback without changing the structured chord meaning or quantizing microtones.
 
-Chord practice transposes from the source key, not from its first chord: the current C-key cycle begins with G7, so selecting D produces an opening A7. It preserves every authored tone, interval, inversion, rhythmic voice, tie, and tuplet. The closest transposition is tried first, followed by whole-exercise octave alternatives ordered by absolute displacement, with upward movement preferred on ties. Register adjustment does not change harmony symbols or key signatures. Sustained and tied tones keep their strings as other voices enter. This is a fretboard feasibility check, not a guarantee of comfortable fingering, and does not apply the performance arranger's chord-tone omissions or revoicing.
+Practice transposes from the declared source key, not from its first chord. Sources currently declare C major, but the importer does not force C or discard modes and local key changes. For example, a C-key exercise beginning with G7 produces A7 when D is selected. It preserves every authored tone, interval, inversion, rhythmic voice, tie, and tuplet. The closest transposition is tried first, followed by whole-exercise octave alternatives ordered by absolute displacement, with upward movement preferred on ties. Register adjustment does not change harmony symbols or key signatures. Sustained and tied tones keep their strings as other voices enter. This is a fretboard feasibility check, not a guarantee of comfortable fingering, and does not apply the performance arranger's chord-tone omissions or revoicing.
 
-For the fretboard check, ordinary unmarked treble notation uses the project's written-guitar convention (one octave above sounding pitch). An explicit MusicXML instrument transposition is applied instead, once, including an explicit zero interval; the importer records this distinction with `transpose.explicit`. Octave-clef and TAB pitches do not receive the implicit octave subtraction. Nonstandard tunings, capo configurations, microtonal frets, or exercises that cannot retain their voicings in every key fail explicitly. Errors identify the source, target key, and failing passage; existing generated assets are preserved. The current G cycle variant needs a whole-exercise octave increase to stay above the guitar's lowest note.
+For the fretboard check, ordinary unmarked treble notation uses the project's written-guitar convention (one octave above sounding pitch). An explicit MusicXML instrument transposition is applied instead, once, including an explicit zero interval; the importer records this distinction with `transpose.explicit`. Octave-clef and TAB pitches do not receive the implicit octave subtraction. Nonstandard tunings, capo configurations, microtonal frets, or exercises that cannot retain their voicings in every key fail explicitly. Errors identify the source, target key, and failing passage; existing generated assets are preserved. No notes are omitted or revoiced to make a key fit.
 
-Automatic performance arrangements require a pitched, non-TAB lead. Percussion and tablature remain available in the model and melody-practice pipeline, but conversion into guitar/vocal parts fails explicitly rather than reinterpreting percussion positions or fret numbers as ordinary treble-staff notation. Chord practice requires pitched guitar notation. Practice transposition retains authored TAB strings and never invents a new fingering; chord practice additionally checks that the resulting exact integer frets lie within 0-15. Melody practice retains its existing nonnegative-fret behavior.
+Automatic performance arrangements require a pitched, non-TAB lead. Percussion and tablature remain supported in the musical model and engraver, but performance conversion rejects them rather than reinterpreting their pitches. All practice exercises now require pitched, standard-tuning guitar material. Transposition preserves authored TAB strings and validates the resulting exact integer frets against 0-20; it never invents new TAB. The old unchecked melody-practice path is removed.
 
 ### Playback Boundary
 
 All musical compilation and tone synthesis run in Python (`generator/playback.py`). Each transformed score receives a sorted timeline of independent tone events with rational quarter-note `onset` and `end` positions. Ties become one continuous attack. Rests and spacers remain silence, including trailing silence. Harmony labels are not auto-accompaniment: only pitches actually present in the selected arrangement or exercise sound. Cue notes are silent, hidden pitched notes sound, and pitchless rhythm slashes are silent.
 
-The `soft-struck-1` profile generates shared mono 48 kHz PCM16 WAVs with a short attack/decay and a seamless sustain loop, plus a shared 50 ms release envelope. Exact sounding pitches, including supported microtones, determine samples. Explicit instrument transposition is applied once; chord practice and generated guitar parts use the existing written-guitar convention. Samples are shared across all scores and voices by content identity, not duplicated per complete performance. There are no SoundFonts, instrument sample bundles, oscillators, or MusicXML parsing at runtime.
+The `soft-struck-1` profile generates shared mono 48 kHz PCM16 WAVs with a short attack/decay and a seamless sustain loop, plus a shared 50 ms release envelope. Exact sounding pitches, including supported microtones, determine samples. Explicit instrument transposition is applied once; all practice and generated guitar parts use the existing written-guitar convention. Samples are shared across all scores and voices by content identity, not duplicated per complete performance. There are no SoundFonts, instrument sample bundles, oscillators, or MusicXML parsing at runtime.
 
 Grace groups honor explicit steal-time-previous/following or make-time; unspecified groups share 25 percent of the following event. Conflicting grace modes, trailing grace groups, unsupported unpitched sound, tied measured tremolo, and incompatible paired tremolo make playback explicitly unavailable while retaining the notation. Measured tremolo uses written subdivision and tuplet ratios. The tone range is 20-18,000 Hz. Asset validation checks containment, SHA-256, PCM format/length, loop endpoints, envelope shape, event ordering, references, and rational timing. Generation remains transactional if validation fails.
 
@@ -146,7 +146,7 @@ Pages are 976x484 pixels with 12-pixel staff spacing, 28-pixel horizontal margin
 
 ### Validation And Migration
 
-Generation validates sources and complete staged output before replacing the previous asset directory. Replacement is always complete, including when the compatibility `--clean` flag is omitted; deleted sources cannot leave stale JSON, pages, or audio. CMake tracks the input list as well as file contents, so additions and removals regenerate the catalogue. Runtime packages contain `catalogue.db` with embedded semantic JSON, PNG pages, generated PCM WAVs, and the OTF UI text font, not MusicXML, MuseScore, or Python dependencies. Python test discovery runs model/import/export tests; the normal asset build additionally qualifies and audits every catalogue score.
+Generation validates sources and complete staged output before replacing the previous asset directory. Replacement is always complete; deleted sources cannot leave stale JSON, pages, or audio. CMake tracks the input list as well as file contents, so additions and removals regenerate the catalogue. Runtime packages contain `catalogue.db` with embedded semantic JSON, PNG pages, generated PCM WAVs, and the OTF UI text font, not MusicXML, MuseScore, or Python dependencies. Python test discovery runs model/import/export tests; the normal asset build additionally qualifies and audits every catalogue score.
 
 Diagnostics identify the file, part, measure, and unsupported element or invalid relationship. Examples include `part P1, measure 4: unsupported musical element <slur>`, `tie stop has no contiguous matching start`, and `optional-chord marker must identify exactly one harmony`.
 
@@ -192,7 +192,7 @@ Building the full package requires Docker/buildx, the ARM embedded GCC toolchain
 
 Generated score assets are created during CMake build/package and copied into the Zero image context. Static font assets remain tracked under `package/assets/fonts`.
 
-`generator/catalogue.py` uses Python's standard-library SQLite module to construct `catalogue.db` after musical, playback, and engraving validation. The database carries application ID `0x56414D50` and catalogue schema version 1, with complete score bundles and menu metadata. Sorted insertion and the absence of wall-clock timestamps make repeated builds reproducible with the same toolchain. It is closed and integrity-checked before publication, without journal/WAL sidecars. Individual JSON files remain available in the build tree for inspection, but both CMake installation and Docker image assembly exclude them.
+`generator/catalogue.py` uses Python's standard-library SQLite module to construct `catalogue.db` after musical, playback, and engraving validation. The database carries application ID `0x56414D50` and catalogue schema version 2, with complete score bundles and menu metadata. Sorted insertion and the absence of wall-clock timestamps make repeated builds reproducible with the same toolchain. It is closed and integrity-checked before publication, without journal/WAL sidecars. Individual JSON files remain available in the build tree for inspection, but both CMake installation and Docker image assembly exclude them.
 
 The default `database.path` is `/usr/share/product-vamp/catalogue.db`; development configuration points at the native generated-assets database. Stored source paths are logical, asset-relative locations, not build-host filenames. Runtime resolves them against `runtime.assets_path` and rejects traversal and escaping symlinks. The JSON file itself need not exist: its parent identifies the score's page directory. Normal startup checks catalogue compatibility; `--once` additionally runs SQLite's integrity probe and checks all page/audio files. Missing, corrupt, or incompatible catalogues fail rather than creating an empty library. `--ensure-database` is no longer a supported command.
 
@@ -217,4 +217,3 @@ python3 modulus.py lsp
 ```
 
 The `tooling.lsp` section in `modulus.json` owns the Pico CMake configure. It unsets host compiler variables before configuring the Pico SDK project so clangd sees the embedded ARM toolchain. When running directly outside the generated Modulus Docker image, set `PICO_SDK_PATH` before refreshing the Pico database.
-