khodo/product-vamp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Cleanup

Browse Source
This commit is contained in:
khodowany
2026-10-04 17:14:13 -05:00
parent f234dd6d51
commit 68df7292ee
33 changed files with 990 additions and 1872 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/composer/implementation_report.md
+59 -71
View File
@@ -1,106 +1,94 @@
# Deterministic composer repair — implementation report
# Reference-focused composer repair
The active production composer was refactored in place. The four complete new solos and independently generated accompaniment are published through the normal CMake target in `build/build_x86_debug/generated_assets/`. The previous release remains under `build/composer_repair/405066e/baseline/`. This is a technically validated candidate musical version, not a claim of listener approval or resemblance to the optional target compositions.
Starting revision: `f234dd6`. The goal of this revision is to make the active composer construct the musical relationships seen in the four supplied solos, using general rules and original song inputs. It does not import their notes or optimize candidate similarity to them. Exact reconstruction from general theory alone is not claimed.
## What changed
## Actual changes
The previous final-score overwrite is removed. One versioned evaluator supplies the actual selected total, with visible raw values, denominators, weights and contributions. Activity now affects selection. Relative rhythm descriptors replace the absolute-onset proxy. Explicit identity and hold contracts remain hard requirements and are not counted again as duplicate fidelity costs.
The active `themes → gestures → routes` path now chooses harmonic arrival timing, target pitch, sounding duration and release before pitch realization. It can construct an approach into an anticipated chord-tone target with both obligations present. Color destinations are available before route filling. Several compatible treatments can survive within one phrase; the local decoration stage preserves existing contracts instead of replacing them.
The normal constructor proposes complete pitch/rhythm routes from the incoming state, then checks the declared source/theme relationship. It no longer chooses a motif prefix and a separate cheap suffix. Source features are considered throughout the head; treatment diversity survives seed retrieval. This fixed a real-song failure where all eight retained seeds could demand exact intervals and exclude every rhythm/contour alternative.
On-time/delayed arrival pitches and their uninterrupted tails remain fixed through route filling and refinement. Other cell endings stay provisional. Internal long values can be reopened into eighth-note motion where the selected thematic contract permits it; exact rhythm identities and committed closing holds remain protected. Four-member chord traversals are available alongside the existing mapped paths.
Generated rhythmic edits preserve complete binary/tuplet groups. Open windows request continuation rhythm and have no compulsory chord-tone stop. Local settles remain open; cadential completion and handoff have actual held-ending contracts. The original realized theme stays available for returns, while answer, extension, fragment and free continuation are reachable operations. Bounded two-sided join revisions compete with the original phrase.
Two general enclosure formulas use an actually mapped whole-tone upper neighbor. A natural sixth on tonic/predominant minor harmony can be a bounded stable color rather than always requiring immediate neighbor resolution. These rules retain written harmony, omissions, scale restrictions on ordinary notes and physical validation. No new scale families or performance effects were added.
Approaches, enclosures, chromatic connections, colors, anticipations and prepared suspensions now compete over provisional phrases **before selection of the phrase/chorus**. Direct alternatives remain. Exact timing, target octave, preparation, resolution, complete sounding span and protected tails are checked. This is still bounded rewriting of complete provisional phrases, not exhaustive joint search over every possible expression and note.
Reference inspection now includes every one of the 514 sounding events across the four complete scores. Original labels in technical-fingering and other-notation containers are retained. The review checks recognized treatments against **original source harmony**, preserves missing initial harmony, and keeps hypotheses separate from validated rules. Remaining unexplained events are reported instead of automatically authorized.
Acoustic and electric have overlapping soft register preferences instead of source-octave anchoring and compulsory electric transposition. Both retain strings 1–6, frets 1–12, no open strings. The existing phrase-placement path, sparse canonical source-only accompaniment and independent spawn-process architecture were retained. No generated role supplies another role's musical inputs.
## One comparison location
The runtime/notation representation is unchanged. Timing, source harmony, the 21 collection patterns, vocal handoff, source melody/vocal content and physical constraints remain. All four new exported XML sounding streams match their saved role documents exactly. Vocal and electric head-melody role documents are byte-equivalent as parsed JSON to the baseline for both songs.
`build/solo_reference_comparison/` contains the compact previous four XMLs/manifest, current output, full reference comparison/audio, complete theory review and verification logs. Start with `index.html`; `analysis/theory.md` supplies the all-event analysis. Solo 1 means acoustic; Solo 2 means electric.
## Evidence and review artifacts
The references are `build/all-of-me-solo-1.xml`, `all-of-me-solo-2.xml`, `autumn-leaves-solo-1.xml`, and `autumn-leaves-solo-2.xml`. Source songs remain authoritative in `data/songs/`. All six input hashes were recorded before work. The optional adapter applies the separately confirmed guitar-written minus-one-octave convention; production import is unchanged. Autumn Leaves aligns at measure 2 / first Gm7 and retains the preceding measure. Four Autumn electric reference events conflict with the production vocal cutoff; neither input is rewritten.
- [Active call path, confirmed findings and budgets](active_path_audit.md)
- [Production data inventory and provenance](data_migration.md)
- [Authoritative evaluator and toggle semantics](evaluation_contract.md)
- [Capabilities, tests and limits](musical_capability_matrix.md)
- [Independent-input proof](dependency_proof.md)
- [Selected phrase traces and local alternative score breakdowns](../../build/composer_repair/405066e/evidence/phrase_traces.md)
- [Machine-readable operation funnels, hashes and evaluations](../../build/composer_repair/405066e/evidence/evidence.json)
- [Baseline/new/source/plain/strict/final listening page](../../build/composer_repair/405066e/auditions/index.html)
- [Separate optional target comparison](../../build/composer_repair/405066e/references/index.html)
Cleanup removed 1.82 GiB of obsolete generated comparison trees/logs and five superseded tracked audition files. Exact removed paths and sizes are recorded in `cleanup.json`. Build toolchains, reusable composition/engraving caches, source songs and the four references were preserved. The old multi-experiment documentation was replaced with current commands and contracts.
The listening page contains complete solos, excerpts, solo-only audio, fixed comparison backing, and independently assembled final backing. Rendering uses consistent neutral synthesis and levels. Signal checks covered 112 WAV files with zero full-scale samples (largest absolute 16-bit sample 16,377); this checks clipping, not perceptual balance or musical quality. The strict view restores each selected phrase's valid provisional version; it does not remove chromatic notes and leave timing holes. A phrase without decoration remains audible in that view.
## Evidence and boundaries
The local strict/decorated comparison isolates selected expression edits. Baseline versus new production changes multiple mechanisms and cannot identify which one caused a subjective improvement. Separate whole-chorus ablations restoring each superseded defect were **not** produced; broken old objectives were not retained as normal production branches. No ear-based audition was performed or claimed.
- [Active path and verified defects](active_path_audit.md)
- [Resource provenance, consumers and cleanup](data_migration.md)
- [Authoritative evaluation](evaluation_contract.md)
- [Musical capability and limits](musical_capability_matrix.md)
- [Independent input contract](dependency_proof.md)
- [Reproduction commands](commands.md)
## Selected output facts
The six-role independent-input exercise passed with real catalogs: standalone, reversed, cold/warm, two-process, optional targets present/deleted, and after an electric-only setting change. Current hashes and actual read traces are in `dependency/dependency.json`. New proposal and route cache tests compare actual uncached/cached results and caller provenance. This does not establish perceptual quality.
| Song / role | New attacks | Sounding span available | Range, sounding MIDI |
|---|---:|---:|---|
| All of Me acoustic | 120 | 128 quarter beats | 42–64 |
| All of Me electric | 106 | 128 quarter beats | 48–67 |
| Autumn Leaves acoustic | 100 | 128 quarter beats | 49–69 |
| Autumn Leaves electric | 103 | 125 quarter beats | 46–70 |
The final score weights were preserved. Construction coverage is reported separately from selection under `selection_trace.joint_gestures`. A supported relationship may never be selected; matching a reference attack is a different fact again. Complete per-phrase evaluation totals remain inspectable.
All four pass full harmonic validation with no unresolved obligations. These counts/ranges are observations, not quotas or evidence of musical preference. The three protected quarter beats at the returning Autumn Leaves vocal are explicitly excluded from the electric solo's allocation.
The first diagnostic profile exposed repeated approach/rhythm/fretboard work. Bounded caches now reuse exact immutable constraints and restore each caller's provenance. No search beam was widened and no machine-speed timeout selects music. Full build costs and final test outcomes are recorded below; concurrent checks are not treated as an isolated benchmark.
For example, All of Me electric has an open continuation over beats 20–42, a local settle on B3 at 104.5–107.5, and a final G3 handoff at 123–126 followed by space. Its selected actions include rhythm and contour development, answers, fragments, a clear return and a free continuation. All of Me acoustic selects a prepared suspension and an anticipation; these are real validated events, not unused vocabulary labels. The full trace reports proposed/valid/retained candidates and the selected actions separately.
This remains a bounded rule-based composer. Some expressive reference relationships are unsupported or not recognized by the conservative optional parser; this is not proof those notes are musically wrong. Guitar placement is a legal-path check, not full ergonomic certification. No ear-based audition or listener approval is claimed.
Autumn Leaves electric selects more free continuations and no explicitly labeled full return. That is a useful listening question about identity retention, not an invitation to enforce a return quota. Autumn Leaves acoustic repeatedly resolves to D4 in its earlier cadential passages; the report exposes this rather than interpreting pitch diversity as automatic improvement.
## Completed checks
## Reference and source handling
- Full Python suite: **576 tests passed in 1700.428s**. Command: repository Python runner plus `-m unittest discover -s tests -p 'test_*.py'`; full log in `build/solo_reference_comparison/tests.log`. This overlapped the full production build.
- Focused joint-arrival/evaluator/comparison tests: **33 passed**; eight comparison-adapter tests also passed after the presentation update.
- Six-role real-catalog dependency exercise: **passed**, including two spawned processes and reference presence/absence; evidence in `dependency/dependency.json`.
- `git diff --check`: passed at the pre-publication checkpoint.
The authoritative source files are still `data/songs/all-of-me.xml` and `data/songs/autumn_leaves.xml`. Named source/target ZIP archives and supplied review Markdown were not found; existing extracted files were used only in their authorized roles. No source was overwritten from another arrangement.
The comparison table aggregates timing-only retained-unit labels (such as diminution) into their direct construction family. The manifest preserves the original raw diagnostic labels. This reporting normalization does not select or edit music.
The four optional `build/*-solo-1.xml` / `*-solo-2.xml` references are unchanged, verified by SHA-256. Their contents were first used here by a separate post-generation comparison, not to author production rules. Solo 1 maps to acoustic; Solo 2 maps to electric. The adapter preserves raw pitches and applies the explicitly confirmed guitar-written minus-one-octave convention only to its normalized comparison. Autumn Leaves aligns at measure 2 / first Gm7 and retains its preceding measure; All of Me starts at measure 1. All four normalized reference solos have a legal placement under the current domain. This is not a complete ergonomic assessment. Four Autumn electric reference events extend beyond the production vocal-protection cutoff and are reported rather than clipped or imported.
## Published results and comparison
The additional transfer review uses independently transcribed four-measure excerpts from the original editions of After You've Gone, Indiana and Hindustan, with declared harmonic reductions and provenance outside `data/`. All six roles completed for each excerpt. These are limited transfer probes, not complete standard/form coverage or pristine external judgments of quality. No production policy was fitted to those outputs.
Normal production assets are published at `build/build_x86_debug/generated_assets/`. The four MusicXML exports in its `solo_xml/` directory are byte-identical to the copies in `build/solo_reference_comparison/generated/`.
## Performance and reproducibility
| Solo | New attacks | Sounding MIDI range | Exact matching reference attacks | Differing onset positions |
|---|---:|---|---:|---:|
| All of Me acoustic | 104 | 50–69 | 1 | 186 |
| All of Me electric | 100 | 51–67 | 3 | 187 |
| Autumn Leaves acoustic | 106 | 43–62 | 3 | 182 |
| Autumn Leaves electric | 97 | 55–70 | 2 | 170 |
Hardware: Intel Core i7-1355U, 12 logical CPUs; two composition processes and two engraving workers. Deterministic expansion budgets are used; elapsed time never selects a winner.
**None is an exact reference match.** These counts are descriptive, not evidence that the new music is closer perceptually. The earlier outputs had 0/3/0/1 exact matching attacks and 202/191/179/182 differing onset positions respectively. Different attack counts change these comparisons; they must not become a production optimization target.
The full candidate build took **1,576.3 seconds (26.3 minutes)**. Solo caches were cold, accompaniment/head caches were warm, and regression/diagnostic work overlapped; this is an observed build cost, **not an isolated cold benchmark or a clean before/after speedup claim**. Individual cold-role elapsed times were 828.6s / 645.1s for All of Me acoustic/electric and 809.8s / 722.9s for Autumn Leaves acoustic/electric. Worker times overlap and must not be added as total wall time. Engraving rendered 92 new pages, reused two complete scores and took 11.3s wall time.
The new winners contain 9/12/13/13 validated expressive contracts respectively. All of Me electric selects one anticipation; the other three winners select none. Early-arrival candidates are constructed, validate and survive pruning in the traces, so their infrequent final selection is now visible separately from lack of capability. No quota was imposed to conceal that result.
The normal cached CMake rebuild took **25.4 seconds**, with all twelve role caches and all six engraving results reused, while regression tests were running. Both source songs' published solo XML files are identical to the staged validated candidates. The profile files distinguish cache lookup from original search work. A later **isolated warm full build took 12.045 seconds** with twelve role-cache hits and two workers. All twelve role documents, their stable reports, and all four solo XML files matched the staged build exactly; `warm-equivalence.json` records their hashes.
All of Me acoustic contains a complete ascending A3–A#3–B3 connection at beats 12–13, and a mapped A3–F#3 approach to G3 at 64–65. All of Me electric sustains B3 as a valid minor ninth over its Am7 context at beats 28–31. These are selected events, not inserted examples or borrowed reference notes. `analysis/selected_phrases.md` records actual themes, full opening rhythms, every selected treatment, hold completion and alternative score breakdowns. Alternatives shown over equal spans are not falsely presented as the complete solo's globally ranked runners-up.
An isolated single-process four-bar **After You've Gone electric solo** benchmark took **19.924s cold** and **0.930s warm**, excluding interpreter startup and output-report writing. Source analysis took about 0.006s separately. Both used the real production catalogs and identical settings; the cold, warm and profiled runs produced the same canonical document hash (`5b2525a214ed7f30971752434faad2a39ca22adddcfc9947c009e1431b77187b`). These are excerpt-role measurements, not estimates for a complete chorus. No concurrent tests or audition rendering ran during these benchmarks.
The first All of Me acoustic theme still begins with a 4.5-beat source-derived rhythmic hold. This and the small number of selected anticipations demonstrate remaining distance from the supplied examples' phrase construction. The next narrow investigation is the interaction between retained source-rhythm contracts and complete-phrase selection: inspect valid arrival alternatives that lose, and source features that prevent editing a provisional opening. Do not respond by adding an enclosure quota, changing accompaniment, fitting reference counts or claiming that more vocabulary alone solved the problem.
A separate cProfile run of that same cold role took 63.770s including profiler overhead. Nested cumulative times were catalog loading 17.889s, complete composer 44.310s, route candidates 23.073s, harmonic validation 2.949s, expression alternatives 1.625s, placement 0.694s and final evaluation 0.509s. JSON writing was about 0.108s; report/provenance construction is also distributed among composer calls and is not fully isolated by that number. These nested times must not be summed. Full function evidence is in `profile-cold/{role.prof,profile.txt,stages.json}`; the measured cold/warm hash proof is `isolated-role-proof.json`.
## Final verification
A construction profile identified sequence retrieval, repeated ranking/representation and harmonic checks as costs. Ranking now computes each key once per cutoff; immutable interval descriptors are memoized. Stable-order tests verify unchanged ordering for tied ranks. A narrow 800-item/30-repeat microbenchmark returned identical selections (0.826s original versus 0.806s optimized); this is deliberately not presented as a whole-build speedup. More efficient cold search remains unfinished performance work; no beam was widened to hide candidate defects.
- All **12 role documents** agree between the cold staged build and the warm normal publication, including notes, timing and TAB. Evidence: `verification.json`.
- All eight non-solo role documents (accompaniment, electric source melody and vocal for both songs) match the previous release exactly as parsed JSON.
- All four MusicXML sounding streams match their selected production role documents. Full-duration harmonic validity, ending completion, and evaluation-contribution sums pass.
- The two original song files and four supplied references remain SHA-256 identical to their preserved inputs.
- Native command `ctest --test-dir build/build_x86_debug -R '^product_vamp$' --output-on-failure` passed in **33.54s**, with the required localhost socket access.
- The single comparison page contains 24 WAVs: complete generated/reference solos, identical-backing mixes, and opening excerpts. All passed PCM clipping checks. All 36 local page links exist. No subjective listening verdict is claimed.
The dependency exercise used real catalogs and an independently authored source. All six role hashes matched when built alone, in reverse order, in two spawned processes, warm/cold, with invalid optional target files present/absent, and after an electric-only setting change. Actual read traces and a runtime guard accompany equality tests. This does not by itself prove absence of an already embedded derivative; the independently authored data provenance is separately reviewed.
## Measured cost
## Commands and verification
Hardware: Intel Core i7-1355U, 12 logical CPUs; two composition workers. Cold solo generation plus full staged asset build took **2249.06s (37.5 minutes)**. Regression and dependency checks overlapped part of that run, so this is not an isolated speed comparison with the earlier release.
All Python commands below were run with:
| Cold solo role | Seconds |
|---|---:|
| All of Me acoustic | 1041.56 |
| All of Me electric | 850.48 |
| Autumn Leaves acoustic | 1328.58 |
| Autumn Leaves electric | 1170.46 |
```
build/build_common/python/bin/python3 build/build_tools/python_runner.py build/build_common/modulus
```
Worker times overlap. Four All of Me non-solo caches were warm; the four Autumn Leaves non-solo roles regenerated in under one second each. Engraving rendered 90 new pages, reused two scores and took 16.65s wall time. Stage/input/read evidence is in `build-profile.json` and `build-inputs.json`.
Append the relevant module/arguments:
The isolated warm CMake publication took **31.49s**, including about eight seconds of CMake/configuration overhead; asset generation reported 23s. All twelve role caches and all six engraving results were reused. Musical equality was verified, not assumed from cache-hit flags.
```
-m unittest discover -s tests -p 'test_*.py'
-m unittest tests.test_composer_repair tests.test_solo_leads.CompleteRouteTests
-m unittest tests.test_solo_themes.ThemeProduction.test_library_routes_match_the_actual_complete_theme
-m audition.dependency_check --output build/composer_repair/405066e/dependency
-m audition.transfer_review --output build/composer_repair/405066e/transfer-final
-m audition.release_evidence --assets build/composer_repair/405066e/candidate --output build/composer_repair/405066e/evidence
-m audition.production_review --assets build/composer_repair/405066e/candidate --baseline build/composer_repair/405066e/baseline --output build/composer_repair/405066e/auditions
```
An isolated 30-repeat structural-proposal microbenchmark returned exactly equal results: **0.4422s** without the proposal cache versus **0.02193s** with it (shared rhythm/position caches already warm). This is a repeated-calculation measurement, **not** a claimed whole-composer speedup. Cold search is still expensive; candidate construction/validation and repeated thematic alternatives remain performance work.
The full generator invocation and reference comparison are recorded in [reproduction commands](commands.md). Publication used `cmake --build build/build_x86_debug --target vamp_generated_assets -j2`. Native verification used `ctest --test-dir build/build_x86_debug -R '^product_vamp$' --output-on-failure`.
Earlier runs deliberately remain in the evidence directory: the first suite exposed timer leakage in supposedly stable reports and obsolete fixed-closure/phrase-count expectations. Those were repaired. A later 565-test run passed 564 tests and exposed the obsolete requirement for an unedited route in every selected fixture. The corrected provenance test passed (115.4s including class construction); the final full-suite rerun passed **all 565 tests in 737.334 seconds**, recorded in `tests-complete.log`. The focused 19-test repair/route run passed. The native suite initially failed sandbox-denied localhost socket operations; rerunning with the required local socket access passed. After publication, the native CTest target passed again in **18.91 seconds**. These infrastructure failures were not musical validation failures.
## Boundaries and next experiment
- Cold construction is still costly. The next performance step should profile purpose-conditioned route retrieval and repeated provenance work with fixed musical hashes, rather than indiscriminately widening or shrinking search.
- The frontier is bounded and can miss a viable theme/destination combination. Exhaustion is reported as budget exhaustion, not a proof of incompatibility; there is no hidden old-generator fallback.
- The final evaluator is inspectable but intentionally small. It does not establish memorability or predict listener preference.
- Current production chooses complete actions before crossing the search frontier; the partial-treatment API supports bounded pending obligations, but this is not an unbounded multi-phrase optimizer.
- Unsupported written chord stacks still produce explicit no-exact-map diagnostics. Complex importer/form conventions beyond existing support are not newly certified. Original rhythms are never silently quantized.
- Legal TAB and approximate travel costs are not a complete biomechanical model. Independent backing cannot guarantee exact gaps around a solo's realized holds; no reactive recomposition is claimed.
- Listen first to whether the introduced idea remains recognizable in Autumn Leaves electric's freer continuation. Compare the plain/strict/final versions before changing density, return counts or chromatic vocabulary. The optional target comparisons remain outside the selection process.
After verification, duplicate full asset stages, two interrupted stages, temporary tone samples and the binary diagnostic profile were removed. The comparison retains the four current scores, compact previous scores, references through links, useful audio and verification evidence in one directory. No previous output is a generation input.