Save

docs/composer/implementation_report.md

@@ -0,0 +1,106 @@
+# Deterministic composer repair — implementation report
+
+The active production composer was refactored in place. The four complete new solos and independently generated accompaniment are published through the normal CMake target in `build/build_x86_debug/generated_assets/`. The previous release remains under `build/composer_repair/405066e/baseline/`. This is a technically validated candidate musical version, not a claim of listener approval or resemblance to the optional target compositions.
+
+## What changed
+
+The previous final-score overwrite is removed. One versioned evaluator supplies the actual selected total, with visible raw values, denominators, weights and contributions. Activity now affects selection. Relative rhythm descriptors replace the absolute-onset proxy. Explicit identity and hold contracts remain hard requirements and are not counted again as duplicate fidelity costs.
+
+The normal constructor proposes complete pitch/rhythm routes from the incoming state, then checks the declared source/theme relationship. It no longer chooses a motif prefix and a separate cheap suffix. Source features are considered throughout the head; treatment diversity survives seed retrieval. This fixed a real-song failure where all eight retained seeds could demand exact intervals and exclude every rhythm/contour alternative.
+
+Generated rhythmic edits preserve complete binary/tuplet groups. Open windows request continuation rhythm and have no compulsory chord-tone stop. Local settles remain open; cadential completion and handoff have actual held-ending contracts. The original realized theme stays available for returns, while answer, extension, fragment and free continuation are reachable operations. Bounded two-sided join revisions compete with the original phrase.
+
+Approaches, enclosures, chromatic connections, colors, anticipations and prepared suspensions now compete over provisional phrases **before selection of the phrase/chorus**. Direct alternatives remain. Exact timing, target octave, preparation, resolution, complete sounding span and protected tails are checked. This is still bounded rewriting of complete provisional phrases, not exhaustive joint search over every possible expression and note.
+
+Acoustic and electric have overlapping soft register preferences instead of source-octave anchoring and compulsory electric transposition. Both retain strings 1–6, frets 1–12, no open strings. The existing phrase-placement path, sparse canonical source-only accompaniment and independent spawn-process architecture were retained. No generated role supplies another role's musical inputs.
+
+The runtime/notation representation is unchanged. Timing, source harmony, the 21 collection patterns, vocal handoff, source melody/vocal content and physical constraints remain. All four new exported XML sounding streams match their saved role documents exactly. Vocal and electric head-melody role documents are byte-equivalent as parsed JSON to the baseline for both songs.
+
+## Evidence and review artifacts
+
+- [Active call path, confirmed findings and budgets](active_path_audit.md)
+- [Production data inventory and provenance](data_migration.md)
+- [Authoritative evaluator and toggle semantics](evaluation_contract.md)
+- [Capabilities, tests and limits](musical_capability_matrix.md)
+- [Independent-input proof](dependency_proof.md)
+- [Selected phrase traces and local alternative score breakdowns](../../build/composer_repair/405066e/evidence/phrase_traces.md)
+- [Machine-readable operation funnels, hashes and evaluations](../../build/composer_repair/405066e/evidence/evidence.json)
+- [Baseline/new/source/plain/strict/final listening page](../../build/composer_repair/405066e/auditions/index.html)
+- [Separate optional target comparison](../../build/composer_repair/405066e/references/index.html)
+
+The listening page contains complete solos, excerpts, solo-only audio, fixed comparison backing, and independently assembled final backing. Rendering uses consistent neutral synthesis and levels. Signal checks covered 112 WAV files with zero full-scale samples (largest absolute 16-bit sample 16,377); this checks clipping, not perceptual balance or musical quality. The strict view restores each selected phrase's valid provisional version; it does not remove chromatic notes and leave timing holes. A phrase without decoration remains audible in that view.
+
+The local strict/decorated comparison isolates selected expression edits. Baseline versus new production changes multiple mechanisms and cannot identify which one caused a subjective improvement. Separate whole-chorus ablations restoring each superseded defect were **not** produced; broken old objectives were not retained as normal production branches. No ear-based audition was performed or claimed.
+
+## Selected output facts
+
+| Song / role | New attacks | Sounding span available | Range, sounding MIDI |
+|---|---:|---:|---|
+| All of Me acoustic | 120 | 128 quarter beats | 42–64 |
+| All of Me electric | 106 | 128 quarter beats | 48–67 |
+| Autumn Leaves acoustic | 100 | 128 quarter beats | 49–69 |
+| Autumn Leaves electric | 103 | 125 quarter beats | 46–70 |
+
+All four pass full harmonic validation with no unresolved obligations. These counts/ranges are observations, not quotas or evidence of musical preference. The three protected quarter beats at the returning Autumn Leaves vocal are explicitly excluded from the electric solo's allocation.
+
+For example, All of Me electric has an open continuation over beats 20–42, a local settle on B3 at 104.5–107.5, and a final G3 handoff at 123–126 followed by space. Its selected actions include rhythm and contour development, answers, fragments, a clear return and a free continuation. All of Me acoustic selects a prepared suspension and an anticipation; these are real validated events, not unused vocabulary labels. The full trace reports proposed/valid/retained candidates and the selected actions separately.
+
+Autumn Leaves electric selects more free continuations and no explicitly labeled full return. That is a useful listening question about identity retention, not an invitation to enforce a return quota. Autumn Leaves acoustic repeatedly resolves to D4 in its earlier cadential passages; the report exposes this rather than interpreting pitch diversity as automatic improvement.
+
+## Reference and source handling
+
+The authoritative source files are still `data/songs/all-of-me.xml` and `data/songs/autumn_leaves.xml`. Named source/target ZIP archives and supplied review Markdown were not found; existing extracted files were used only in their authorized roles. No source was overwritten from another arrangement.
+
+The four optional `build/*-solo-1.xml` / `*-solo-2.xml` references are unchanged, verified by SHA-256. Their contents were first used here by a separate post-generation comparison, not to author production rules. Solo 1 maps to acoustic; Solo 2 maps to electric. The adapter preserves raw pitches and applies the explicitly confirmed guitar-written minus-one-octave convention only to its normalized comparison. Autumn Leaves aligns at measure 2 / first Gm7 and retains its preceding measure; All of Me starts at measure 1. All four normalized reference solos have a legal placement under the current domain. This is not a complete ergonomic assessment. Four Autumn electric reference events extend beyond the production vocal-protection cutoff and are reported rather than clipped or imported.
+
+The additional transfer review uses independently transcribed four-measure excerpts from the original editions of After You've Gone, Indiana and Hindustan, with declared harmonic reductions and provenance outside `data/`. All six roles completed for each excerpt. These are limited transfer probes, not complete standard/form coverage or pristine external judgments of quality. No production policy was fitted to those outputs.
+
+## Performance and reproducibility
+
+Hardware: Intel Core i7-1355U, 12 logical CPUs; two composition processes and two engraving workers. Deterministic expansion budgets are used; elapsed time never selects a winner.
+
+The full candidate build took **1,576.3 seconds (26.3 minutes)**. Solo caches were cold, accompaniment/head caches were warm, and regression/diagnostic work overlapped; this is an observed build cost, **not an isolated cold benchmark or a clean before/after speedup claim**. Individual cold-role elapsed times were 828.6s / 645.1s for All of Me acoustic/electric and 809.8s / 722.9s for Autumn Leaves acoustic/electric. Worker times overlap and must not be added as total wall time. Engraving rendered 92 new pages, reused two complete scores and took 11.3s wall time.
+
+The normal cached CMake rebuild took **25.4 seconds**, with all twelve role caches and all six engraving results reused, while regression tests were running. Both source songs' published solo XML files are identical to the staged validated candidates. The profile files distinguish cache lookup from original search work. A later **isolated warm full build took 12.045 seconds** with twelve role-cache hits and two workers. All twelve role documents, their stable reports, and all four solo XML files matched the staged build exactly; `warm-equivalence.json` records their hashes.
+
+An isolated single-process four-bar **After You've Gone electric solo** benchmark took **19.924s cold** and **0.930s warm**, excluding interpreter startup and output-report writing. Source analysis took about 0.006s separately. Both used the real production catalogs and identical settings; the cold, warm and profiled runs produced the same canonical document hash (`5b2525a214ed7f30971752434faad2a39ca22adddcfc9947c009e1431b77187b`). These are excerpt-role measurements, not estimates for a complete chorus. No concurrent tests or audition rendering ran during these benchmarks.
+
+A separate cProfile run of that same cold role took 63.770s including profiler overhead. Nested cumulative times were catalog loading 17.889s, complete composer 44.310s, route candidates 23.073s, harmonic validation 2.949s, expression alternatives 1.625s, placement 0.694s and final evaluation 0.509s. JSON writing was about 0.108s; report/provenance construction is also distributed among composer calls and is not fully isolated by that number. These nested times must not be summed. Full function evidence is in `profile-cold/{role.prof,profile.txt,stages.json}`; the measured cold/warm hash proof is `isolated-role-proof.json`.
+
+A construction profile identified sequence retrieval, repeated ranking/representation and harmonic checks as costs. Ranking now computes each key once per cutoff; immutable interval descriptors are memoized. Stable-order tests verify unchanged ordering for tied ranks. A narrow 800-item/30-repeat microbenchmark returned identical selections (0.826s original versus 0.806s optimized); this is deliberately not presented as a whole-build speedup. More efficient cold search remains unfinished performance work; no beam was widened to hide candidate defects.
+
+The dependency exercise used real catalogs and an independently authored source. All six role hashes matched when built alone, in reverse order, in two spawned processes, warm/cold, with invalid optional target files present/absent, and after an electric-only setting change. Actual read traces and a runtime guard accompany equality tests. This does not by itself prove absence of an already embedded derivative; the independently authored data provenance is separately reviewed.
+
+## Commands and verification
+
+All Python commands below were run with:
+
+```
+build/build_common/python/bin/python3 build/build_tools/python_runner.py build/build_common/modulus
+```
+
+Append the relevant module/arguments:
+
+```
+-m unittest discover -s tests -p 'test_*.py'
+-m unittest tests.test_composer_repair tests.test_solo_leads.CompleteRouteTests
+-m unittest tests.test_solo_themes.ThemeProduction.test_library_routes_match_the_actual_complete_theme
+-m audition.dependency_check --output build/composer_repair/405066e/dependency
+-m audition.transfer_review --output build/composer_repair/405066e/transfer-final
+-m audition.release_evidence --assets build/composer_repair/405066e/candidate --output build/composer_repair/405066e/evidence
+-m audition.production_review --assets build/composer_repair/405066e/candidate --baseline build/composer_repair/405066e/baseline --output build/composer_repair/405066e/auditions
+```
+
+The full generator invocation and reference comparison are recorded in [reproduction commands](commands.md). Publication used `cmake --build build/build_x86_debug --target vamp_generated_assets -j2`. Native verification used `ctest --test-dir build/build_x86_debug -R '^product_vamp$' --output-on-failure`.
+
+Earlier runs deliberately remain in the evidence directory: the first suite exposed timer leakage in supposedly stable reports and obsolete fixed-closure/phrase-count expectations. Those were repaired. A later 565-test run passed 564 tests and exposed the obsolete requirement for an unedited route in every selected fixture. The corrected provenance test passed (115.4s including class construction); the final full-suite rerun passed **all 565 tests in 737.334 seconds**, recorded in `tests-complete.log`. The focused 19-test repair/route run passed. The native suite initially failed sandbox-denied localhost socket operations; rerunning with the required local socket access passed. After publication, the native CTest target passed again in **18.91 seconds**. These infrastructure failures were not musical validation failures.
+
+## Boundaries and next experiment
+
+- Cold construction is still costly. The next performance step should profile purpose-conditioned route retrieval and repeated provenance work with fixed musical hashes, rather than indiscriminately widening or shrinking search.
+- The frontier is bounded and can miss a viable theme/destination combination. Exhaustion is reported as budget exhaustion, not a proof of incompatibility; there is no hidden old-generator fallback.
+- The final evaluator is inspectable but intentionally small. It does not establish memorability or predict listener preference.
+- Current production chooses complete actions before crossing the search frontier; the partial-treatment API supports bounded pending obligations, but this is not an unbounded multi-phrase optimizer.
+- Unsupported written chord stacks still produce explicit no-exact-map diagnostics. Complex importer/form conventions beyond existing support are not newly certified. Original rhythms are never silently quantized.
+- Legal TAB and approximate travel costs are not a complete biomechanical model. Independent backing cannot guarantee exact gaps around a solo's realized holds; no reactive recomposition is claimed.
+- Listen first to whether the introduced idea remains recognizable in Autumn Leaves electric's freer continuation. Compare the plain/strict/final versions before changing density, return counts or chromatic vocabulary. The optional target comparisons remain outside the selection process.