khodo/product-vamp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Cleanup

Browse Source
This commit is contained in:
khodowany
2026-10-04 22:29:06 -05:00
parent 0e68f5364b
commit ef62af892b
40 changed files with 1754 additions and 789 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/composer/implementation_report.md
+46 -54
View File
@@ -1,84 +1,76 @@
# Complete-phrase solo rewrite
# Exact-target composer repair — release status
Starting revision: `68df7292ee85918a0cdbd0965139ccb2ceecb130`. This revision replaces the active solo constructor. Source import, harmony, exact timing, typed expression validation, independent roles, guitar placement, notation, playback and atomic asset publication remain in use.
**Not released. Exact production reproduction of the four targets has not been achieved.** Valid derivations exist for all four complete event streams, but a target-constrained derivation is not an unrestricted production winner. The existing published assets remain intact. The previous release's report is [archived here](baseline_0e68f53.md); its tests and performance numbers do not apply to this rewrite.
The selected direction is a new melody over the source changes, with optional head references. The source head no longer supplies a compulsory rhythmic prefix or protected initial hold. The four supplied solos remain outside production and unchanged; exact reproduction or listener preference is not claimed.
Starting revision: `0e68f5364bb0196ab1b7087ecc186f5d7446c42b`. Preserved baseline: `build/solo_reference_comparison/repair/baseline/`. Its `inputs.json` records source/data/reference hashes and initial cache state. The sources remain `data/songs/all-of-me.xml` and `data/songs/autumn_leaves.xml`; no archive source has replaced either.
## Construction and simplification
## Acceptance and target identity
The normal path is `production.generate_role → composer.compose → construction.Builder → rhythm.Rhythms/routes.Routes → scoring.evaluate → placement → validation → notation/export`.
The exact target files are `build/all-of-me-solo-1.xml`, `build/all-of-me-solo-2.xml`, `build/autumn-leaves-solo-1.xml`, and `build/autumn-leaves-solo-2.xml`. Solo 1 is acoustic; Solo 2 is electric. They are untouched. The optional adapter applies the confirmed guitar-written octave convention and merges ties while preserving repeated attacks. Autumn aligns at the first Gm7 in measure 2, including the entire preceding measure at negative chorus-relative times. All of Me aligns at measure 1.
Each proposal chooses a phrase purpose, rhythm groups, direction, harmonic destination, arrival timing, sounding duration and optional treatment together. Complete pitch paths realize those choices. Open continuations need no chord-tone stop; settling, completion and handoff have actual destination contracts. Anticipations, approaches, enclosures, connections, colors, blues treatments and prepared suspensions enter before route selection and must validate as complete relationships.
The latest approved development scope allows target-guided general-rule work and shared-coefficient fitting, but prohibits stored tunes, per-song recipes and runtime target reads. No fitted model is promoted automatically. Optional development tools and their artifacts remain outside production imports and cache identities.
An introduced realized phrase supplies the home idea. Development, answer, deliberate return and optional head reference are distinct reachable actions. A return protects a short heard rhythm and directional signature with harmonic adaptation; ordinary phrases have no compulsory three-note prefix. This signature is not a perceptual guarantee of recognition.
The baseline exact comparison returned failure: All of Me acoustic/electric had 0/3 exact matching attacks; Autumn acoustic/electric had 0/0. Differing onset positions were 205, 201, 211 and 178 respectively, and rests differed in all four. These are exact-event diagnostics, not musical-quality scores.
One versioned evaluator reports purpose, incoming continuity, contextual repetition and broad register costs. Physical travel selects fingering, not the winning tune. One-phrase lookahead averages normalized completed-phrase scores across variable durations; it remains a bounded heuristic, not a whole-chorus optimum. Route construction has its own explicitly separate heuristic.
## Changes in this working tree
The solo package decreases from **6,232 to 1,729 Python lines**, and from 30 to 19 modules. Thirteen obsolete constructors/helpers were removed and two focused modules added. Old plan/draft/paraphrase API tests were retired; retained musical and representation contracts were migrated. Removed tests are not counted as passing tests. No legacy production fallback remains.
The hash-preselected phrase constructor and separate route solver are replaced by enumerable joint musical actions, one interpretable rational evaluator and bounded deterministic backtracking. Actions include ordinary notes/rests, complete degree routes, realized-theme returns/answers, colors, approaches, enclosures and anticipations. Pending treatments carry their actual future targets and deadlines across actions. Whole-line guitar placement follows selection; it cannot change the tune or inspect another generated part.
## Inputs and data
General expressive rules now cover delayed exact-target approaches, separated enclosures, reattacked anticipations and anticipated colors. Color octave enumeration no longer loses alternating F-sharp registers through rounding. The base scale vocabulary remains the same 21 collections; exceptions have typed, bounded contracts. Full target endings and source-derived pickup allocations are supported. Electric may overlap the returning vocal pickup as authorized; source vocal events remain unchanged.
The 21 scales, 2,954 sequence paths and 3,538 rhythm snippets remain intact. `solo.json` contains independently authored parameterized actions, nine atomic rhythm groups and migrated authored cells. `grammar.json` and `melodic_vocabulary.json` were retired after their consumers migrated. Accompaniment fields and resources were not changed.
Performance changes remove redundant parent catalog parsing, add content-addressed non-executable catalog JSON, memoize pure chord/color/spelling checks, and avoid building full rational contribution reports for thousands of rejected candidates. The lightweight ranking total and detailed evaluation are equal and tested. Existing independent process workers remain available.
References: `build/all-of-me-solo-1.xml`, `all-of-me-solo-2.xml`, `autumn-leaves-solo-1.xml`, `autumn-leaves-solo-2.xml`. Solo 1 maps to acoustic; Solo 2 to electric. The optional reader applies the confirmed guitar-written octave convention and preserves raw evidence. Autumn Leaves aligns at the first Gm7 in reference measure 2; its preceding measure remains in the comparison. Four Autumn electric reference events extend beyond the source-defined vocal allocation. Neither the target nor the vocal timing is changed to force equality.
The development fitter uses only shared whitelisted musical features and integer updates. An initial trial exposed two defects: unordered source-interval evidence and per-attack rewards that favored subdivisions. Ordered source relationships and duration-normalized rewards repaired them. A later trace exposed indistinguishable pending targets; their relative delay, step and hold now enter selection explicitly. Interrupted diagnostic trials are not reported as completed fitting or benchmark results.
All six strings remain available on frets 1–12 without open strings. Ties retain one sounding event and one position. No performance-expression changes were introduced. The existing independently generated accompaniment, electric melody and vocals are intentionally preserved by this solo rewrite.
## Evidence and limitations
## Verification and review
The staged build completed with `--check`: six performance scores, complete solo XML, notation/TAB, playback assets, and independent accompaniment for both current sources. It is under `build/solo_reference_comparison/repair/staged`, not the published asset directory. The exact gate returned exit code 1:
The final Python command was the repository Python runner followed by `-m unittest discover -s tests -p 'test_*.py'`: **435 tests passed in 218.783 seconds**. During implementation, repeat-build tests caught profiling timings leaking into stable diagnostics. Timings now live only in the separate build profile; the final suite includes those repeat-build checks. Earlier interrupted/stale-import runs are not counted as passing verification.
| Output | Generated / target attacks | Exactly matching attacks | Differing onset positions | Exact result |
|---|---:|---:|---:|---|
| All of Me acoustic | 118 / 137 | 2 | 189 | Fail |
| All of Me electric | 128 / 141 | 0 | 232 | Fail |
| Autumn Leaves acoustic | 106 / 121 | 1 | 177 | Fail |
| Autumn Leaves electric | 143 / 115 | 5 | 195 | Fail |
The explicit dependency command passed all six roles on independently authored input: alone/cold, reverse/warm, parallel/cold, invalid optional targets present, targets removed, and after an electric-only setting change. Actual reads and canonical document hashes are retained. This synthetic test is not a whole-song quality assessment.
Rests differ in every case. These results are not close reproduction, and attack counts are not a substitute for matching the requested music. The full comparison page is `build/solo_reference_comparison/repair/comparison/index.html`. It includes untouched references, the baseline and staged XML, complete solo-only and fixed-backing WAVs, opening excerpts, and exact note differences. Rendering used one neutral treatment and constant shared gains; no ear-based audition was performed.
The optional transfer command passed all 18 role builds on evaluation-only four-bar excerpts from After You've Gone, Hindustan and Indiana. Their provenance remains under `tests/evaluation_sources/`. This demonstrates limited transfer; it does not establish arbitrary meter, form or chord support.
The actual selected events and TAB were checked against all four reparsed exports: all agree. Thus these discrepancies arise in composition/selection, not from an exporter shifting the selected melody. Source vocal and electric head documents are unchanged from the baseline. Accompaniment documents changed with the shared difficulty configuration and the source-derived participation allocation; they are still independently generated. Source and reference SHA-256 values, and the previously published solo exports, remain unchanged. Evidence: `repair/asset-verification.json`.
The comparison page includes full generated, previous and reference solo audio, common-backing mixes and opening excerpts. The same neutral renderer and constant comparison gains are used. Comparison reads happen only after generation. No ear-based audition or listener approval is claimed.
The fitting trial was stopped after **40 completed passes of its configured 200-pass maximum**, without convergence. Every pass still had four incorrect rollouts; the best had only eight matching prefix attacks in total across the four cases. This is an interrupted diagnostic trial, not a completed 200-pass result and not a proof that every shared model is infeasible. Its best coefficients remain development-only under `repair/fitting`; none were copied to production data.
## Evidence locations
A further normal-production check of that best checkpoint completed All of Me acoustic: **5 matching attacks, 223 differing onset positions, and different rests**; it also failed the exact gate. The other fitted-model jobs were interrupted without completed exports. This is not a four-case fitted-model release test. Its completed XML and comparison are in `repair/fitted` and `repair/fitted-comparison`; the interruption is recorded in `repair/fitted-build-status.json`. The complete four-case results above use the unfitted production defaults.
- [Active path and findings](active_path_audit.md)
- [Resource/code migration and provenance](data_migration.md)
- [Evaluation contract and search limitation](evaluation_contract.md)
- [Capability matrix](musical_capability_matrix.md)
- [Independent-input verification](dependency_proof.md)
- [Reproduction commands](commands.md)
The default selection trace explains an important failure. All of Me acoustic selected 29 arpeggio actions and six returns; electric selected 32 scalar actions, nine returns and one ordinary tone. No standalone rest won in either output. Many legal expressive candidates were proposed and rejected by score. The role's action-kind preferences overwhelm much of the vocabulary, and the local policy has no complete-phrase destination comparison. Adding legal actions has not repaired that selection limitation.
`build/solo_reference_comparison/` is the sole comparison location. `previous/` retains the immediate pre-rewrite output and input hashes; `generated/` contains the replacement. The complete operation/pruning trace and local/next-phrase evaluations are in the manifest. `analysis/selected_phrases.md` provides readable selected events, destinations, holds, expression evidence and alternative scores.
### Commands actually validated
## Remaining boundaries
All commands use `build/build_common/python/bin/python3 build/build_tools/python_runner.py build/build_common/modulus` as their prefix. Exact invocations are in [commands.md](commands.md).
The reference solos are outside-review examples, not a numerical loss. The replacement output remains new music, not those four scores. Counts and matching attacks do not establish greater musical quality.
- `-m unittest discover -s tests -p 'test_*.py'`: **446 tests passed in 348.905 seconds** (`repair/verified-tests.log`).
- `-m unittest tests.test_solo_comparison`: **9 tests passed in 0.083 seconds** after removing an invalid optional report link (`repair/comparison-tests.log`).
- `-m audition.dependency_check --output .../dependency-final`: passed all six roles, including an operational electric register change that changed that role's document and left the other roles intact.
- Full staged `-m generator ... --check --jobs 2 --composition-jobs 2 --export-solo-xml`: passed. The first attempt exposed an unfillable final quarter-beat; a tempo-gated sixteenth note/rest primitive and a regression test repaired that failure before the successful build.
- `-m audition.solo_compare ... --require-exact`: failed all four outputs, as shown above. This is the release-blocking musical result, not a passing check.
- Full-score selected-event/TAB verification and source/reference/published-baseline hash checks: passed (`repair/asset-verification.json`).
The search retains a small frontier and limited lookahead. It can choose another complete phrase, but does not perform arbitrary two-sided revision of committed music. Some exact thematic returns may still sound too weak; the selected rhythm and contour contract is less strict than exact intervals. Acceptable candidates can share a zero musical penalty, leaving the stable role key to break ties.
The staged asset run took **430.402 seconds** with two composition workers and warm generic catalog/non-solo caches. Tests and development fitting were running concurrently, so that time is not an isolated speed benchmark. Solo role times were 156.117, 228.292, 182.301 and 174.160 seconds in that run. No cold-generation speedup is claimed from these numbers. The full recorded stage timings and actual read traces are in `repair/build-profile.json`.
Unsupported written harmony produces an explicit diagnostic. Optional expression rules do not repair ambiguous source chords or authorize arbitrary chromatic notes. Legal TAB and coarse tempo limits are not full biomechanical certification. The next useful listening check is the four complete comparisons: whether their openings establish memorable ideas and whether later returns remain recognizable through the freer continuations.
After other generation/rendering jobs stopped, `-m audition.benchmark --source data/songs/all-of-me.xml --role acoustic_solo --cache .../performance/cache --output .../performance/cold` took **134.722 seconds** for the role, plus 0.046 seconds for source analysis, with one worker and an empty concrete/catalog cache. Repeating it with `--output .../performance/warm` took **2.068 seconds** with a role-cache hit. Hardware/runtime: 12 reported CPUs, Linux 7.2.3 x86-64, repository Python environment; cProfile disabled. Cold, warm and parallel-full-build documents have the same hash `adcc372315aa10cae8a956cc5002700859c81cdcf4eeaa59b4ae6f8b41161baf`. Evidence: `repair/performance/summary.json` and the two benchmark records.
## Published assets and measured cost
The requested cold-generation reduction is **not achieved**. Warm memoization avoids recomposition, but the wider action enumeration is still expensive. The historical baseline's different music, algorithm and benchmark conditions do not support a controlled speedup ratio. Candidate construction/evaluation remains the dominant recorded stage; no further beam expansion is justified by these results.
Full assets are published at `build/build_x86_debug/generated_assets/`; the four exported solos are in `solo_xml/`. All twelve musical role documents match between cold staged generation and cached publication. The eight non-solo documents are exactly equal to the preserved baseline. Every exported solo sounding event and TAB position matches the selected stream. All committed holds and typed harmonic contracts passed validation. All six source/reference file hashes are unchanged.
- [Active path, reproduced findings and search limits](active_path_audit.md)
- [Data migration and provenance](data_migration.md)
- [Authoritative evaluation and fitting contract](evaluation_contract.md)
- [Capabilities and unimplemented behavior](musical_capability_matrix.md)
- [Independent-input checks](dependency_proof.md)
- [Reproduction and exact-release commands](commands.md)
| Solo | New attacks | Sounding MIDI range | Occupancy |
|---|---:|---|---:|
| all_of_me/acoustic | 117 | 47–67 | 0.781 |
| all_of_me/electric | 139 | 52–72 | 0.824 |
| autumn_leaves/acoustic | 153 | 47–67 | 0.859 |
| autumn_leaves/electric | 109 | 52–70 | 0.732 |
Detailed winning/runner-up contributions, selected note groups and proposal/valid/retained/selected counts are in `build/solo_reference_comparison/repair/decisions.md`. Superseded diagnostic logs are consolidated in `repair/diagnostic-history.zip`; current evidence, the baseline and targets remain intact.
These descriptive measurements are not targets or proof of musical improvement. None of the four generated solos exactly matches its reference. All four normalized references have legal pitch/position paths within the existing guitar constraints; the Autumn electric vocal-allocation conflict remains explicit.
The search is a local action policy, not a global complete-phrase optimizer. Dedicated suspension/blues constructors and two-sided phrase revision are not present in this replacement; validators remain. Source influence and the short realized home theme are limited models of melodic identity. A catalog containing thousands of paths does not mean all those paths currently compete as complete musical ideas. No new held-out source-standard evaluation set was added. Existing synthetic transposition/meter/harmony tests do not establish generalization to an unseen standard or subjective musical quality.
The native integration test `ctest --test-dir build/build_x86_debug -R '^product_vamp$' --output-on-failure` passed in **30.45 seconds**, with required localhost access. The optional comparison adapter suite passed **8 tests in 0.018 seconds** after the audio-directory update. All **36 comparison WAVs** passed PCM clipping checks, and all **50 page links** resolved. The signal review is not an ear-based audition.
The earliest demonstrated remaining failure is selection: legal target-compatible actions lose even at the opening, and later action choices collapse toward a few favored families. The next justified investigation is whether a shared context model can distinguish those conflicting choices over complete phrases, before adding more rules, expanding the beam, or running another long fit. The finite trial here does not settle that question. There is no automatic fallback to reference notes or previous output.
The isolated final staged asset build took **107.15 seconds** wall time, including **91.31 seconds** in the performance-part stage. All twelve role caches missed. Two composition workers ran in separate processes; no tests or audition rendering overlapped. Engraving was warm and reused all six scores. The **27.57-second** normal CMake publication reused all twelve role caches and six engraving results.
| Cold solo role | Role seconds | Construction seconds | Route seconds |
|---|---:|---:|---:|
| all-of-me/electric_solo | 40.82 | 26.27 | 21.47 |
| all-of-me/acoustic_solo | 44.38 | 29.96 | 24.90 |
| autumn_leaves/electric_solo | 35.56 | 33.09 | 28.31 |
| autumn_leaves/acoustic_solo | 44.04 | 41.71 | 35.80 |
Worker times overlap. Role time includes resource setup, notation and validation beyond construction. Route time is included in construction, not an additional cost. Hardware/process evidence: 12 logical CPUs, Linux x86-64, repository Python 3.13; `build-profile.json` contains actual reads, worker settings and stage times. The first worker use still spends time loading the full catalogs. Route search remains the dominant construction cost.
An isolated one-process All of Me acoustic build took **39.80 seconds** without cProfile. The diagnostic run before the performance-only optimization took 163.09 seconds **with cProfile enabled**, so those times are not a valid speedup ratio. Profiling exposed repeated pitch hashing/stringification and harmony lookups. Memoization and hoisted identical calculations preserved the complete musical document hash `2b6b35011e353d166d287a8bfff6074723e62559b5e64ff978084623e1520e67`, also matching final production. The rewrite changes music and search structure; its timing is not presented as a semantics-preserving speedup over the old composer.
Cleanup removed **151.2 MiB** of superseded comparison audio, stages, probes and temporary rendering tones during this rewrite. The immediate baseline, four user references, build toolchains, caches, published assets and useful verification evidence remain. Current audio is consolidated under `audio/` and linked from one `index.html`; `cleanup.json` records removed paths.
Technical completion, exact target equality and listener preference are separate. No ear-based audition is claimed. The release remains blocked unless all four unrestricted exported comparisons pass; a valid or stylistically plausible mismatch is not substituted for that requirement.