product-vamp/docs/composer/implementation_report.md

# Complete-phrase selection and shared-rule learning

**Exact reproduction is not achieved. The published baseline is unchanged.** The four complete authored solos remain the acceptance criterion. Legal construction, matching prefixes, learned rankings and passing tests do not satisfy it.

Starting revision: `ef62af892bce168e2e9846dc6035b414f55d8959`. The preceding report is [archived](baseline_ef62af8.md). Current evidence is consolidated in `build/solo_reference_comparison/structured/`; `baseline/` preserves the previous complete comparison output. Saved output is never a production input.

## Active implementation

The production path is `production.generate_role → solo.composer.compose → construction.Builder → placement → validation/export`. The former builder committed the cheapest next action and backtracked only after failure. It now retains histories through common exact-time frontiers and selects among completed event streams. Limits are 16 histories, 24 successors per history and eight completed exports. A route, sounding tail and pending harmonic obligation survive a computational frontier.

`Selector.complete` is the authoritative `phrase-selection-2` evaluator. It merges explicit ties, reconstructs silence and reports the contributions actually used for selection. Action labels no longer earn numerical rewards. `scale_thirds` means alternate scale-degree walks; `arpeggio` walks written chord members. A completed opening owns the home identity, separately from the eight-note recent-history buffer.

A controlled test retains a locally more expensive opening and selects it when it produces the better completed phrase. Tests also cover partition invariance, pending continuation, arpeggios, home identity, deferred validation and exact incremental scoring. These establish selector behavior, not aesthetic quality or global optimality.

No source song, scale family, rhythm catalog, accompaniment vocabulary, fret domain, runtime schema or performance expression changed. Production data changes remove action-family bonuses and replace the backtrack budget. Missing/no-collection harmony now rejects optional colors before parsing `N.C.`. Fast retrieval uses canonical evaluation for tied fragments and learned contextual rules.

## Complete default-build result

The staged build passed `--check` and produced six arrangement scores, complete solos and independent accompaniment for both songs. It is in `structured/staged`, not the published asset directory.

| Output | Generated / target attacks | Matching pitch/onset/release attacks | Full exact result |
|---|---:|---:|---|
| All of Me acoustic | 116 / 137 | 0 | Fail |
| All of Me electric | 333 / 141 | 2 | Fail |
| Autumn Leaves acoustic | 164 / 121 | 1 | Fail |
| Autumn Leaves electric | 311 / 115 | 6 | Fail |

Rests differ in every case. These are substantial failures, especially the electric parts' continuous activity. Counts are diagnostics, not substitute acceptance thresholds. The exact comparison command returned **1**.

The [comparison page](../../build/solo_reference_comparison/structured/comparison/index.html) includes baseline, candidate and unchanged reference XML; complete solo-only and fixed-backing WAVs; opening excerpts; final independent backing; and exact differences. Rendering uses consistent neutral treatment and constant gains. **No ear-based audition was performed.**

Reparsed exports agree with all four selected event streams. Every TAB position is legal; ties retain position. Harmonic validation reports no unresolved treatments. All 32 completed-candidate totals equal a fresh canonical evaluation (`asset-verification.json`, `evaluation-recheck.json`). The mismatch is in composition/search/selection, not export.

## Failure diagnosis

All four targets are playable on strings 1–6, frets 1–12 without open strings under the confirmed guitar-written convention. All of Me aligns at measure 1; Autumn Leaves at the first Gm7 in measure 2, retaining the preceding measure at negative form-relative time. Sources and references are unchanged.

Stateful constrained derivation reaches all four complete streams with valid harmonic contracts and zero diagnostic history cuts. This demonstrates coverage only. The ordinary first successor cutoff drops every whole-action-compatible target opening:

| Case | Proposed actions | Exact compatible openings | Retained |
|---|---:|---:|---:|
| All of Me acoustic | 9,853 | 1 | 0 |
| All of Me electric | 9,853 | 2 | 0 |
| Autumn Leaves acoustic | 8,492 | 2 | 0 |
| Autumn Leaves electric | 8,492 | 9 | 0 |

Evidence: `exact-opening-pruning.json`. An earlier clipped-prefix diagnostic counted a shortened Autumn acoustic hold as surviving; `opening-pruning.json` records that weaker historical observation. The observer now checks complete committed events, with a regression preventing this mistake. It never supplies candidates or changes scores.

Final ranking is also insufficient. Complete targets would lose under default weights even if retained:

| Case | Selected total | Complete target total |
|---|---:|---:|
| All of Me acoustic | −2787/64 | 945/64 |
| All of Me electric | −4551/128 | 1287/128 |
| Autumn Leaves acoustic | −1420/33 | 14/11 |
| Autumn Leaves electric | −420/11 | 239/22 |

Lower wins. `evaluation-diagnosis-all.json` records component differences. Source interval-pair resemblance, directed motion, register cost and duration preferences still favor continuous movement. The current features do not establish the longer ordered relationships needed to specify these authored tunes. This is not a proof that every generic model must fail.

The source-only four-bar diagnostic exports eight completed alternatives and identifies the automatic winner. Seven first-bar realizations differ, but all share the first four attacks. Their complete events, home identities, contracts, actions, frontier counts and scores are in `completed-phrases/candidates.json`. Eight retained alternatives do not by themselves demonstrate useful opening diversity.

## Development learning

The prefix perceptron is removed. The replacement forms exact rational inequalities between complete ordinary outputs and complete examples. It learns shared bounded integer coefficients per role. Allowed features are relative musical categories and bounded two/three-condition rules, not song names, hashes, absolute measure indices, event arrays or per-song recipes. New conjunctions require demonstrated feature collisions. Production does not import the solver or read examples.

The optional dependency is pinned to `z3-solver==4.13.3.0`. Exact zero/one-coefficient interval enumeration handles sparse cases; integer optimization handles the remainder. Feasibility and optimization have deterministic work limits. Incompatibility or exhaustion yields a diagnosis, not a substituted tune.

The first four-example round compared each target with eight saved complete alternatives. It found acoustic `duration/1 = 9` and electric `metric/eighth/1_2 = 15`. These sparse coefficients separate the sampled alternatives; **they are not a learned jazz grammar or evidence of reproduction**. They remain in `structured/round-1/data`, with explicit source/target/artifact hashes and training provenance. Nothing was promoted to production `data/`.

Successive rounds can accumulate full-output counterexamples using `--previous-round`. Source/example identities, evaluator, fixed priors and musical policies must agree; only learned weights may differ. Incompatible histories are rejected and duplicate inequalities removed. Later fitting cannot silently forget earlier failed complete outputs. No target-prefix update is used.

A fresh ordinary build of that proposal completed under `learned-stage`, using four workers and no target arguments. It passed technical checks but failed the exact gate in all four cases:

| Learned output | Generated / target attacks | Exact matching attacks | Exact result |
|---|---:|---:|---|
| All of Me acoustic | 214 / 137 | 0 | Fail |
| All of Me electric | 244 / 141 | 0 | Fail |
| Autumn Leaves acoustic | 315 / 121 | 0 | Fail |
| Autumn Leaves electric | 280 / 115 | 0 | Fail |

The [learned-model comparison](../../build/solo_reference_comparison/structured/learned-comparison/index.html) contains the complete exports, consistent auditions and exact differences. All 32 saved learned-candidate scores and four selected event/TAB streams passed independent rechecking. All eight complete non-solo role documents remained byte-canonically identical to the default stage despite the changed solo weights (`full-song-role-independence.json`). No inference about musical preference follows from that technical result.

The second fit accumulated **64 complete-output inequalities** from both ordinary builds. It found acoustic `space/after_arrival = -37` and electric `duration/1_4 = 19`. Both acoustic targets are still excluded at the first retrieval cutoff. The two electric openings survive that check, which proves neither full retention nor reproduction. This proposal was not staged or published, and no third full rollout is claimed. The result demonstrates how a sparse fit can separate sampled full outputs through coarse rhythmic statistics without recovering the ordered melody. Repeating the same fit or widening the beam is not a demonstrated remedy. Evidence is in `round-2/training.json`.

Learning explicitly depends on the authored examples. Runtime target-free is **not training independence**. These repeatedly studied examples are not held-out data. Transfer to separate standards has not been demonstrated here and is not substituted for exact reproduction.

## Validation

Commands use `build/build_common/python/bin/python3 build/build_tools/python_runner.py build/build_common/modulus`; [commands.md](commands.md) records arguments.

- `-m unittest discover -s tests -p 'test_*.py'`: **458 run: 456 passed, two optional solver skips, 1,139.453 seconds**. A preceding run had one `N.C.` lookahead error; it was repaired before the full rerun.
- With `PYTHONPATH=build/development_packages`, `-m unittest tests.test_joint_actions tests.test_structured_selection`: **35 passed in 24.256 seconds** (`successor-cache-tests.log`). This covers later exact shortcuts, solver changes, accumulated-round compatibility, strict diagnostic matching and incoming-state cache invalidation. The full suite was not rerun after every performance edit.
- `-m audition.dependency_check --output .../dependency-cached`: passed all six roles alone/cold, reversed/warm, two-process/cold, with invalid targets present/absent, and after an operational electric-only register change. All case hashes match the preceding `dependency-final` and `dependency` runs. Actual reads are recorded.
- Full `-m generator ... --check --composition-jobs 2 --export-solo-xml`: passed for the default stage. The separate learned-data run with `--composition-jobs 4` also passed. Actual read traces contain no reference or other part's concrete output.
- `-m audition.solo_compare ... --require-exact`: failed all four in **both** full builds, preventing a success claim.
- `structured/verify_completed.py`, through the Python wrapper: passed full selected-event, TAB, tie and canonical-score checks for both stages (64 completed candidates total).
- `git diff --check`: passed at the latest review.

The dependency exercise uses a synthetic one-bar source and real catalogs, not four complete-song cold/warm repetitions. [dependency_proof.md](dependency_proof.md) gives hashes and scope.

## Performance and cleanup

Hardware reports Intel Core i7-1355U, 12 logical CPUs, Linux 7.2.3 x86-64. The two-worker default stage took **4,832.443 seconds**: 4,817.592 in performance generation, 1.418 playback compilation and 12.655 engraving. Solo role times were 2,126.676 / 2,711.550 seconds for All of Me and 1,958.730 / 2,101.732 for Autumn Leaves. Other roles took about 0.2–0.5 seconds each. Tests/diagnostics ran concurrently; this is not an isolated benchmark.

This is a substantial regression from the prior one-action policy. Wider comparison is not a generation speedup. A 20-second live sample found retrieval, state advancement and frontier scoring dominant; engraving is not the bottleneck.

Pure-operation caching, incremental evaluation, short-circuit theme equality and direct retrieval costs remove repeated work. All eight four-bar candidates and every frontier were identical across 227.751-, 126.421- and 92.633-second runs with different concurrent loads; these do **not** establish a controlled whole-build speedup. A paired 9,853-action microbenchmark measured median 0.274 seconds with full feature reports versus 0.125 with direct costs, every cost equal. That claim is limited to warm-cache retrieval.

Successor-ranking memoization also preserves all eight candidates and every frontier. Its four-bar run took 154.177 seconds while the four learned workers ran, with 90 cache hits and 645 misses. The different load prohibits interpreting that elapsed time as a cache speedup or slowdown. The cache distinguishes catalog identity, the three incoming pitches, release time, sustained/rest context, pending obligations, weights and budget; complete home-aware evaluation still occurs per history. It removes repeated ranking work without changing search breadth.

The learned four-worker full build took **2,007.666 seconds**, including 1,994 seconds of performance generation and approximately 13 seconds of engraving. Solo role times were 1,708.084 / 1,920.536 seconds for All of Me and 1,989.394 / 1,945.428 for Autumn Leaves. Its music, weights, optimizations, worker count and concurrent load differ from the default run. The shorter wall time cannot be attributed to parallelism alone. Cold iteration remains expensive; no arbitrary performance target has been met or promised.

Default-build workers started before the final performance fixes. `frozen-inputs.json` and `post-freeze-repair.json` preserve that distinction. The no-chord guard does not apply to these sources; equality and export checks cover the shortcuts. The learned build has its own `learned-frozen-inputs.json` and predates only the final successor-cache optimization, recorded in `learned-post-freeze-repair.json`. Neither full run is presented as a cold benchmark of every final code byte.

Incomplete task-local stages and superseded caches totaling about 12.8 MB were removed; `cleanup.json` records paths. References, sources, baseline, completed comparisons and useful verified caches remain. Temporary work stays in the consolidated ignored build directory.

## Remaining boundary

The selector repair and learning mechanism are implemented. **The exact musical objective remains unmet.** No learned parameters or staged assets have been promoted. Independent parts, harmonic/physical safeguards and runtime representation remain intact.

The next bottleneck is specific: complete-phrase discrimination and early retrieval disagree, and the current event features permit degenerate rhythmic policies to beat the examples. More beam width, more unconstrained vocabulary, another prefix fit or hidden target retrieval would not establish the requested solution. Any next model change must face accumulated complete ordinary counterexamples and the four full-output target-free gate.

There is also a narrower theme-construction limitation: `related()` transposes or inverts the entire realized opening through strictly mapped pitches. It does not adapt the opening's expressive contracts. A legal chromatic/color-bearing home therefore need not have a legal generated return under this constructor. All four default outputs selected zero returns/answers. This is a construction/representation limitation, not evidence that the scorer has learned to avoid unmusical returns. Full operation counters and winner/runner-up contributions are in `structured/decisions.md`.

See [active_path_audit.md](active_path_audit.md), [data_migration.md](data_migration.md), [evaluation_contract.md](evaluation_contract.md), and [musical_capability_matrix.md](musical_capability_matrix.md).