Game Features

docs/ingestion.md

@@ -0,0 +1,88 @@
+# Election ingestion
+
+`fpol_ingest` is a separate C++ process using Ligo's raw HTTPS client and Noto/SQLite. It supports conditional ETag/Last-Modified requests, a 25-second network deadline, a 16 MiB response limit, exponential failure backoff up to one hour, immutable changed-payload archives, immutable observations, quarantine, and current projections. Enabled count/outcome feeds are polled approximately every 60 seconds; candidate watches daily. Responses request identity encoding. No browser scrapes providers directly.
+
+## Honest readiness
+
+As of the implementation session, **no live provider is verified**. The environment cannot open sockets or resolve outbound DNS. Open America's published schema informed a tested adapter; synthetic replay tests are not transport validation. civicAPI has only a capture-only probe: its race/candidate contract still needs a real successful response before a normalizer can be written. The `official-json` adapter is a normalized integration contract, not a claim that arbitrary state websites already emit it. Additional state-specific CSV/XML/JSON adapters and reviewed PDF imports are still required for national coverage.
+
+Open America documents counts from 22 states, no pre-election candidate catalog, and no race calls. Its `final`, leader, precinct percentage, and candidate `w` fields never award a winner bonus. Documentation: [Open America election API](https://openamerica.io/elections/api/). Attribution in the UI should retain the feed URL and timestamps; the source is official state election offices via Open America.
+
+## Configure, capture, review, activate
+
+```sh
+fpol_admin --database var/fpol.sqlite import data/real/us-2026-10-04-partial.json
+fpol_ingest --database var/fpol.sqlite configure data/sources/openamerica-nc-house.json
+fpol_ingest --database var/fpol.sqlite probe openamerica-nc-house
+fpol_ingest --database var/fpol.sqlite status
+# Inspect the archived raw response and exact candidate/round mappings first.
+fpol_ingest --database var/fpol.sqlite activate openamerica-nc-house
+fpol_ingest --database var/fpol.sqlite run
+```
+
+Configuration starts disabled and unverified. The sample NC mappings use official ballot spellings as **proposals**; they have not been matched to an actual provider capture. Update exact aliases after review. A configuration change disables the source and clears its per-race verification checks. Activation requires a validated live observation for **every configured race/role**, not merely an HTTP 200 or one race in the response. Before election night, an empty results payload cannot satisfy this requirement; real draft readiness may need a provider that offers verifiable race coverage before counts arrive.
+
+`probe` fetches even a disabled source. `once` polls only enabled sources whose next check is due. `disable ID` stops scheduled polling. `activate ID` selects this provider for each configured role/race, replacing that assignment; do this deliberately, with reviewed data. Existing selected counts are never added to another provider's counts.
+
+Raw changed responses are stored in `source_payloads` keyed by source plus content hash. Use the local SQLite client or a backup to inspect/export them; they are not served through player APIs. `source_checks` records only successful live validations. `observations` retains normalized records, `current_observations` selects the current record, `coverage` assigns one count source and one outcome source per contest, and `source_quarantine` records rejected responses and their archive ID.
+
+## Adapter configuration
+
+A configuration has `id`, `kind`, HTTPS `url`, `catalog_id`, and `races`. Each assignment has an internal `race_id` and `role` (`counts` or `outcome`). Standard HTTPS hostnames on port 443 are supported; redirects are reported as HTTP errors for operator review.
+
+For `openamerica`, also supply `office` (`House`, `Senate`, `Governor`), `counts_until` in UTC, and each assignment's `provider_id` plus a `candidates` object mapping exact `name|party-code` to stable candidacy IDs. Explicit `null` mappings can exclude an aggregate write-in row; unknown rows quarantine the response. Separate fusion ballot lines can map to the same candidacy and their votes are summed once. Duplicate provider rows, wrong-race IDs, negative counts and impossible totals are rejected. Do not map different people together or assign aggregate write-ins a roster slot.
+
+Open America does not expose a round identifier. The configured cutoff prevents a later election from silently overwriting November counts. Review each state's feed semantics, including ranked-choice reporting; do not activate an adapter if it cannot supply first-choice counts. After the cutoff, reviewed first-round corrections need an official-source adapter with explicit round metadata. Sample NC cutoffs are conservative operational bounds, not a national runoff calendar.
+
+The `official-json` endpoint returns:
+
+```json
+{
+  "schema_version": 1,
+  "observations": [
+    {
+      "race_id": "2026-nc-senate-general",
+      "kind": "counts",
+      "round": "november_first",
+      "source_time": "2026-11-04T03:00:00Z",
+      "total_votes": 1000,
+      "votes": [
+        { "candidate_id": "2026-nc-senate-general-roy-cooper", "votes": 480 }
+      ]
+    },
+    {
+      "race_id": "2026-nc-senate-general",
+      "kind": "outcome",
+      "round": "general",
+      "source_time": "2026-11-04T04:00:00Z",
+      "status": "reporting",
+      "winner_id": null
+    }
+  ]
+}
+```
+
+**The numbers above are schema examples, not election results.** `source_time` may be null when unknown; use UTC `Z` or `+00:00` otherwise. Source timestamps and receipt/check times remain separate. Counts may contain zero values, but zero total or a missing candidate produces an unknown score. Candidate totals cannot exceed the reported race total; the total may include explicitly excluded write-ins. Only `november_first` counts are accepted. A later runoff/RCV outcome may use its own round identifier without altering those counts.
+
+Outcome status is `reporting`, `pending_round`, `called`, or `certified`. Only called/certified outcomes have a `winner_id`. A newer reporting/pending record retracts a provisional call. A certified outcome takes precedence over subsequent provisional records; a newer certified correction may change the winner. Older source timestamps are archived without moving the current projection backward. Missing races do not erase previous data. Failed or ambiguous responses preserve the last good observation and mark source health stale.
+
+## Candidates and reviews
+
+```sh
+fpol_ingest configure data/sources/nc-candidate-watch.json
+fpol_ingest watch nc-candidate-watch
+fpol_ingest configure data/sources/civicapi-status.json
+fpol_ingest probe civicapi-status
+```
+
+`candidate-watch` captures changed public source files daily for human review; it does not blindly parse PDF names or silently change a frozen draft pool. Import a new validated catalog ID after reviewing changes. Preserve stable candidacy IDs across spelling/party-line updates; merge fusion lines; exclude aggregate write-ins. Individually qualified write-in candidates need their own reviewed identity and eligibility evidence. Old catalog documents and draft snapshots remain immutable.
+
+[The inventory](../data/contests_2026.json) covers the 50 states' scheduled November House/Senate/governor contests, including safe and uncontested seats, with FL/OH special Senate contests distinct. Round data stays separate from contest identity. House at-large seats use internal district `01`; map providers' `00` explicitly. The inventory is a scope checklist, not evidence that candidate or result coverage is complete.
+
+## Replay without overstating verification
+
+```sh
+fpol_ingest replay SOURCE_ID captured-or-fixture.json
+```
+
+Replay archives and validates the file but never grants live verification. It can publish observations only into an explicitly fictional `mode: demo` catalog; real catalogs require network observations. Demo replays can assign empty demo coverage slots so practice standings can be exercised. Replaying an old file never unlocks real drafting. The first milestone's immutable `results` snapshots and `score-preview` remain a separate v1 lab contract; the league scorer uses `v2-november-share`.