Development troubleshooting and lessons

Source-code links on this page require an invitation to the private GameQ repository.

On narrow screens, scroll sideways inside wide diagrams to see the full flow.

GameQ reached a working handheld by separating faults in the emulator, external memory, LCD transport and Python session control. The same “green square” could have come from several layers; an FPS counter alone did not identify which one. This page records the evidence that narrowed those possibilities and the contracts that now prevent silent failure.

For ordinary installation and play, use getting started and playing. For component boundaries and diagrams, use architecture. Exact physical results and artifact hashes belong in the hardware test record.

Current baseline and evidence limits

0.1.7GQ is the accepted gameplay baseline on main. The operator confirmed walking, CAPS speed toggling and save/reload. Its build passed 274 tests, with additional host replays and compiled ARM checks described in the hardware record. Full power-off save loading was explicitly confirmed on 0.1.6GQ.

No returned 0.1.7GQ SD diagnostics or measured physical FPS have been collected. CAPS LED indication, no-SD room startup, complete redraws after dialogs, recovery Cancel and failed-save power handling still need explicit hardware checks. A passing host test and an operator report answer different questions; neither should be relabeled as the other.

Diagnose the layer before changing it

Wide diagrams scroll sideways on smaller screens.

%%{init: {"flowchart": {"useMaxWidth": false}}}%%
flowchart TD
    Start[Preserve files and exact symptom] --> Load{Load failed?}
    Load -->|Yes| ROM[Read rom-check and last-error]
    Load -->|No| Fault{Native fault recorded?}
    Fault -->|Yes| CPU[Inspect registers and RAM]
    Fault -->|No| Room{Unexpected room transition?}
    Room -->|Yes| Trace[Compare mailbox and raw keys]
    Room -->|No| Capture{Source image correct?}
    Capture -->|No| Core[Inspect input and game state]
    Capture -->|Yes| LCD[Inspect LCD transfer and validity]

Start by preserving the card’s saves and GameQ/system files before another load overwrites recent diagnostics. Record installed version, exact error, power source, slot, key pressed and whether a reboot occurred. Use Capture a failure for the collection procedure. Do not infer a fresh failure from a retained file alone.

Which record answers which question?

Record	Useful evidence	Interpretation limit
rom-check.txt	Current transfer, source/PSRAM mismatch kind, complete passes, retries and timing registers	A success may have used retries; inspect the counters
diagnostic.txt	Cartridge/version, early CPU samples, mode, frame/native timing, cache hashes and LCD traffic	Timing is a sampled window, not necessarily a whole-session average
last-frame.pgm	160×144 source image before amber conversion, scaling and SPI	It does not show what the physical panel received
bank0.bin	SRAM copy of ROM bank zero	One bank does not establish every bank’s correctness
wram.bin, vram.bin, hram-io.bin	Stopped emulated memory after a captured fault	Old dumps can remain; check whether the current diagnostic identifies them
hub-trace.txt	Last 16 room boundaries: mailbox, raw key matrix, frame pad and returned result	Earlier versions did not capture every initiating input
last-error.txt	Last exception retained across clean room interactions	It deliberately survives later success
.sav, .sav.bak, .sav.tmp, .crash.sav	Save generations and fault-time RAM	File existence or expected length alone is not an in-game save checksum

1. The green square was a symptom, not a diagnosis

Early builds sometimes showed Red’s opening and then a solid or corrupted green square. Cancel still returned a timing report. That established that the shell could respond, but not that the emulated game was progressing correctly.

In 0.1.2GQ, the captured state at frame 463 was already in a repeated RST38 loop. On canonical Red, the opcode at 0x0038 restarts to the same address, pushing return addresses until it overwrites RAM. A late dump therefore showed consequences of the failure, not the instruction that first led there.

Change: a first-fault trap stops at the first opcode fetch from that handler and records registers, stack context, selected banks, input and in-place RAM. The shell arms it only when ROM size and bank-zero fingerprint match canonical Red. Other cartridges may legitimately use RST38 and retain normal behavior.

What this established: later faults can preserve useful pre-loop evidence. It did not prove that every earlier green screen had one common cause. LCD captures, PSRAM comparisons and CPU state remain separate observations.

Sources: first-fault capture and ROM callback, trap selection and diagnostic output. Regression: core trap/state tests and boot trap/dump tests.

A separate reproducible sprite bug

The pinned Peanut-GB optional high-accuracy sprite sorter wrote past its 10-entry array during Red’s opening; AddressSanitizer reproduced it on the host. GameQ disables that optional sorter and runs the normal rendering path. This is an identified software defect, separate from the unresolved electrical cause of intermittent PSRAM corruption. The current opening replay and sanitizer test checks the selected configuration.

2. PSRAM needed an explicit access contract

Hardware produced intermittent mismatches, first at 0x017a8f, later at 0x0a1029 (b1 != e7) and 0x0a58a9 (a8 != b7). Retrying could allow a load to proceed. One successful readback was therefore insufficient evidence for unrestricted byte-wise ROM access during play.

The current design separates four contracts:

Boundary	Required condition	Response to failure
Source to PSRAM	Complete bank, aligned word stores and immediate readback	Stop the transfer
First source pass to second pass	Same complete-bank fingerprint	Report source drift before PSRAM reads
PSRAM verification	One entire scan matches, within three attempts	Reject the bank; never combine good fragments
PSRAM to SRAM cache	Complete copied bank matches its admitted fingerprint	Retry up to three copies, then stop the core

The verifier now fetches each external 32-bit word once and compares all four bytes locally. The earlier byte helper could reread the same external word four times while discarding three bytes per transaction. Verification remains a read-only check; retries do not rewrite memory to force success.

ROM execution uses bank zero plus ten switchable banks in internal SRAM. Verified aligned copies happen only on cache misses. A 15,000-frame host replay reduced bank copies from 52,352 to 49 with ten slots, while sampled CPU/RAM and images matched. That is a copy-count result, not a Q frame-rate measurement.

Timing changes and the unresolved cause

The runtime selects 30 MHz PSRAM and eight chip-select-high clocks, and pauses 4 microseconds after each 64-byte read burst. The helper drains mapped writes, waits for abort completion and idle state, checks register readback, and restores the interrupt mask. Unsupported modes and timeouts fail closed.

These choices follow documented memory/controller timing requirements; see external-memory timing. Compiled ARM tests model controller states, guards and timeout paths. They do not model signal integrity, refresh behavior or physical SPI waveforms.

Hardware result: the returned 0.1.6GQ card recorded 64 banks, 64 verification passes, no load retries and no cache retry/fault in its retained game session. The user played and loaded a save after a full reboot. Remaining limit: several access, cache and timing changes accumulated during bring-up. This successful run does not isolate which change resolved the original corruption. Do not describe “bad PSRAM,” “the clock was the cause,” or retry removal as established conclusions.

Sources: word access, bank verifier, PSRAM timing. Regression: source/moving-fault tests, cache tests, ARM controller and ARM verifier.

3. Diagnostics could mislead by being stale

After 0.1.5GQ load failures, rom-check.txt still contained an older 0.1.4GQ success. The exception and room trace recorded the new failure. The loader had written its verification report only on success.

Fix: _copy_rom_to_psram() records the transfer result from finally, before reloading the room resets native counters. The report includes completed pass bytes, failure phase, mismatch kind and retries. A diagnostic-write failure cannot replace the original loading exception.

By contrast, last-error.txt intentionally retains the last exception across success. The accepted 0.1.6GQ card still contained the old 0.1.5GQ error, byte for byte. Treat records as a set with version/session context, not a collection where every filename describes the newest boot.

Source: transfer recording. Regression: test_failed_psram_check_replaces_stale_success_and_records_native_evidence and test_diagnostic_write_failure_preserves_original_transfer_exception in boot tests.

4. Room interactions cross a real emulator boundary

The initial room report said shelves, the clerk and the cabinet opened a blank text box, then GameQ desk. It was tempting to treat every transition as one mailbox bug. Subsequent traces narrowed the evidence:

• The 0.1.4GQ card contained one cartridge and two hidden .gb metadata files. The old catalog counted those files as games. This was a concrete catalog bug.
• On 0.1.5GQ, two shelf-zero requests reached cartridge loading and then failed PSRAM verification. All 12 captured mailboxes matched their async returns. Those failures were not lost shelf commands.
• Earlier command-2 records requested settings, but their initiating keys were not captured. The cause of that earlier desk behavior remains unknown.

Changes and contracts: filter dotfiles, directories and the old SD hub; snapshot sorted shelf assignments until rescan; retain the exact selected filename; capture mailbox and raw keys before display cleanup; drain held native-dialog keys before resuming the room. Cancel in the room intentionally opens GameQ desk, so a carried-over Cancel must not be mistaken for a new action.

The 16-byte mailbox has a fixed reserved WRAM location and publishes its command last. The bridge accepts only bounded acknowledgements and startup coordinates. The ROM also stores a shelf argument before the far call because Red’s bank-switch helper overwrites register A. Tests exercise all shelf indices and valid/invalid return positions using the actual room ROM.

The restored state is x/y/facing, not a suspended cartridge or whole emulator snapshot. The hub never loads a Pokémon save. See hub.md for the ABI, source dispatch, library regressions, actual-ROM tests and mailbox tests.

5. Display size, color and speed are different problems

The early display used a small 160×144 green image. Scaling changed gameplay to 266×240 with preserved proportions, and the room to the full 320×240 panel. Four amber shades match the Q’s visual theme. Pixel palette flags must be masked before looking up a shade; the flags identify source palettes, not extra colors.

Larger output increases wire traffic. A full gameplay redraw sends 127,680 pixel bytes, a theoretical 17.02 ms at requested 60 MHz before CPU work and commands. A native frame time or emulated FPS value therefore cannot describe display smoothness by itself. Normal mode adapts display cadence; CAPS runs emulation without pacing waits. Both still process input and save state every frame.

0.1.7GQ optimization: compare normalized rows against the existing capture and send only rows whose known LCD contents differ. The crucial added state is row validity. A row that failed midway through transfer must not be skipped just because its capture buffer already contains the new pixels. Clearing the panel, switching width or unwinding an SPI exception invalidates the rows. Command failures now propagate, release chip select and force later redraws.

Host comparisons matched 54,000 frames of panel reconstruction and game state. At one render per four frames, walking traffic fell 40.93% in the new-game replay. Seventeen final-linked ARM display-error cases checked failure propagation, chip-select release and complete retry redraws. These establish behavior and traffic reduction, not physical timing. The operator reports 0.1.7GQ works well; no new measured Q FPS is claimed.

Sources: renderer, binding cleanup, pacing and SPI ownership. Regression: mapping, dirty-row and transfer-failure tests and pacing/display cleanup tests.

6. Saving needs both game-level proof and clear RAM ownership

Early runs produced correctly sized SRAM files without proving that Pokémon had completed a valid in-game save. The 0.1.3GQ copied save was invalid, and reload had not been tested. A file named .sav was not sufficient acceptance.

The returned 0.1.6GQ card provided stronger evidence: two distinct 32 KiB save generations passed Red’s main checksum and CONTINUE name-presence checks. The operator loaded the save both by relaunching and after a full power-off/reboot. 0.1.7GQ separately passed the operator’s save/reload check.

Current save handling has several independent guarantees:

• Ownership: Python owns the RAM bytearray and the native binding roots it for the active session. Room/game handoffs release old references before new allocations. A failed-power flag belongs to its SaveKeeper, not a global object that could retain or affect a later game.
• Persistence: quiet-time flushing and the periodic fallback track dirty RAM. Temporary writes are synced before promotion, and .bak preserves the previous generation across FAT rename failures. A read error cannot silently turn into a new game.
• Fault containment: emulator faults write .crash.sav; the regular save remains intact because the game may have failed partway through its own SAVE.
• Power failure UI: a failed power-hold flush pauses the game. The game coroutine owns retry, return and separately confirmed discard. The keyboard callback never draws over the emulator or shuts down after that failed flush.

These guarantees do not make arbitrary card removal or FAT power loss safe. The failed-save power dialog remains a physical acceptance item. Its Cancel returns to the same live game; the separate failed-save dialog when leaving a game uses Cancel to discard and return to the room. Do not merge those two control contracts in tests or player instructions.

Sources: save and power flow, library return handling, native save attachment. Regression: save generations, ownership and power tests and cartridge handoff tests.

Working on the next change

1. Start from the accepted baseline and preserve its artifact/save evidence.
2. Name the layer and the observable contract the change affects.
3. Reproduce it with source, diagnostic context and the smallest useful harness.
4. Compare complete outputs or state where possible, rather than asserting only that execution completed. Keep emulation, transport and physical timing claims distinct.
5. Run the relevant tests, then obtain hardware evidence for behavior a host or ARM model cannot establish. Record unresolved checks explicitly.

Use build.md for the reproducible toolchain and make test for the host suite. Some actual-ROM tests require the built artifacts; make firmware rebuilds the resident room and runs the suite in the container snapshot. hardware-test.md defines the device checks.

The next native-game work starts from the DOOM feasibility report. A real host level and standalone ARM link exist. Mutable PSRAM qualification and a combined firmware link remain gates; neither a host screenshot nor a standalone engine size establishes that DOOM runs on the Q.