2077-gps-mod/docs/compaction-handoff.md

# Compaction Handoff

Current date/time context: 2026-06-26, local timezone America/Chicago.

## 2026-06-26 Static Update

- Installed RED4ext state is safe: the game plugin directory contains
  `EdgeWeightGPS.dll.disabled`, not an active `EdgeWeightGPS.dll`.
- Latest committed safe source before this update: `55f4087 Document VAND GPS
  search graph`.
- Current source has `kEnableGpsTrafficEdgeWeightPatch = false` and
  `kEnableGpsLocalSearchHooks = false`.
- The real route search is confirmed at `0x44f054`, called from
  `0x44cc7c`. It is an A*/Dijkstra-style loop: validate packed handles, push
  start state, pop best open-list state, traverse adjacent handles, call
  provider filter slot `+0x08`, call provider edge-cost slot `+0x10`, update
  predecessor/cost, then materialize handles through `0x44eb68`.
- Both base and filtered cost-provider vtables point slot `+0x10` at
  `0x44f838`. The provider vtable RVAs are still `0x2ae6120` (base) and
  `0x2ae60f8` (filtered).
- `0x44f838` computes geometric edge distance and multiplies by
  `provider[0x08 + ((neighborPoint[0x13] & 0x3f) * 4)]`.
- Base provider constructor `0x451158` fills all 64 class multipliers with
  `1.0`. Filtered provider constructor `0x44b9b4` changes the vtable and
  `0x44bda4` copies route masks/special zones into the provider, but ordinary
  GPS route edge cost is still effectively geometric distance.
- `0x44f7bc` is the heuristic helper, not edge cost. It returns distance to the
  target scaled by the constant at `0x1431ef3a0` unless the special route-shape
  helper supplies an adjusted distance.
- `0x44eb68` only walks predecessor links and writes packed handles. The richer
  final route records are produced afterward by `0x44cc7c`/query result code.
- The packed handles resolve into `VAND` navigation blobs from
  `base\worlds\03_night_city\_compiled\default\navigation_*.streamingsector`,
  not directly into `all.traffic_persistent`. Final route records can still be
  joined to `worldTrafficLanePersistent.nodeRefHash`, but that is downstream of
  the search.
- Added `tools/analyze_vand_navigation.py` to decode VAND blobs from extracted
  streamingsector JSON. On the three extracted samples:
  `files=3 blobs=300 points=5059 coords=5046`, all masks were `0x0003`, and
  dominant classes were `1`, `4`, `5`, `3`.
- Practical implication: traffic `maxSpeed` and `all.lane_connections`
  probabilities are not the edge-cost surface. The next viable patch should
  either hook `0x44f838` and classify edges spatially against precomputed
  highway/arterial corridors, or patch the navigation VAND graph/classes after
  extracting the relevant streamingsector resources.

## 2026-06-26 Spatial Edge-Cost Prototype

- Added `tools/generate_spatial_edge_grid.py`, which combines
  `work/raw-segment-json/all.traffic_persistent.json` lane flags with
  `work/traffic-companions/json/all.lane_polygons.json` polygons.
- Generated
  `red4ext/EdgeWeightGPS/src/GeneratedSpatialRoadGrid.hpp`. The grid is
  332 x 361 cells at 32 m/cell, origin `(-4640, -7392)`, with final cell
  counts `H=7574`, `R=19617`, `G=20`, `P=996`, `unknown=91645`.
- Added disabled-by-default runtime plumbing behind
  `kEnableGpsSpatialEdgeWeightPatch = false`. When enabled, it attaches only
  `GPSRouteProducer 0x44cc7c` and `GPSEdgeCost 0x44f838` unless the old verbose
  local-search probe is also enabled.
- The patch reads search-state vectors as `(world X, height, world Y)`, samples
  the current endpoint, neighbor endpoint, and midpoint, chooses the strongest
  spatial road class, then applies a multiplier after native `0x44f838`
  returns. Current test multipliers are highway `0.62`, road `1.0`,
  GPS-only `1.20`, pavement `1.35`, unknown `1.05`.
- The patch intentionally does not rewrite VAND point class bits. Static
  materialization code branches on those bits later, so changing them in data is
  riskier than adjusting cost at the edge-cost vtable slot.
- Fixed `DetourGpsEdgeCost` logging: native `0x44f838` indexes the provider
  cost table by neighbor point class, not source point class.
- Build verification passed in the Fedora toolbox:
  `toolbox run -c 2077 tools/build_red4ext_shim.sh`.
- The resulting DLL remains uninstalled. Game install still has only
  `EdgeWeightGPS.dll.disabled`, not an active plugin DLL.
- Follow-up live-tuning support adds
  `red4ext/plugins/EdgeWeightGPS/spatial_weights.bin`: five little-endian
  float32 values in highway, road, GPSOnly, pavement, unknown order. The plugin
  polls this file on route-producer entry, so changing it and replotting a route
  applies new weights without rebuild/relaunch.
- `tools/write_spatial_weights.py` can write the active file or generate
  presets. Generated presets live in `work/spatial-weights/`:
  `vanilla.bin`, `current.bin`, `highway-free.bin`,
  `highway-expensive.bin`, and `surface-extreme.bin`.

Important caveat:

- Runtime logs show long visible GPS routes are decomposed into many
  `0x44cc7c` local searches. The `ret_rva=0x70aa22` caller with
  `query_f78=10` and `query_ptr98_rva=0x8d76c4` mostly produces tiny connector
  searches, not the whole route. Do not gate the spatial patch only on that
  caller unless a later test proves it covers the desired route segment.
- Live inversion test log:
  `logs/EdgeWeightGPS_spatial_inversion_no_route_change_1845.log`.
  The plugin reloaded highway-free weights at `+74681ms`
  (`H=0.05`, all others `19.0`), then highway-expensive weights at `+235135ms`
  (`H=19.0`, all others `0.05`). The four repeated map routes after the flip
  produced final `GPSQueryResultFetch 0x7094b8` route records that were exactly
  identical to the pre-flip route records:
  route 3 vs 7, 4 vs 8, 5 vs 9, and 6 vs 10 all had identical lane handles and
  packed metadata.
- Practical conclusion: the spatial `0x44f838` detour is active and changes
  many local route-producer edge costs, but it does not change the final
  selected world-map route records for the tested player GPS routes. Treat this
  hook as a local-search/connector lever, not the decisive full-route weighting
  surface.

## Current Installed State

- Installed RED4ext plugin is disabled:
  `Cyberpunk 2077/red4ext/plugins/EdgeWeightGPS/EdgeWeightGPS.dll.disabled`.
- The latest tested DLL is also preserved disabled as:
  `Cyberpunk 2077/red4ext/plugins/EdgeWeightGPS/EdgeWeightGPS.dd1423b.dll.disabled`.
- There is no active `EdgeWeightGPS.dll` in the plugin directory.
- Source defaults are currently read-only for route behavior again:
  `kEnableGpsSpatialEdgeWeightPatch = false`. Focused final-route result tracing
  remains available behind `kEnableGpsQueryResultTraceHooks = true`.
- Latest archived test log:
  `logs/EdgeWeightGPS_spatial_inversion_no_route_change_1845.log`.
- Game is cold as of the latest user message.

## New Toolbox Tools

The Fedora toolbox `2077` now has:

- `rizin` 0.7.4
- `python3-capstone` 5.0.5
- `python3-pefile` 2024.8.26

If adding Python-only tooling, create/use a venv and maintain `requirements.txt`
instead of one-off pip installs.

## Active Probe Configuration

In `red4ext/EdgeWeightGPS/src/Main.cpp`:

- `kEnableGpsProviderClassPatch = false`
- `kEnableGpsProviderClassPatchTrace = false`
- `kEnableGpsTraceHooks = true`
- `kEnableGpsQueryLifecycleHooks = false`
- `kEnableGpsRouteJobLifecycleHooks = false`
- `kEnableGpsRouteJobStartTrace = false`
- `kEnableGpsQueryResultTraceHooks = true`
- `kEnableGpsSpatialEdgeWeightPatch = false`
- `kEnableGpsLocalSearchHooks = false`
- `kEnableRouteUiTraceHooks = false`

If rebuilt with the current defaults, active hooks are focused map-route submit
and final route-result fetch tracing only. The spatial edge-cost patch requires
`kEnableGpsSpatialEdgeWeightPatch = true`. The full local search/cost hooks below require
`kEnableGpsLocalSearchHooks = true`:

- query lifecycle:
  - `GPSQuerySubmit 0x70a42c`
  - `GPSQueryDispatch 0x70a570`
  - `GPSQueryResultFetch 0x7094b8`
  - `GPSQueryStatus 0xaa5704`
- `GPSSearch 0x44f054`
- `GPSRouteProducer 0x44cc7c`
- `GPSResolveHandle 0x44e1a8`
- `GPSEdgeCost 0x44f838`
- filtered provider filter `0x44ff68`
- base provider filter `0x450b08`

Async route-job hooks (`0x8d17d8`, `0x126b28`, `0x883cd8`,
`0x11f5f60`, `0x818ba8`) are split behind
`kEnableGpsRouteJobLifecycleHooks` and currently disabled to keep the next
run focused on solved route result records plus local search/cost stats.

The filter probe records pass/fail totals by GPS point class and by bits in
`point + 0x10`, tested against provider masks at `+0x108` and `+0x10a`.

## Why This Probe Exists

Data and behavior patches tested so far:

- Traffic `maxSpeed` patches affected NPC freeway speed but not GPS route
  choice.
- Lane connection probability patches did not change GPS route choice.
- Global cost-table patch produced weird routing but mostly confirmed we were
  near the solver, not a clean highway preference knob.
- Provider class override patches fired correctly but produced no perceptible
  route change after preserving vanilla class `14`.

Static disassembly clarified the provider split:

- Provider vtable slot `+0x08` is an admissibility filter, not cost.
- Provider vtable slot `+0x10` is geometric edge cost, multiplying distance by
  `provider[0x08 + class * 4]`.
- We patched only slot `+0x10` inputs. If slot `+0x08` filtering dominates the
  candidate graph, those cost changes can fire and still not visibly steer GPS.

## Important Native Findings

GPS pipeline landmarks:

- shared query submit: `0x70a42c`
- query dispatch: `0x70a570`
- route producer: `0x44cc7c`
- local search loop: `0x44f054`
- packed-handle resolver: `0x44e1a8`
- edge cost: `0x44f838`
- query result fetch: `0x7094b8`

The search loop is A*/Dijkstra-like:

- initialize/open list with start node
- pop best node
- check destination handle
- traverse adjacent lane handles
- call provider filter
- call provider cost
- update candidate cost/best predecessor

Provider details:

- Base provider vtable RVA: `0x2ae6120`
- Filtered provider vtable RVA: `0x2ae60f8`
- Base provider constructor `0x451158` fills 64 class multipliers with `1.0`
  and sets mask `+0x108` to `0xffff`.
- Filtered provider constructor `0x44b9b4` changes vtable to `0x2ae60f8`.
- Filtered provider commonly has class `14 = 0.05`; preserve this unless
  deliberately testing it.

Route result records:

- `0x7094b8` output records are 0x28-byte route segment records.
- `h00` in those records matches `worldTrafficLanePersistent.nodeRefHash`.
- Route records can be joined to `work/raw-segment-json/all.traffic_persistent.json`.
- The low byte of route-record `u08` is not the same as the search point class.
- The log's human wall timestamp has a millisecond rollover bug around second
  boundaries. Use the `+Nms` elapsed field for ordering and durations.

Packed-handle resolver:

- `0x44e1a8` decodes the packed route handle used by the search/materialization
  side:
  - low 20 bits: point index
  - bits 20..47: segment index
  - high bits: generation/check value
- It resolves the handle into a 0x68-byte segment record and a 0x14-byte point
  record. The point record's `+0x13 & 0x3f` is the search-side class consumed by
  `0x44f838`.
- Live logging corrected the selected post-call return RVAs to:
  - `0x44cb9e`: 0x38 intermediate-record builder. It writes packed handle,
    `point+0x10` mask, `point+0x13` class, and coordinates from
    `segment+0x18`.
  - `0x44dd97`: geometry/materialization path. It walks the `segment+0x28`,
    `segment+0x30`, and `segment+0x38` arrays for the resolved point.
- The current probe opens a five-second resolver trace window before the
  original `0x70a42c` submitter runs when the caller is `0x8d20d4` and
  `query_fcc=5`. Inside that window it logs selected resolver callsites with
  per-return-site caps instead of spending the whole budget on startup noise.
- Each resolver line includes the decoded handle, segment/point pointers,
  segment pointer fields `+0x08` through `+0x38`, raw segment bytes, raw point
  bytes, raw `segment+0x08` metadata, the `segment+0x28` point-index record,
  and small raw dumps from `segment+0x20`, `+0x30`, and `+0x38`. The goal is to
  determine whether the selected-path segment still carries, or points to, the
  traffic lane hash/resource flags before the final 0x28 route records are
  emitted.

## Latest 23:08 Test Result

The user clicked:

- 23:08 continue on main menu
- 23:30 space to continue
- 23:46 map open
- 23:56 side job
- 24:06 Sinnerman
- 24:14 Claire
- 24:21 custom
- 24:30 Aldecaldo camp
- 24:47 far gig east
- 25:03 Violence
- 25:10 cyberpsycho unfinished highway

Use:

```bash
python3 tools/summarize_gps_route_records.py \
  --traffic-json work/raw-segment-json/all.traffic_persistent.json \
  logs/EdgeWeightGPS_query_route_probe_2308.log
```

Important result table:

- qid 1: startup/minimap route, 18 records, 50 ms, 1216.5 pavement length.
- qid 2: side job, 12 records, 32 ms, 578.6 pavement length.
- qid 3: Sinnerman, 23 records, 45 ms, 897.4 pavement, 160.4 highway,
  89.8 road.
- qid 4: Claire, 18 records, 34 ms, 1216.5 pavement.
- qid 5: custom, 17 records, 44 ms, 1006.3 pavement, 241.0 highway,
  38.0 road.
- qid 6: Aldecaldo camp, 80 records, 325 ms, 5853.5 road, 1492.9 highway,
  1111.5 pavement, 24.6 gpsonly.
- qid 7: far east gig, 81 records, 144 ms, 5086.2 road, 1523.0 highway,
  1111.5 pavement, 24.6 gpsonly.
- qid 8: Violence, 55 records, 97 ms, 1991.5 pavement, 358.5 highway,
  86.1 road.
- qid 9: cyberpsycho unfinished highway, 58 records, 147 ms, 2657.4 pavement,
  519.0 highway, 113.2 road, 56.7 gpsonly.

All 362 route-record handles matched traffic lane hashes in the extracted
resource.

Local search/provider aggregate from this run:

- 159 searches, 4036 provider filter calls, only 3 filter failures.
- Filter pass classes: `3:3114`, `5:451`, `15:310`, `1:116`, `4:42`.
- Edge source classes: `3:3244`, `5:487`, `15:318`, `1:117`, `4:58`.
- Edge return RVAs: `0x44f457:4033`, `0x44f629:151`,
  `0x8d74d3:30`, `0x8d76b0:10`.

Inference:

- Provider vtable slot `+0x08` filtering is almost not pruning these route
  searches, so it is not the dominant reason class-cost patches failed to
  visibly steer GPS.
- Search-side edge classes are only `{1,3,4,5,15}` in this run. Final emitted
  route-record classes include `{0,1,2,3,4,5,7,8,9}` and class 7/8/9 are
  highway-heavy. Therefore final route-record `u08` low byte and search point
  `point+0x13` class are different domains.

## Latest 46:50 Resolver Test Result

Archived log:
`logs/EdgeWeightGPS_resolve_handle_probe_4650.log`.

User timing:

- 46:50 Continue on main menu
- 47:23 Space to continue
- 47:35 map open
- 47:43 side job
- 47:51 Sinnerman
- 47:57 Claire
- 48:09 custom
- 48:2x Aldecaldo camp
- 48:57 game cold / `OnExit`

Route summary:

- routes=6, records=170, matched route records to lane hashes=170/170.
- qid 1: startup/minimap, 18 records, 43 ms, 1216.5 pavement.
- qid 2: side job, 12 records, 34 ms, 578.6 pavement.
- qid 3: Sinnerman, 23 records, 45 ms, 897.4 pavement, 160.4 highway,
  89.8 road.
- qid 4: Claire, 18 records, 35 ms, 1216.5 pavement.
- qid 5: custom, 19 records, 45 ms, 1006.3 pavement, 343.4 highway,
  38.0 road.
- qid 6: Aldecaldo, 80 records, 340 ms, 5853.5 road, 1492.9 highway,
  1111.5 pavement, 24.6 gpsonly.

Resolver sample findings:

- The first resolver probe captured only the startup guard window because the
  guessed return RVAs were post-call-off by a few bytes.
- It still proved `GPSResolveHandle 0x44e1a8` attached successfully.
- Actual resolver returns in the first 64 logged samples:
  `0x44dd97:57`, `0x44cb9e:7`.
- Resolver point classes in those samples: `4:60`, `5:4`.
- The 0x68 segment header contains generation/counts/pointers; final
  `nodeRefHash` values are not inline in those 0x68 bytes.
- Static disassembly shows `segment+0x10` is the 20-byte point array,
  `segment+0x18` is the coordinate array, and the materializer indexes
  `segment+0x28` by `pointIndex*12`.

## Next Work

- Pivot away from the spatial `0x44f838` cost detour for world-map routing.
  It is proven active but does not alter final selected route records under
  extreme inverse weights.
- Statically disassemble the caller chain around the full map-route submit and
  result path: `0x8d20d4`, `0x70a42c`, `0x70a570`, `0x7094b8`, `0x520783`,
  and the async route-service cluster around `0x8d17d8` through `0x8d49xx`.
- Identify where the final 0x28 route records are selected or copied before
  `0x7094b8` returns them. The current evidence says that is closer to the real
  full-route weighting/selection surface than the local `0x44cc7c` searches.
- Avoid post-`0x7094b8` rewriting unless every upstream option fails; that would
  likely only change displayed path records and risks desynchronizing route
  instructions from the actual planner.

## Useful Commands

Build/install:

```bash
toolbox run -c 2077 ./tools/build_red4ext_shim.sh
toolbox run -c 2077 ./tools/install_red4ext_shim.sh
```

Clear plugin log:

```bash
truncate -s 0 '/var/home/salt/.var/app/com.valvesoftware.Steam/.local/share/Steam/steamapps/common/Cyberpunk 2077/red4ext/plugins/EdgeWeightGPS/EdgeWeightGPS.log'
```

Inspect current plugin log:

```bash
tail -n 240 '/var/home/salt/.var/app/com.valvesoftware.Steam/.local/share/Steam/steamapps/common/Cyberpunk 2077/red4ext/plugins/EdgeWeightGPS/EdgeWeightGPS.log'
```

Disassemble with rizin in toolbox:

```bash
toolbox run -c 2077 rizin -q -A '/var/home/salt/.var/app/com.valvesoftware.Steam/.local/share/Steam/steamapps/common/Cyberpunk 2077/bin/x64/Cyberpunk2077.exe'
```

## Caution

- Do not revert user changes or old untracked logs unless explicitly asked.
- Commit early/often.
- Do not install Python packages one by one with pip; use `requirements.txt`.