diff --git a/docs/compaction-handoff.md b/docs/compaction-handoff.md index 40fe245..51c270d 100644 --- a/docs/compaction-handoff.md +++ b/docs/compaction-handoff.md @@ -6,8 +6,8 @@ Current date/time context: 2026-06-26, local timezone America/Chicago. - Installed RED4ext state is safe: the game plugin directory contains `EdgeWeightGPS.dll.disabled`, not an active `EdgeWeightGPS.dll`. -- Latest committed safe source before this update: `55f4087 Document VAND GPS - search graph`. +- Latest committed source before this handoff update: + `7c2ef33 Hook full GPS solver node weights`. - Current source has `kEnableGpsTrafficEdgeWeightPatch = false` and `kEnableGpsLocalSearchHooks = false`. - The real route search is confirmed at `0x44f054`, called from @@ -105,18 +105,74 @@ Important caveat: ## Current Installed State -- Installed RED4ext plugin is disabled: - `Cyberpunk 2077/red4ext/plugins/EdgeWeightGPS/EdgeWeightGPS.dll.disabled`. -- The latest tested DLL is also preserved disabled as: - `Cyberpunk 2077/red4ext/plugins/EdgeWeightGPS/EdgeWeightGPS.dd1423b.dll.disabled`. -- There is no active `EdgeWeightGPS.dll` in the plugin directory. -- Source defaults are currently read-only for route behavior again: - `kEnableGpsSpatialEdgeWeightPatch = false`. Focused final-route result tracing - remains available behind `kEnableGpsQueryResultTraceHooks = true`. +- Active RED4ext test plugin is installed: + `Cyberpunk 2077/red4ext/plugins/EdgeWeightGPS/EdgeWeightGPS.dll`. +- Older DLLs are preserved disabled as: + `Cyberpunk 2077/red4ext/plugins/EdgeWeightGPS/EdgeWeightGPS.dll.disabled` + and `EdgeWeightGPS.dd1423b.dll.disabled`. +- Source defaults now enable the full async solver multiplier hook: + `kEnableGpsMultiplierHooks = true`. +- The current behavior patch is deliberately narrow and diagnostic: + `DetourGpsNodeMultiplier` hooks native `0x40bb40`, applies only when + `job+0xc4 == 2` (driving route mode), reads lane/node flags from + `node+0x88`, and multiplies vanilla node edge cost by highway `0.35`, + GPSOnly `1.10`, pavement `1.25`, and all other flags `1.0`. +- `0x40bb00` auxiliary/reverse multiplier is hooked only for signature logging; + it is not patched yet. +- The plugin log was truncated immediately before installing this build. - Latest archived test log: `logs/EdgeWeightGPS_spatial_inversion_no_route_change_1845.log`. - Game is cold as of the latest user message. +## 2026-06-26 Full Async Solver Findings + +- Static route pipeline: + `0x8d20d4` map route submit -> `0x70a42c` query manager -> + `0x70a570` dispatch -> `0x8d17d8`/`0x8d1880` async enqueue. +- Worker loop around `0x126c06` consumes queued route requests from + `routeSystem+0x47c0`. +- Worker calls either `0x818928` directly or calls `0x883cd8` first, then + `0x818928`. Successful results pass through `0x11f5f60`. +- `0x883cd8` resolves/projects start and end candidates with helpers including + `0x198fb8`, `0x886958`, and `0x197b04`, then calls `0x1caa7a0`. +- `0x1caa7a0` initializes the full async traffic solver: it sets + `solver+0x10 = 1`, stores route mode at `solver+0xc4`, sets route masks at + `solver+0xc0/+0xc2`, copies candidate arrays into `solver+0x70/+0x80`, and + calls `0x88367c`. +- `0x88367c` marks start candidate lane/node records with `node+0x9a |= 1`, + marks end candidates with `node+0x9a |= 2`, computes initial costs at + node-state `+0x10/+0x14`, and queues nodes into open list `solver+0xd0`. +- `0x818928` calls `0x8189f0`; when complete, `0x818ba8` builds the output + route records. +- `0x8189f0` advances with `0x40ade4`, then materializes output with + `0x818ba8`. +- `0x40ade4` is the true full-route solver advance loop. It pops items from + `solver+0xd0` while `solver+0xdc != 0`, calls expansion helpers including + `0x40b304` and `0x40b540`, then marks state byte `+0x22 = 2`. +- `0x40b304` and `0x40b540` expand neighbor lists from lane/node records at + `solver+0xf8 + index*0xa0`. +- Neighbor admissibility is `0x40b774`, which checks flags at `node+0x88`: + pass iff `(solver+0xc0 & flags88) != 0` and + `(solver+0xc2 & flags88) == 0`. +- Main cost formula in `0x40b304`: + `g = current.g + delta * 0x40bb40(solver,node) * directionMultiplier`; + heuristic is straight-line distance to the solver target position; `f = g+h`. +- `0x40ba58` is the relax/update helper. It writes candidate `f` to node-state + `+0x14`, `g` to `+0x10`, progress to `+0x18`, predecessor state to `+0x1e`, + marks state byte `+0x22 = 1`, and pushes to open list `solver+0xd0`. +- Vanilla `0x40bb40` is the key full-solver per-node multiplier. For driving + modes it treats `Road` (`0x0010`) as `1.0` or `1.2`, with an extra `1.1` + multiplier if byte `node+0x93 == 1`. It does not special-case `Highway` + (`0x4000`). +- Vanilla non-road fallback uses the float table at `0x3154d28`: + `10, 6, 2, 0, 4, 2.5, 1.5, 0`. +- Vanilla `0x40bb00` auxiliary multiplier returns `1.1` for most road cases, + or uses the table at `0x3154d38` (`4, 2.5, 1.5, 0`) for one mode-2/retry + branch. +- Practical inference: the full solver already has a real edge-cost surface, + but vanilla highways are merely roads. The installed build patches that exact + multiplier instead of the earlier local VAND search callback. + ## New Toolbox Tools The Fedora toolbox `2077` now has: @@ -132,6 +188,7 @@ instead of one-off pip installs. In `red4ext/EdgeWeightGPS/src/Main.cpp`: +- `kEnableGpsMultiplierHooks = true` - `kEnableGpsProviderClassPatch = false` - `kEnableGpsProviderClassPatchTrace = false` - `kEnableGpsTraceHooks = true` @@ -143,8 +200,9 @@ In `red4ext/EdgeWeightGPS/src/Main.cpp`: - `kEnableGpsLocalSearchHooks = false` - `kEnableRouteUiTraceHooks = false` -If rebuilt with the current defaults, active hooks are focused map-route submit -and final route-result fetch tracing only. The spatial edge-cost patch requires +If rebuilt with the current defaults, active hooks are focused map-route submit, +final route-result fetch tracing, and the full solver multiplier hook at +`0x40bb00`/`0x40bb40`. The spatial edge-cost patch requires `kEnableGpsSpatialEdgeWeightPatch = true`. The full local search/cost hooks below require `kEnableGpsLocalSearchHooks = true`: @@ -159,6 +217,8 @@ and final route-result fetch tracing only. The spatial edge-cost patch requires - `GPSEdgeCost 0x44f838` - filtered provider filter `0x44ff68` - base provider filter `0x450b08` +- full solver auxiliary multiplier `0x40bb00` +- full solver node multiplier `0x40bb40` Async route-job hooks (`0x8d17d8`, `0x126b28`, `0x883cd8`, `0x11f5f60`, `0x818ba8`) are split behind diff --git a/docs/traffic-system-debrief.md b/docs/traffic-system-debrief.md index c031313..664f225 100644 --- a/docs/traffic-system-debrief.md +++ b/docs/traffic-system-debrief.md @@ -873,15 +873,139 @@ identical record counts. Therefore `0x44f838` remains a real local graph edge cost callback, but changing it does not change the final selected route records for the tested full player GPS routes. +Static disassembly after that failure found a separate, broader async traffic +solver upstream of final route-record materialization. The useful path is: + +```text +map route submit 0x8d20d4 +shared query submit 0x70a42c +query dispatch 0x70a570 +async request enqueue 0x8d17d8 / 0x8d1880 +worker queue loop 0x126c06 +candidate/projection setup 0x883cd8 +solver start wrapper 0x1caa7a0 +solver candidate initialization 0x88367c +route job start/advance 0x818928 +solver advance loop 0x40ade4 +neighbor expansion 0x40b304 / 0x40b540 +relax/update helper 0x40ba58 +output route build 0x818ba8 +``` + +This path is distinct from the local `0x44f054` VAND search. The worker around +`0x126c06` pulls queued route requests from `routeSystem+0x47c0`. Depending on +state it either calls `0x818928` directly or first calls `0x883cd8`, which +resolves/projects the start and end positions into candidate arrays. The +wrapper at `0x1caa7a0` starts the solver by storing the route mode at +`solver+0xc4`, writing route masks at `solver+0xc0/+0xc2`, copying candidate +arrays into `solver+0x70/+0x80`, and calling `0x88367c`. + +`0x88367c` marks start candidate lane/node records with `node+0x9a |= 1`, marks +end candidates with `node+0x9a |= 2`, computes initial node-state costs at +offsets `+0x10/+0x14`, and queues initial states into the open list at +`solver+0xd0`. + +`0x40ade4` is the full-route solver advance loop. It processes the open list +while `solver+0xdc != 0`, calls helpers including `0x40b304` and `0x40b540` to +expand neighbors, and marks completed node states with byte `+0x22 = 2`. +Current lane/node records are addressed as: + +```text +laneNode = solver + 0xf8 + currentNodeIndex * 0xa0 +``` + +`0x40b304` walks one neighbor list from the current record; `0x40b540` walks a +second list with a slightly different edge-record shape. Both filter neighbors +through `0x40b774`, which reads the 16-bit lane/node flags at `node+0x88`: + +```text +pass iff (solver+0xc0 & flags88) != 0 + and (solver+0xc2 & flags88) == 0 +``` + +The main cost equation visible in `0x40b304` is: + +```text +g = current.g + progress_delta + * 0x40bb40(solver, currentLaneNode) + * directionMultiplier +h = distance(neighborPosition, solverTargetPosition) +f = g + h +``` + +`directionMultiplier` is normally `1.0`; for reverse/special progress branches +it can come from `0x40bb00(solver, currentLaneNode)` or become `FLT_MAX`. +`0x40ba58` is the relax/update function. When a candidate improves, it writes +`f` to node-state `+0x14`, `g` to `+0x10`, progress to `+0x18`, predecessor to +`+0x1e`, marks state byte `+0x22 = 1`, and pushes the state back into +`solver+0xd0`. + +The decisive vanilla multiplier is `0x40bb40`. For driving-style modes it reads +the same `node+0x88` flags and handles them roughly as: + +```text +if mode == 1: + pavement flag 0x0008 -> 1.0 + otherwise -> 1.5 + +if mode in {0, 2, 4}: + if Road flag 0x0010: + base = 1.0 or 1.2 depending on a low node flag + if node+0x93 == 1: base *= 1.1 + return base + else: + return table[solver+0x54] + +fallback: + return 1.0 +``` + +The non-road table at `0x3154d28` starts: + +```text +10.0, 6.0, 2.0, 0.0, 4.0, 2.5, 1.5, 0.0 +``` + +The auxiliary multiplier at `0x40bb00` usually returns `1.1` for road cases, +or, for one mode-2 branch, reads a smaller table at `0x3154d38`: + +```text +4.0, 2.5, 1.5, 0.0 +``` + +The important design finding is that vanilla has a real edge-cost surface in +the final async solver, but it does not special-case the `Highway` flag +(`0x4000`). A highway lane that is also a road is simply a road. That matches +the original symptom: the solver is weighted enough to avoid obviously invalid +surfaces, but it has no strong reason to stay on highway corridors for long +vehicle trips. + +The current RED4ext prototype therefore hooks `0x40bb40` and only changes +driving-mode calls (`solver/job + 0xc4 == 2`). It reads flags from +`node+0x88`, leaves vanilla results intact for ordinary roads and unknown +surfaces, and applies this diagnostic multiplier set: + +```text +Highway 0.35 +GPSOnly 1.10 +Pavement 1.25 +Other 1.00 +``` + +`0x40bb00` is currently hooked only for signature logging. If the node +multiplier test moves routes but produces weird reverse/turnaround behavior, +the next step is to tune or gate the auxiliary multiplier as well. + ## Future Improvements Better versions of this mod could: -- identify the broader async traffic route search/expansion loop around the - `0x8d25xx` to `0x8d49xx` cluster -- disassemble the full map-route submit/result chain around `0x8d20d4`, - `0x70a42c`, `0x70a570`, `0x7094b8`, and the `0x520783` result drain to find - the route selection or copy step before final records are emitted +- tune the `0x40bb40` solver multiplier after live A/B testing confirms visible + route changes +- inspect and, if needed, patch `0x40bb00` so reverse/special progress branches + receive the same road-class policy as forward expansion +- add a tiny live-tuning file for the solver multipliers, similar to the earlier + `spatial_weights.bin`, once the hook is confirmed stable - replace the crashing `0x8d46cc` detour with a narrower inline patch or a safer caller-side hook - decode enough of the `VAND` graph to join navigation points to traffic lane diff --git a/tools/disassemble_pe_range.py b/tools/disassemble_pe_range.py new file mode 100644 index 0000000..9b9958c --- /dev/null +++ b/tools/disassemble_pe_range.py @@ -0,0 +1,101 @@ +#!/usr/bin/env python3 +"""Disassemble a PE .text range by RVA and annotate direct branch targets.""" + +from __future__ import annotations + +import argparse +import math +import struct +from pathlib import Path + +from capstone import Cs, CS_ARCH_X86, CS_MODE_64 +from capstone.x86 import X86_OP_IMM, X86_OP_MEM, X86_REG_RIP + +from find_pe_string_xrefs import Section, parse_pe + + +def find_section(sections: list[Section], rva: int) -> Section: + for section in sections: + if section.contains_rva(rva): + return section + raise ValueError(f"RVA 0x{rva:x} is not inside any PE section") + + +def find_section_or_none(sections: list[Section], rva: int) -> Section | None: + for section in sections: + if section.contains_rva(rva): + return section + return None + + +def parse_rva(text: str) -> int: + return int(text, 16 if text.lower().startswith("0x") else 16) + + +def format_memory_annotation(data: bytes, sections: list[Section], target_rva: int) -> str: + section = find_section_or_none(sections, target_rva) + if section is None: + return f"rip_rva=0x{target_rva:x}" + + offset = section.rva_to_offset(target_rva) + if offset < 0 or offset >= len(data): + return f"rip_rva=0x{target_rva:x}" + + available = data[offset : min(len(data), offset + 8)] + fields = [f"rip_rva=0x{target_rva:x}"] + if len(available) >= 4: + u32 = struct.unpack_from("= 8: + u64 = struct.unpack_from(" int: + parser = argparse.ArgumentParser() + parser.add_argument("pe", type=Path) + parser.add_argument("rva", type=parse_rva) + parser.add_argument("--size", type=lambda value: int(value, 0), default=0x200) + parser.add_argument("--before", type=lambda value: int(value, 0), default=0) + args = parser.parse_args() + + data = args.pe.read_bytes() + image_base, sections = parse_pe(data) + start_rva = max(0, args.rva - args.before) + section = find_section(sections, start_rva) + end_rva = min(section.virtual_address + section.raw_size, start_rva + args.size) + start_offset = section.rva_to_offset(start_rva) + code = data[start_offset : start_offset + (end_rva - start_rva)] + + disassembler = Cs(CS_ARCH_X86, CS_MODE_64) + disassembler.detail = True + + print(f"image_base=0x{image_base:x} section={section.name} range=0x{start_rva:x}..0x{end_rva:x}") + for insn in disassembler.disasm(code, image_base + start_rva): + rva = insn.address - image_base + annotation = "" + if insn.group(1) or insn.group(2): # jump/call + for operand in insn.operands: + if operand.type == X86_OP_IMM: + annotation = f" ; target_rva=0x{operand.imm - image_base:x}" + break + elif insn.operands: + memory_annotations = [] + for operand in insn.operands: + if operand.type == X86_OP_MEM and operand.mem.base == X86_REG_RIP: + target_rva = (insn.address + insn.size + operand.mem.disp) - image_base + memory_annotations.append(format_memory_annotation(data, sections, target_rva)) + if memory_annotations: + annotation = " ; " + " ; ".join(memory_annotations) + marker = ">>" if rva == args.rva else " " + print(f"{marker} 0x{rva:08x}: {insn.mnemonic:<8} {insn.op_str}{annotation}") + + return 0 + + +if __name__ == "__main__": + raise SystemExit(main())