Calibrate and install GPS lane weighting archive

.gitignore

@@ -1 +1,5 @@
 *.swp
+__pycache__/
+*.pyc
+/work/
+/work-*.txt

README.md

@@ -33,8 +33,8 @@ python3 tools/patch_traffic_lanes.py exported-json-dir patched-json-dir
 The patcher recursively scans WolvenKit JSON exports for `lanes` arrays that look
 like `worldTrafficLanePersistent` data and applies configurable speed weighting:
 
-- highways get boosted toward a preferred GPS speed
+- highways get boosted toward a preferred GPS speed in the game's lane scale
-- regular roads get a moderate speed floor
+- regular roads keep a moderate speed floor
 - pavement, crosswalk, and disabled traffic lanes are left alone
 - optional non-highway speed caps can make highways more attractive
 
@@ -48,10 +48,17 @@ toolbox run --container 2077 bash -lc '$HOME/.dotnet/tools/cp77tools --help'
 ```
 
 See [docs/tooling.md](docs/tooling.md) for the end-to-end archive build flow.
+See [docs/traffic-system-debrief.md](docs/traffic-system-debrief.md) for the
+detailed feasibility and traffic graph notes.
 
 ## Status
 
-This is not yet a packaged playable mod because this workspace does not contain
+A test archive was built from the local Flatpak Steam Phantom Liberty install and
-the Cyberpunk archive resources. The toolbox has the required CLI tooling now, so
+installed to:
-the next step is running the archive build script against a local Cyberpunk 2077
+
-install path.
+```text
+Cyberpunk 2077/archive/pc/mod/zz_edge_weight_gps.archive
+```
+
+Build outputs are ignored by git. The installed archive should be treated as
+experimental until route behavior is checked in game.

docs/traffic-system-debrief.md

@@ -0,0 +1,274 @@
+# Cyberpunk 2077 Traffic/GPS System Debrief
+
+This is based on the public NativeDB RTTI dump, WolvenKit exports from the local
+Steam install, and the generated `all.traffic_persistent` resource from the
+Phantom Liberty archive.
+
+## The High-Level Shape
+
+Cyberpunk appears to split "navigation" into at least two major domains:
+
+- pedestrian/NPC navigation over navmesh resources
+- vehicle/GPS routing over generated traffic lane resources
+
+The public script/native type dump exposes `worldNavigationScriptInterface` and
+`AINavigationSystem` methods for navmesh path queries, but those are character
+navigation tools. They are not the car GPS. The car GPS system type exists as
+`gamegpsGPSSystem`, but its public RTTI surface does not expose planner methods
+that a REDscript mod can override.
+
+That means the practical mod surface is data, not algorithm replacement. The
+native `gamegpsGPSSystem` likely builds or queries a graph from traffic lane
+resources, then renders a GPS line from the selected lane path.
+
+## Relevant Game Types
+
+The useful RTTI types are:
+
+- `gamegpsGPSSystem`
+- `gamegpsSettings`
+- `worldTrafficPersistentResource`
+- `worldTrafficPersistentData`
+- `worldTrafficLanePersistent`
+- `worldTrafficLanePlayerGPSInfo`
+- `worldTrafficLanePersistentFlags`
+
+`gamegpsSettings` contains display/refresh settings:
+
+- `lineEffectOnFoot`
+- `lineEffectVehicle`
+- `fixedPathOffset`
+- `fixedPortalMappinOffset`
+- `pathRefreshTimeInterval`
+- `lastPlayerNavmeshPositionRefreshTimeIntervalSecs`
+- `maxPathDisplayLength`
+
+Those affect GPS presentation and refresh behavior, not edge selection.
+
+`worldTrafficPersistentResource` is the key resource type. Its root chunk has
+`data: worldTrafficPersistentData`, which has:
+
+- `lanes: array<worldTrafficLanePersistent>`
+- `neighborGroups: array<array<Uint16>>`
+
+Each lane has the actual graph metadata:
+
+- `outLanes`
+- `inLanes`
+- `outline`
+- `accumulatedLengths`
+- `crowdCreationInfo`
+- `maxSpeed`
+- `deadEndStart`
+- `length`
+- `width`
+- `area`
+- `flags`
+- `subGraphId`
+- `playerGPSInfo`
+- `neighborGroupIndex`
+- `nodeRefHash`
+- `laneNumber`
+- `seqNumber`
+- `isReversed`
+- `roadMaterials`
+- `polygon`
+
+`playerGPSInfo` contains:
+
+- `subGraphId`
+- `stronglyConnectedComponentId`
+
+That strongly suggests the GPS planner has a precomputed connectivity graph and
+uses connected-component/subgraph metadata to reject impossible paths quickly.
+
+## Resource Location
+
+In this install, the important resource is:
+
+```text
+base\worlds\03_night_city\sectors\_generated\traffic\all.traffic_persistent
+```
+
+It exists in both the basegame Night City archive and the Phantom Liberty Night
+City archive. For a Phantom Liberty install, the EP1 archive version should win:
+
+```text
+archive/pc/ep1/ep1_1_nightcity.archive
+```
+
+WolvenKit can extract, serialize, deserialize, and pack this resource on Linux
+through `wolvenkit.cli` 8.18.1.
+
+## Lane Flags
+
+`worldTrafficLanePersistentFlags`:
+
+```text
+FromRoadSpline = 1
+Bidirectional = 2
+PatrolRoute = 4
+Pavement = 8
+Road = 16
+Intersection = 32
+NeverDeadEnd = 64
+TrafficDisabled = 128
+CrossWalk = 256
+GPSOnly = 512
+ShowDebug = 1024
+Blockade = 2048
+Yield = 4096
+NoAIDriving = 8192
+Highway = 16384
+NoAutodrive = 32768
+```
+
+The important flags for GPS weighting are `Road`, `Intersection`, `GPSOnly`, and
+`Highway`. The important flags to avoid touching are `Pavement`, `CrossWalk`,
+`TrafficDisabled`, and `Blockade`.
+
+## Observed Lane Distribution
+
+The local EP1 `all.traffic_persistent` export contains `31,717` lanes.
+
+Observed `maxSpeed` distribution:
+
+```text
+all lanes:
+  1: 12284
+  15: 11638
+  10: 4675
+  17: 3013
+  14: 96
+  8: 8
+  9: 2
+  4: 1
+
+highway lanes:
+  17: 2688
+  15: 164
+  1: 2
+
+road lanes:
+  15: 11474
+  10: 495
+  17: 325
+  14: 96
+  8: 4
+
+pavement lanes:
+  1: 12280
+  10: 4180
+  8: 4
+  9: 2
+  4: 1
+```
+
+The values are not real-world mph/kph. They are compact game-scale lane speeds.
+Because highways are already mostly `17` and normal roads are mostly `15`, a
+large value like `90` would be reckless. The mod currently uses conservative
+defaults:
+
+- highway speed target: `25`
+- road speed floor: `15`
+
+That changes `2,506` lanes:
+
+- `2,419` highway boosts
+- `87` road floor boosts
+
+It leaves pavement/crosswalk/blocked/disabled lanes alone.
+
+## What The Mod Actually Does
+
+The mod does not replace the pathfinding algorithm.
+
+It edits the traffic graph's lane metadata so the native GPS planner has better
+cost inputs. If the planner factors `maxSpeed` into traversal cost in the usual
+way, then a segment with the same physical length but higher `maxSpeed` becomes
+cheaper. That nudges the native planner toward highways and high-capacity roads
+without having to hook the planner itself.
+
+Conceptually:
+
+```text
+old cost ~= lane.length / lane.maxSpeed
+new cost ~= lane.length / patched_lane.maxSpeed
+```
+
+The actual game formula is native and not exposed in RTTI, so this is an
+inference. The inference is supported by the presence of `maxSpeed`, `length`,
+connectivity data, and GPS-specific lane graph metadata in the same resource.
+
+## Why Not Just Write A New A*?
+
+A standalone A* is easy. Integrating it into Cyberpunk's GPS is the hard part.
+
+The game already has:
+
+- traffic lane connectivity
+- lane polygons/outlines
+- player lane matching
+- destination lane matching
+- dynamic refresh
+- minimap/world map rendering
+- quest mappin target integration
+- portal/entrance handling
+- route line effects
+
+The script-facing `gamegpsGPSSystem` type does not expose hooks for replacing
+that pipeline. A custom planner would still need a way to feed its result back
+into the GPS renderer and route-following systems. Without native RED4ext hooks,
+that is much riskier than patching the resource data the existing pipeline
+already consumes.
+
+## Packaging Flow
+
+The working flow is:
+
+1. Extract `all.traffic_persistent` with WolvenKit CLI.
+2. Serialize the CR2W resource to JSON.
+3. Patch lane `maxSpeed` values.
+4. Deserialize the patched JSON back to CR2W.
+5. Put the CR2W back under the original resource path.
+6. Pack an archive.
+7. Place the archive in `archive/pc/mod`.
+
+The generated archive currently contains exactly:
+
+```text
+base\worlds\03_night_city\sectors\_generated\traffic\all.traffic_persistent
+```
+
+## Risks And Unknowns
+
+The main unknown is the native GPS cost formula. If it ignores `maxSpeed`, this
+patch will not meaningfully change routes. If it uses speed but with caps or
+category penalties, the effect may be subtle. If it uses `maxSpeed` for both GPS
+and traffic simulation, the patch may also affect some traffic behavior on
+highways.
+
+The Oodle library was not available inside the toolbox, so WolvenKit packed with
+its Kraken fallback. WolvenKit successfully round-tripped and listed the archive,
+but in-game testing is still required.
+
+The patch targets the EP1 traffic resource because Phantom Liberty is installed.
+On a non-PL install, the basegame archive resource would need to be patched
+instead.
+
+## Future Improvements
+
+Better versions of this mod could:
+
+- build district-specific presets
+- increase highway speed to `22`, `25`, or `30` based on playtesting
+- lower intersection penalties only where highway ramps are misclassified
+- inspect `lane_connections` to identify ramp/arterial topology
+- patch `GPSOnly` connector lanes separately
+- generate a diff report of every changed lane with flags, length, speed, and
+  graph component
+- ship multiple archives: conservative, strong, and experimental
+
+The next practical step is in-game route testing between known bad waypoint
+pairs, especially cases where vanilla GPS exits highways too early or cuts
+through surface streets.

samples/traffic_resource.json

@@ -5,7 +5,7 @@
         "lanes": [
           {
             "flags": 16400,
-            "maxSpeed": 55,
+            "maxSpeed": 17,
             "length": 240.0,
             "laneNumber": 1,
             "playerGPSInfo": {
@@ -15,7 +15,7 @@
           },
           {
             "flags": 16,
-            "maxSpeed": 25,
+            "maxSpeed": 10,
             "length": 80.0,
             "laneNumber": 2,
             "playerGPSInfo": {

tests/test_patch_traffic_lanes.py

@@ -1,6 +1,7 @@
 import json
 import subprocess
 import sys
+import tempfile
 from pathlib import Path
 
 
@@ -8,7 +9,7 @@ ROOT = Path(__file__).resolve().parents[1]
 TOOL = ROOT / "tools" / "patch_traffic_lanes.py"
 
 
-def test_patch_sample(tmp_path):
+def run_patch_sample(tmp_path):
     output = tmp_path / "patched.json"
 
     result = subprocess.run(
@@ -28,6 +29,15 @@ def test_patch_sample(tmp_path):
     patched = json.loads(output.read_text(encoding="utf-8"))
     lanes = patched["Data"]["RootChunk"]["data"]["lanes"]
 
-    assert lanes[0]["maxSpeed"] == 90
+    assert lanes[0]["maxSpeed"] == 25
-    assert lanes[1]["maxSpeed"] == 45
+    assert lanes[1]["maxSpeed"] == 15
     assert lanes[2]["maxSpeed"] == 10
+
+
+def test_patch_sample(tmp_path):
+    run_patch_sample(tmp_path)
+
+
+if __name__ == "__main__":
+    with tempfile.TemporaryDirectory() as tmp:
+        run_patch_sample(Path(tmp))

tools/analyze_traffic_lanes.py

@@ -0,0 +1,77 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import json
+from collections import Counter
+from pathlib import Path
+from typing import Any
+
+from patch_traffic_lanes import FLAGS, as_int, has_flag, looks_like_lane
+
+
+def visit(value: Any):
+    if isinstance(value, dict):
+        lanes = value.get("lanes")
+        if isinstance(lanes, list) and any(looks_like_lane(lane) for lane in lanes):
+            for lane in lanes:
+                if looks_like_lane(lane):
+                    yield lane
+        for child in value.values():
+            yield from visit(child)
+    elif isinstance(value, list):
+        for child in value:
+            yield from visit(child)
+
+
+def bucket(flags: int) -> str:
+    if has_flag(flags, "Highway"):
+        return "highway"
+    if has_flag(flags, "Pavement"):
+        return "pavement"
+    if has_flag(flags, "CrossWalk"):
+        return "crosswalk"
+    if has_flag(flags, "Road"):
+        return "road"
+    return "other"
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("json_file", type=Path)
+    args = parser.parse_args()
+
+    data = json.loads(args.json_file.read_text(encoding="utf-8"))
+    speeds: dict[str, Counter[int]] = {
+        "all": Counter(),
+        "highway": Counter(),
+        "road": Counter(),
+        "pavement": Counter(),
+        "crosswalk": Counter(),
+        "other": Counter(),
+    }
+    flags_counter: Counter[int] = Counter()
+
+    for lane in visit(data):
+        flags = as_int(lane.get("flags"))
+        speed = as_int(lane.get("maxSpeed"))
+        if flags is None or speed is None:
+            continue
+        speeds["all"][speed] += 1
+        speeds[bucket(flags)][speed] += 1
+        flags_counter[flags] += 1
+
+    for name, counter in speeds.items():
+        print(f"{name}: {sum(counter.values())} lanes")
+        print("  " + ", ".join(f"{speed}:{count}" for speed, count in counter.most_common(20)))
+
+    print("top flags:")
+    for flags, count in flags_counter.most_common(20):
+        names = [name for name, bit in FLAGS.items() if flags & bit]
+        print(f"  {flags}: {count} ({', '.join(names)})")
+
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

tools/build_lane_weight_archive.sh

@@ -2,36 +2,41 @@
 set -euo pipefail
 
 if [[ $# -lt 3 ]]; then
-  echo "usage: $0 <game-dir> <archive-or-archive-dir> <work-dir>" >&2
+  echo "usage: $0 <game-dir> <archive-or-archive-dir> <work-dir> [resource-regex] [resource-path] [mod-name]" >&2
   exit 2
 fi
 
 game_dir=$1
 archive_path=$2
 work_dir=$3
-container=${CP77_TOOLBOX_CONTAINER:-2077}
+resource_regex=${4:-'all\.traffic_persistent$'}
-tool='${HOME}/.dotnet/tools/cp77tools'
+resource_path=${5:-'base/worlds/03_night_city/sectors/_generated/traffic/all.traffic_persistent'}
+mod_name=${6:-zz_edge_weight_gps}
 
 raw_dir="${work_dir}/01_raw"
 json_dir="${work_dir}/02_json"
 patched_json_dir="${work_dir}/03_json_patched"
 patched_raw_dir="${work_dir}/04_raw_patched"
+pack_root="${work_dir}/${mod_name}"
 packed_dir="${work_dir}/05_packed"
+resource_dir=$(dirname "${resource_path}")
+resource_file=$(basename "${resource_path}")
 
-mkdir -p "${raw_dir}" "${json_dir}" "${patched_json_dir}" "${patched_raw_dir}" "${packed_dir}"
+mkdir -p "${raw_dir}" "${json_dir}" "${patched_json_dir}" "${patched_raw_dir}" "${pack_root}/${resource_dir}" "${packed_dir}"
 
-toolbox run --container "${container}" bash -lc \
+tools/cp77_toolbox.sh extract "${archive_path}" \
-  "${tool} extract '$archive_path' --gamepath '$game_dir' --outpath '$raw_dir' --regex 'traffic.*persistent|persistent.*traffic|traffic.*lane|world.*traffic'"
+  --gamepath "${game_dir}" \
+  --outpath "${raw_dir}" \
+  --regex "${resource_regex}"
 
-toolbox run --container "${container}" bash -lc \
+tools/cp77_toolbox.sh convert serialize "${raw_dir}" --outpath "${json_dir}"
-  "${tool} convert serialize '$raw_dir' --outpath '$json_dir'"
 
 python3 tools/patch_traffic_lanes.py "${json_dir}" "${patched_json_dir}" --copy-unchanged
 
-toolbox run --container "${container}" bash -lc \
+tools/cp77_toolbox.sh convert deserialize "${patched_json_dir}" --outpath "${patched_raw_dir}"
-  "${tool} convert deserialize '$patched_json_dir' --outpath '$patched_raw_dir'"
 
-toolbox run --container "${container}" bash -lc \
+cp "${patched_raw_dir}/${resource_file}" "${pack_root}/${resource_path}"
-  "${tool} pack '$patched_raw_dir' --outpath '$packed_dir'"
+
+tools/cp77_toolbox.sh pack "${pack_root}" --outpath "${packed_dir}"
 
 echo "packed archive output: ${packed_dir}"

tools/cp77_toolbox.sh

@@ -2,6 +2,6 @@
 set -euo pipefail
 
 container="${CP77_TOOLBOX_CONTAINER:-2077}"
-tool='${HOME}/.dotnet/tools/cp77tools'
+tool="${HOME}/.dotnet/tools/cp77tools"
 
-exec toolbox run --container "${container}" bash -lc "${tool} $*"
+exec toolbox run --container "${container}" "${tool}" "$@"

tools/discover_traffic_resources.sh

@@ -8,11 +8,10 @@ fi
 
 archive_path=$1
 output=${2:-traffic-resources.txt}
-container=${CP77_TOOLBOX_CONTAINER:-2077}
-tool='${HOME}/.dotnet/tools/cp77tools'
 
-toolbox run --container "${container}" bash -lc \
+tools/cp77_toolbox.sh archive --list \
-  "${tool} archive --list --regex 'traffic.*persistent|persistent.*traffic|traffic.*lane|world.*traffic' '$archive_path'" \
+  --regex 'traffic.*persistent|persistent.*traffic|traffic.*lane|world.*traffic' \
+  "${archive_path}" \
   > "${output}"
 
 echo "wrote ${output}"

tools/patch_traffic_lanes.py

@@ -208,8 +208,8 @@ def parse_args() -> argparse.Namespace:
     parser = argparse.ArgumentParser()
     parser.add_argument("input", type=Path, help="WolvenKit JSON file or export directory")
     parser.add_argument("output", type=Path, help="Patched JSON file or output directory")
-    parser.add_argument("--highway-speed", type=int, default=90)
+    parser.add_argument("--highway-speed", type=int, default=25)
-    parser.add_argument("--road-speed-floor", type=int, default=45)
+    parser.add_argument("--road-speed-floor", type=int, default=15)
     parser.add_argument("--non-highway-cap", type=int, default=None)
     parser.add_argument("--copy-unchanged", action="store_true")
     parser.add_argument("--dry-run", action="store_true")