Diagnostic Probes¶
Aggregate episode reward tells you whether a policy is good, not why — or what specific
capability it lacks. NetForge ships a suite of targeted diagnostic probes, each of which
plants a controlled situation, runs the policy, and scores one capability in [0, 1].
They live in netforge_rl.diagnostics.
The capability suite¶
| Probe | Capability | What it measures |
|---|---|---|
MemoryProbe |
memory |
Isolating a foothold planted at reset — can Blue act on a fact it must remember? |
NoisySIEM |
attention |
Finding one real compromise amid heavy SIEM noise, penalising false positives |
DelayedTelemetry |
temporal |
Containing an intrusion when the SIEM feed is delayed (log_latency=6) |
FalsePositiveRestraint |
precision |
Restraint on a fully clean network — not quarantining healthy hosts |
OTKineticResponse |
safety |
Isolating a compromised OT/PLC host before kinetic destruction |
TopologyShift |
generalization |
Isolating a target while the surrounding network churns |
Running them¶
from netforge_rl.diagnostics import all_diagnostics, run_diagnostic
from netforge_rl.baselines.policies import HeuristicBluePolicy
for probe in all_diagnostics():
result = run_diagnostic(probe, HeuristicBluePolicy(seed=0), seed=0)
print(f'{probe.capability:14} {probe.name:26} {result.score:.2f} {result.details}')
run_diagnostic returns a DiagnosticResult(diagnostic, capability, policy, score, details).
details carries probe-specific evidence (e.g. the isolation tick, false-positive count, or
whether kinetic destruction occurred).
The suite is discriminative: a heuristic blue policy beats a random policy on every
containment-style probe, and scores are deterministic under seed (see the tests in
tests/diagnostics/test_capability_probes.py).
Capability cards¶
capability_card runs the whole suite across seeds and summarises a policy as one
artifact — a JSON of per-capability scores plus a radar chart:
from netforge_rl.diagnostics.capability_card import capability_card
from netforge_rl.baselines.policies import HeuristicBluePolicy
card = capability_card(
lambda: HeuristicBluePolicy(seed=0),
seeds=(0, 1, 2),
out_dir='runs/cards',
name='heuristic_blue',
)
print(card['capabilities'], card['overall'])
# writes runs/cards/heuristic_blue_card.json and .png (radar chart, if matplotlib present)
The card makes two policies instantly comparable at a glance: instead of one aggregate number, you see the shape of their competence across memory, attention, temporal reasoning, precision, safety, and generalization.
How a probe works¶
Each probe subclasses Diagnostic and overrides a few hooks:
class Diagnostic:
scenario_type = 'ransomware'
max_ticks = 50
def build_env(self, seed=0): ... # optionally set special config (latency, churn)
def setup(self, env): ... # plant the situation post-reset
def early_stop(self, env): ... # stop as soon as the objective is met
def score(self, env, ticks_used): ...# return a DiagnosticResult in [0, 1]
Writing a new probe is a matter of planting state in setup and scoring in score — for
example, a credential-hygiene probe would plant a leaked token and score Blue on rotating it
before Red escalates to Root.
Oracle information-asymmetry¶
Separately, DiagnosticsWrapper is a PettingZoo wrapper that injects an oracle observation
(perfect information) alongside each agent's real observation and reports their L2 distance as
info['information_asymmetry']. It quantifies how much the fog of war and SIEM filtering hide
from the agent at each step — a per-step measure of partial observability.