Benchmarks & Competition¶

Throughput¶

python -m benchmarks.throughput --batches 1024 4096 --steps 50 --warmup 2
# -> results/throughput.json

Measured on CPU. agent-steps/s = env-steps/s × agents:

Backend	Batch	env-steps/s	agent-steps/s
Legacy Python	1	~300	~1,200
JAX vectorized	1024	158,540	634,160
JAX vectorized	4096	270,795	1,083,181

Competition eval¶

benchmarks/competition_eval.py runs submitted agents across scenarios and seeds and maintains separate red/blue leaderboards (results/leaderboard.json).

import numpy as np
from benchmarks.competition_eval import RandomAgent, SubmissionResult, evaluate, submit_to_leaderboard

class MyBlueAgent:
    def reset(self) -> None:
        pass

    def act(self, obs: dict, agent_id: str) -> np.ndarray:
        mask = obs["action_mask"]
        types = np.where(mask[:32])[0]
        targets = np.where(mask[32:])[0]
        return np.array([types[0], targets[0]], dtype=np.int64)

sub = SubmissionResult(name="my_blue", team="blue")
evaluate(sub, red_agent=RandomAgent(), blue_agent=MyBlueAgent(),
         scenarios=["ransomware"], seeds=list(range(10)))
submit_to_leaderboard(sub)

Scoring:

Blue = SLA_uptime × 50 − compromised × 2 − MTTC × 0.1 + blue_reward × 0.1
Red = compromised × 2 + exfiltrated × 0.01 − SLA_uptime × 10 + red_reward × 0.1

python -m benchmarks.competition_eval --name my_agent --team blue --episodes 10
python -m benchmarks.competition_eval --leaderboard

Full-sweep runner¶

benchmarks/run_benchmark.py runs a policy across all scenarios and seeds with CI95 and writes ranked, per-team leaderboards. Red policies: random, heuristic, killchain.

# Blue heuristic across all scenarios, 20 seeds
python -m benchmarks.run_benchmark --name heuristic --team blue --seeds 20

# Kill-chain red vs heuristic blue
python -m benchmarks.run_benchmark --name killchain --team red --red killchain --blue heuristic --seeds 20

# Train vs held-out generalization gap
python -m benchmarks.run_benchmark --name killchain --team red --red killchain --gap --seeds 20

# Multi-policy leaderboard (padding excluded from host counts)
python -m benchmarks.build_leaderboard --episodes 5 --max-steps 150

See Baselines for the policy descriptions.

Environment spec¶

python -m benchmarks.env_spec --json   # -> results/env_spec.{md,json}

Metrics in episode `info`¶

Each step returns per-agent info keys:

Key	Meaning
`compromised_hosts` / `isolated_hosts` / `active_hosts`	active host counts (padding excluded)
`SLA_Uptime_Percentage`	rolling fraction of healthy, online hosts
`MTTC` / `containment_time`	mean time to containment
`detection_rate`	fraction of compromises that were isolated
`Total_Exfiltrated_Data`	cumulative exfiltration
`false_positives` / `successful_exploits` / `services_restored`	per-step action outcomes
`deception_hits` / `deception_efficacy`	red actions striking decoys/honeytokens, and the ratio
`attack_techniques` / `attack_coverage`	MITRE ATT&CK technique ids exercised by Red, and the fraction of the taxonomy
`blue_score` / `red_score` / `normalized_reward`	composite scores
`agent_energy`	remaining action budget
`information_asymmetry`	present under `DiagnosticsWrapper`: L2 distance to the oracle view

The __curriculum__ key is present when using CurriculumWrapper and contains phase, phase_index, mean_reward, window_fill, phase_advanced.