- Python 100%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
|
Some checks failed
CI / Python 3.11 (push) Has been cancelled
CI / Python 3.12 (push) Has been cancelled
CI / Python 3.13 (push) Has been cancelled
CI / Python 3.14 (push) Has been cancelled
CI / Optional SDK integrations (push) Has been cancelled
CI / Evaluation artifact smoke (push) Has been cancelled
CI / External Reelay case study (push) Has been cancelled
|
||
| .github/workflows | ||
| docs | ||
| evaluation | ||
| examples | ||
| schemas | ||
| src/agentic_rv | ||
| tests | ||
| .gitignore | ||
| CITATION.cff | ||
| CONTRIBUTING.md | ||
| LICENSE | ||
| pyproject.toml | ||
| README.md | ||
Agentic-RV
Formalism-agnostic Runtime Verification for agentic AI systems.
Agentic-RV connects an agent runtime to one or more user-supplied monitors without requiring either side to understand the other. Adapters translate framework and protocol activity into a versioned canonical event model; projections and abstractions give each monitor only its authorized view; normalizers translate native verdicts into a common domain; and a deterministic policy chooses a reaction that the active adapter can actually enforce.
The core package does not implement a particular runtime-verification formalism. Your monitor may be generated from temporal logic, backed by a rule engine, learned, stateful, or remote. It only needs to implement the small monitor protocol described below.
Project status:
0.1.0is an alpha research implementation. The event, monitor, trace, and deployment envelopes are explicitly versioned, but production users should still pin the package version and test their exact adapter/SDK combination.
Contents
- Why Agentic-RV
- Architecture
- What the framework can guarantee
- Installation
- Quick start
- Guided tutorial and examples
- Write a monitor
- Connect an agent system
- Policy, deadlines, and failures
- Traces and offline replay
- Evaluation and paper artifacts
- Documentation map
- Design inspiration
- Development
Why Agentic-RV
Agent frameworks expose different callbacks, request types, identifiers, and control points. Monitoring tools expose different input languages and verdict domains. Directly coupling every framework to every monitor creates a matrix of one-off integrations.
Agentic-RV separates that matrix into stable seams:
- A system adapter observes or interposes at an agent/framework boundary and
emits a
CanonicalEvent. - The runtime assigns a total per-runtime sequence and routes matching events to monitors.
- A monitor-specific projection removes unauthorized fields, then an optional abstraction converts the remaining view into the monitor's input domain.
- The user-supplied monitor consumes an ordered
MonitorRequestand may return a native verdict. - A normalizer maps that native value to
permit,reject,inconclusive, orerror. - For a held decision point, the policy combines synchronous responses and selects a reaction supported by that adapter boundary.
Adding a monitor does not require changing an adapter. Adding an adapter does not require knowing which verification formalism produced the monitor.
Architecture
flowchart LR
S["Agentic AI system"] --> A["Framework / protocol adapter"]
A -->|"canonical observation or held decision"| R["Ordered runtime"]
R -->|"raw canonical records"| T[("JSONL audit trace")]
R --> P["Per-monitor projection"]
P --> B["Optional abstraction"]
B --> M["User monitor<br/>Python or HTTP"]
M --> N["Verdict normalizer"]
N --> D["Deterministic policy"]
D -->|"supported pre-effect reaction"| A
D -. "post-hoc advisory only" .-> H["Audit / configured advisory handler"]
A -->|"allow or validated replacement"| E["Underlying operation"]
A -->|"block"| X["OperationBlocked"]
One private event-loop thread owns sequencing and one serial queue per monitor. This gives synchronous callbacks and asynchronous clients the same ordered delivery pipeline without binding monitor state to the application's event loop.
The built-in integrations are:
| Boundary | Passive visibility | Pre-effect interposition |
|---|---|---|
| LangGraph | Runnable/LangChain callback handler | Compiled-graph proxy, state-update proxy, and guarded native tools |
| MCP | Wrapped client-session operations | initialize, call_tool, and read_resource |
| A2A | Client proxy or official interceptor surface | Stable discovery, message, task, subscription, and cancellation calls |
| Other systems | Subclass BaseAdapter |
Emit observations and explicitly hold decision points |
What the framework can guarantee
Agentic-RV's prevention guarantee is deliberately narrow:
A reaction such as
blockorreplacecan affect an operation only when an interposing adapter creates a decision event before that operation, waits for policy selection, advertises the reaction as supported, and applies the selected reaction before dispatching downstream.
Consequently:
- A wrapped, pre-effect operation can be withheld while synchronous monitors run.
- A passive callback, completion event, asynchronous monitor result, or late result is post-hoc. It may be audited or sent to a configured advisory handler, but it cannot retroactively block or replace completed work.
- Enabling an adapter in configuration constructs it lazily; it does not automatically patch an SDK object. The application must use the wrapper, callback, or interceptor at the relevant boundary.
- Only instrumented boundaries are visible. Calls performed elsewhere, remote agent internals, and side effects that occur before a wrapper is entered are outside the guarantee.
- Deadlines are best-effort arbitration deadlines, not hard real-time bounds. Python scheduling, projection code, local monitor code, transport latency, and shutdown can add delay.
- The runtime verifies routing, ordering, normalization, and policy mechanics. It does not prove that a user monitor, specification, abstraction, or projection is semantically correct.
See Security and guarantees for the full threat model and deployment checklist.
Installation
Python 3.11 or newer is required. The core depends only on PyYAML and can be used with custom adapters without installing any agent SDK.
Install the checkout for development:
python -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[all,dev]"
Or select only the integration you use:
python -m pip install "agentic-rv[langgraph]"
python -m pip install "agentic-rv[mcp]"
python -m pip install "agentic-rv[a2a]"
The all extra installs all three optional SDK families. Importing the Agentic-RV
core and adapter modules remains possible without those SDKs; strict deployment
validation checks enabled adapter dependencies unless asked not to.
Researchers regenerating the evaluation reports can additionally install
.[evaluation]. That extra contains Matplotlib for plots only. The external
case study has its own exact Ollama model-manifest lock, hash-pinned Reelay
environment, and two preflights. Ollama, model weights, and Reelay are
deliberately absent from the core, integration, development, and evaluation
extras.
Quick start
The primary onboarding path uses a real compiled LangGraph StateGraph, a
user-supplied Python monitor, and a guarded tool boundary. It permits
read_records and blocks delete_records before the protected function runs.
No model, API key, network service, or external database is required.
python -m pip install -e ".[langgraph]"
python -m agentic_rv.cli validate examples/langgraph_quickstart/deployment.yaml
python -m examples.langgraph_quickstart.app
The important output is:
ALLOW: read_records -> executed:read_records
BLOCK: delete_records -> blocked before execution
SIDE EFFECTS: ["read_records"]
The last line is the pre-effect proof: the rejected call never reached the underlying function. The application performs the three essential integration steps:
with Deployment.load("deployment.yaml") as deployment:
adapter = deployment.langgraph
assert adapter is not None
guarded_tool = adapter.guard_tool(real_tool, name="real_tool")
compiled_graph = build_graph(guarded_tool)
monitored_graph = adapter.wrap_graph(compiled_graph)
result = monitored_graph.invoke(input_state)
The monitor and policy are connected declaratively:
schema_version: agentic-rv.deployment/v1
runtime:
policy:
strategy: deny_overrides
failure_mode: closed
reject_reaction: block
monitors:
- id: action-policy
implementation:
type: python
factory: my_project.monitor:create_monitor
kwargs:
blocked_actions: [delete_records]
subscriptions: [langgraph.tool]
projection:
include:
- event_id
- sequence
- kind
- phase
- payload.tool
- payload.call
normalizer:
type: mapping
mapping:
permit: permit
reject: reject
mode: synchronous
mandatory: true
deadline_ms: 250
allowed_reactions: [allow, block, log]
queue_size: 0
adapters:
langgraph:
enabled: true
decision_timeout_ms: 500
Configuration may be YAML, JSON, or TOML. It is strict: unknown keys, duplicate keys, duplicate monitor IDs, implicit type coercions, and unsupported schema versions fail validation. Inspect the fully expanded canonical form with:
python -m agentic_rv.cli inspect examples/langgraph_quickstart/deployment.yaml
Loading is lazy. Parsing a Deployment does not invoke monitor factories, open a
trace, or construct adapters. The runtime is materialized on first
deployment.runtime access, and an enabled adapter on first access to that
adapter. Entering the context eagerly starts monitors so normal first-use
interposition does not spend its decision budget on startup. Context exit closes
materialized resources.
When running from a source checkout, the documented commands use
python -m agentic_rv.cli so the example-local monitor modules remain on
Python's import path. Installed application packages can use the equivalent
agentic-rv console command.
Guided tutorial and examples
Start with the guided tutorial. It offers:
- a roughly 20-minute path through the first monitored graph, monitor protocol, and data pipeline;
- a full 60–90 minute path through lifecycle, redaction, abstraction, epsilon, multi-monitor policy, failures, validated replacement, traces/replay, an HTTP monitor, MCP, A2A, and custom adapters;
- exact commands, expected results, files to inspect, experiments, and the guarantee or limitation demonstrated at each step.
After the full route, the optional LangGraph/Reelay incident-response capstone connects a real local Ollama model, an iterative LangGraph agent loop, and a genuine external temporal monitor. It includes safe and unsafe model-selected paths, pre-effect intervention, exact model-manifest verification, and sanitized LLM/formal-verdict evidence. Its external components are installed separately from the normal example suite.
Every runnable example lives in its own folder under
examples/. The examples index lists
dependencies and the recommended order. The built-in protocol examples use real
SDK surfaces: a compiled LangGraph graph, an official MCP client/server over the
SDK's in-memory transport, and an official A2A JSON-RPC client with a
deterministic mock HTTP boundary.
Write a monitor
A Python monitor is a callable or an object with evaluate(request). Evaluation
may be synchronous or asynchronous. Optional start(), reset(), and close()
lifecycle hooks may also be synchronous or asynchronous. The configured factory
itself must be synchronous and return the monitor object.
from collections.abc import Mapping
from agentic_rv import MonitorRequest, MonitorResult
class MyMonitor:
def start(self) -> None:
pass
def reset(self) -> None:
# Restore initial state before offline replay.
pass
def evaluate(self, request: MonitorRequest) -> MonitorResult | None:
observation = request.observation
if not isinstance(observation, Mapping):
return None
payload = observation.get("payload")
action = payload.get("action") if isinstance(payload, Mapping) else None
reject = request.decision and action == "dangerous_action"
return MonitorResult(
verdict="unsafe" if reject else "safe",
diagnostics={"action": action},
final=reject,
)
def close(self) -> None:
pass
def create_monitor() -> MyMonitor:
return MyMonitor()
Map the native values explicitly:
implementation:
type: python
factory: my_package.monitors:create_monitor
normalizer:
type: mapping
mapping:
safe: permit
unsafe: reject
The import target uses package.module:attribute syntax and must be importable in
the runtime environment. None means no verdict, not permit. A monitor may
instead return a raw JSON-compatible native verdict; MonitorResult is useful
when it also has diagnostics, a replacement proposal, or a finality flag.
Remote monitors use the same request semantics over a versioned JSON/HTTP
protocol: Agentic-RV sends agentic-rv.monitor-request/v1, the service returns
the native agentic-rv.monitor-result/v1, and the runtime creates the distinct
normalized/audited agentic-rv.monitor-response/v1. See
Monitor authoring for lifecycle, ordering,
normalization, deadline, replay, and HTTP envelope requirements.
An abstraction returning None has different epsilon semantics: no monitor
request or policy contribution is produced for that event. Subscription plus
projection/abstraction define applicability, so test epsilon paths explicitly.
Connect an agent system
Custom or unsupported framework
Subclass BaseAdapter, emit a decision immediately before the controllable
operation, advertise only reactions you implement, apply the decision, and record
the intervention outcome. Register its class/factory under adapters.custom to
have Deployment inject the runtime and construct it on demand. The
DemoAdapter and runnable
examples/custom_adapter/
form the minimal reference implementation.
from agentic_rv import Reaction
from agentic_rv.adapters import BaseAdapter
from agentic_rv.errors import OperationBlocked
class MyAdapter(BaseAdapter):
adapter_type = "my_framework"
def call_tool(self, tool, **arguments):
event, decision = self.request_decision_sync(
kind="my_framework.tool.call",
payload={"tool": tool.__name__, "arguments": arguments},
supported_reactions=frozenset(
{Reaction.ALLOW, Reaction.LOG, Reaction.BLOCK}
),
)
if decision.reaction is Reaction.BLOCK:
self.record_intervention_sync(
event,
decision,
success=True,
details={"operation_dispatched": False},
)
raise OperationBlocked(
"tool call blocked by runtime policy",
event_id=event.event_id,
details=decision.to_dict(),
)
self.record_intervention_sync(
event,
decision,
success=True,
details={"operation_dispatched": True},
)
return tool(**arguments)
LangGraph
Enable adapters.langgraph, obtain the configured adapter, and wrap a compiled
graph:
with Deployment.load("deployment.yaml") as deployment:
adapter = deployment.langgraph
assert adapter is not None
monitored_graph = adapter.wrap_graph(compiled_graph)
result = monitored_graph.invoke({"messages": messages})
wrap_graph creates pre-effect decision points around supported graph operations
and injects the passive callback handler for internal visibility. For a native
tool callable:
guarded_search = adapter.guard_tool(search, name="search")
If only passive telemetry is wanted, use
config={"callbacks": [adapter.handler]}. A callback cannot prevent an operation.
The LangGraph quickstart and
monitor-pipeline example exercise both
the real compiled-graph proxy and guarded functions.
MCP
Enable adapters.mcp and wrap the client session actually used by the
application:
async with Deployment.load("deployment.yaml") as deployment:
adapter = deployment.mcp
assert adapter is not None
monitored_session = adapter.wrap_session(session)
await monitored_session.initialize()
result = await monitored_session.call_tool("search", {"query": "RV"})
Interposing mode is the default for an enabled MCP adapter. Set mode: passive
to observe without changing control flow.
The MCP example runs an official client and server
over the SDK's in-memory transport.
A2A
Enable adapters.a2a and either wrap the client or pass the adapter's official
duck-typed interceptor when constructing the SDK client:
async with Deployment.load("deployment.yaml") as deployment:
adapter = deployment.a2a
assert adapter is not None
monitored_client = adapter.wrap_client(client)
responses = [response async for response in monitored_client.send_message(request)]
The proxy observes exceptions, cancellation, and stream completion. The official interceptor surface cannot report every transport failure or end-of-stream event, so prefer the proxy when complete lifecycle visibility matters. The A2A example uses the official client factory, protobuf messages, JSON-RPC transport, and streaming response API.
See Adapters for method coverage, event kinds, replacement shapes, context propagation, and integration limitations.
Policy, deadlines, and failures
Synchronous monitors participate in held decisions. Delivered responses from asynchronous monitors and observation events are advisory: they use the same projection, abstraction, normalization, audit, and post-hoc policy path, but do not delay the operation. Projection/abstraction failures are audited; they contribute to policy only when they occur for an applicable synchronous held decision. The policy-and-failures example compares multiple monitors, priority arbitration, fail-open/closed behavior, and an asynchronous advisory handler on the same real LangGraph application.
Built-in arbitration strategies are:
| Strategy | Behavior |
|---|---|
deny_overrides |
A mandatory reject wins; otherwise mandatory failures use the configured failure mode. |
unanimous |
Every applicable mandatory monitor must permit; reject and failure handling still applies. |
priority |
The highest-ranked response decides. Mandatory monitors rank before optional monitors, then higher numeric priority, then monitor ID. |
Mandatory timeout, error, no-verdict, or inconclusive outcomes use:
| Failure mode | Candidate reaction |
|---|---|
closed |
reject_reaction |
open |
permit_reaction |
observe_only |
observe_reaction |
The candidate is then constrained twice: the decisive monitor registrations must
authorize it through allowed_reactions, and the decision event must advertise it
through supported_reactions. If the adapter cannot support it, policy follows a
deterministic fallback order beginning with unsupported_fallback. Therefore,
failure_mode: closed does not by itself promise a physical block at a boundary
that cannot block.
Each decision point has an overall adapter deadline captured before lazy startup.
Startup, sequencing, projection, abstraction, queueing, evaluation,
normalization, policy, and replacement validation all consume it. A monitor may
also have deadline_ms; its queue wait and evaluation consume that budget.
Reaching the overall deadline invokes failure policy even if some monitor
responses arrived.
Timed-out synchronous work may finish later; that late result is audited and
handled as post-hoc advice. The monitor's serial worker waits for it to finish so
stateful monitor order is preserved.
Post-hoc candidates are configured with posthoc_reject_reaction and
posthoc_error_reaction. To give a selected advisory decision an external effect,
set runtime.advisory_handler: package.module:callable; the callable receives
(event, decision) and must implement the application-specific escalation,
notification, or compensation itself.
Replacement proposals are disabled by default. Enabling them requires a decisive monitor proposal, explicit monitor authority, adapter support, and a trusted application-side validator. See the validated-replacement example.
A bounded queue_size protects memory but has a semantic cost: overflow means the
monitor missed an event and is marked desynchronized. Later live deliveries to
that monitor fail until recovery. reset_monitors() clears desynchronization only
when the monitor's reset hook truly reconstructs or rebaselines a valid live
state. Otherwise stop or replace that live runtime; offline replay can evaluate a
complete trace but deliberately cannot resume live traffic. queue_size: 0 is
unbounded.
Full semantics are in Architecture and Configuration.
Traces and offline replay
Enable JSONL audit output explicitly:
runtime:
trace:
enabled: true
path: traces/run-001.jsonl
flush_each_record: true
For file-backed configuration, a relative path is resolved beside the configuration file. Declarative tracing creates a new file exclusively and refuses to overwrite or append to an existing path.
Important: the trace receives the raw canonical event before per-monitor projection, plus normalized/native monitor responses and decisions. A safe monitor projection does not sanitize the trace. Protect the file as sensitive application data; Agentic-RV does not encrypt, sign, rotate, or redact it declaratively.
Replay canonical events through the currently configured monitors:
agentic-rv replay deployment.yaml traces/run-001.jsonl
agentic-rv replay deployment.yaml traces/run-001.jsonl --summary-only
Replay requires a fresh runtime, accepts a strictly increasing canonical sequence
with unique event IDs, optionally calls monitor reset(), and runs only once.
It never invokes adapters or policy and cannot change historical execution. The
CLI disables configured trace output during replay so it does not produce a
partial mixed trace.
The trace/replay example records a real LangGraph run to a collision-safe path, resets a fresh stateful monitor, and proves replay does not rerun the graph or side effect.
See Events and traces for schemas, ordering, serialization, and programmatic replay.
Evaluation and paper artifacts
The repository includes a deterministic quantitative harness under
evaluation/ and a separate
evaluation guide that defines the measurement boundaries,
research questions, raw artifacts, and threats to validity.
Install the reporting dependency and run the quick profile while developing:
python -m pip install -e ".[all,dev,evaluation]"
python -m evaluation.benchmarks run \
--profile quick \
--output evaluation/results/quick
The quick profile is a pipeline smoke check, not publication evidence. A paper
candidate uses the larger paper profile on a quiet, thermally stable machine
and retains the complete output directory:
python -m evaluation.benchmarks run \
--profile paper \
--output evaluation/results/paper-linux-x86_64
python -m evaluation.benchmarks report \
--input evaluation/results/paper-linux-x86_64
Choose a fresh output directory for every run; the harness refuses to overwrite
raw evidence. For the qualitative external-monitor experiment, follow the
case-study README exactly. It
pins both the Ollama model manifest and Reelay outside Agentic-RV, probes a real
schema-constrained generation, verifies the fixed Rye specification in a
subprocess preflight, and keeps malformed native monitor input outside the
agent process. In that isolated environment, the harness also enables paired
reelay.direct and reelay.agentic_rv conditions over the same native monitor
and valuation; without the reviewed Reelay version, those optional conditions
are recorded as unavailable rather than silently substituted.
Documentation map
- Guided tutorial: a LangGraph-first, command-by-command route through the framework's main functionality.
- Evaluation guide: quantitative profiles, measurement boundaries, artifacts, reproducibility controls, and the external Reelay case study.
- Runnable examples: one self-contained folder per integration or feature, with expected output and focused experiments.
- Architecture: component boundaries, ordering, online and offline flows, arbitration, and extension points.
- Monitor authoring: Python and HTTP monitor protocols, lifecycle, state, verdicts, and conformance guidance.
- Configuration: complete deployment reference, defaults, validation, projections, secrets, and normalizers.
- Adapters: LangGraph, MCP, A2A, and custom adapter usage and coverage.
- Events and traces: canonical events, correlation, JSONL audit records, replay, and serialization.
- Security and guarantees: precise assurance boundary, trusted components, data exposure, operational risks, and checklist.
- Paper-to-implementation map: concise traceability between framework concepts, code components, invariants, and defensible claims.
- Machine-readable schemas: event, monitor request, native result, normalized response, and trace JSON Schemas.
Design inspiration
ROSMonitoring demonstrates a useful runtime-verification pattern: declarative instrumentation, thin system-facing mediation, an explicit external-oracle boundary, and formalism-agnostic online/offline monitoring. Agentic-RV takes inspiration from that usability and separation of concerns, but is an independent implementation for agentic AI frameworks and protocols. Its canonical event, monitor, policy, capability-negotiation, adapter, and trace models are specific to this project.
Development
Install all development dependencies, then run the same checks expected of a change:
python -m pip install -e ".[all,dev]"
ruff check src tests examples evaluation
ruff format --check src tests examples evaluation
mypy src/agentic_rv
pytest --cov=agentic_rv --cov-branch
The test suite exercises adapter transparency, sync/async boundaries, policy and failure semantics, configuration errors, lifecycle behavior, projection, serialization, trace replay, and HTTP protocol handling. New adapters should preserve exact call arguments, return values, exceptions, cancellation, and streaming behavior on allow/passive paths.
See CONTRIBUTING.md for contribution constraints. Cite the project using CITATION.cff. Agentic-RV is distributed under the MIT License.