- Python 100%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
| examples | ||
| src/recite | ||
| tests | ||
| .env.example | ||
| .gitignore | ||
| LICENSE | ||
| pyproject.toml | ||
| README.md | ||
| uv.lock | ||
ReCITE: Real-world CausalIty from Textual Evidence Benchmark
ReCITE (Real-world CausalIty from Textual Evidence) is a benchmark for evaluating LLM causal reasoning on real-world academic text.
Preprint: Can Large Language Models Infer Causal Relationships from Real-World Text?
Overview
ReCITE evaluates whether LLMs can infer causal relationships from authentic academic texts. Unlike synthetic benchmarks with explicit causal statements, ReCITE uses real papers where causal relationships are often implicit, requiring genuine reasoning to extract.
Key findings (from the study):
- Best model (Claude Opus 4.5) achieves only 0.535 F1 (LLM-as-a-Judge, weighted)
- Performance degrades significantly as relationships become less explicit
- Even with ground-truth node names provided, causal reasoning remains the primary bottleneck
Installation
git clone https://github.com/Ryan-Saklad/ReCITE.git
cd ReCITE
uv pip install -e .
Analysis extras (plots + OpenAlex tooling):
uv pip install -e ".[analysis]"
Development + tests:
uv pip install -e ".[dev]"
uv run -m pytest
For API-based benchmarking, set your OpenRouter API key:
export OPENROUTER_API_KEY=your_key_here
Optional defaults (DB + results source + models):
export RECITE_DB_PATH=recite_results.db
export RECITE_RESULTS_SOURCE=recite_results.db # or "hf"
export RECITE_OUTPUT_DIR=output
export RECITE_VERBOSE=0 # set to 1 for verbose logs
export RECITE_JUDGE_MODEL=deepseek/deepseek-v3.2
export RECITE_CORRECTION_MODEL=openai/gpt-5-mini
export RECITE_DEFAULT_MODEL=qwen/qwq-32b
export RECITE_ABLATION_MODEL=meta-llama/llama-3.1-8b-instruct
Quick Start
from recite import load_dataset
# Load the benchmark dataset from HuggingFace
dataset = load_dataset()
print(f"Loaded {len(dataset)} samples") # 292 samples
# Access a sample
sample = dataset[0]
print(f"Title: {sample.title}")
print(f"Nodes: {sample.num_nodes}, Edges: {sample.num_edges}")
print(f"Explicitness: {sample.explicitness:.0%}")
# Filter by domain
medical = dataset.filter_by_domain("Medicine")
# Filter by difficulty (lower explicitness = harder)
hard_samples = dataset.filter_by_max_explicitness(0.3)
Running Your Model
from recite import load_dataset, load_prompt, calculate_graph_metrics
dataset = load_dataset()
prompt_template = load_prompt("causal_graph_generation")
for sample in dataset:
# Prepare prompt
prompt = prompt_template.replace("NUM_NODES", str(sample.num_nodes))
# Call your model
response = your_model(system=prompt, user=sample.input_text)
# Parse response (should be JSON with "relationships" key)
predicted_edges = parse_response(response)
# Evaluate
precision, recall, f1, shd, norm_shd = calculate_graph_metrics(
sample.edges,
predicted_edges
)
print(f"Sample {sample.id}: F1={f1:.3f}, SHD={shd:.0f}")
See examples/ for complete working examples.
CLI Usage
Run the full benchmark on any model via OpenRouter:
# Run benchmark on all 292 samples
recite run qwen/qwq-32b
# Run with node names provided (easier mode)
recite run qwen/qwq-32b --with-nodes
# Run on specific samples
recite run qwen/qwq-32b --samples 1,2,3,4,5
# Run multiple models concurrently
recite run --models qwen/qwq-32b,openai/gpt-5.2 --max-model-concurrent 2
# Overwrite existing responses
recite run qwen/qwq-32b --overwrite
# View your local results (deterministic evaluation)
recite results
# Export results to JSON
recite export -o results.json
Filtering quick refs:
- Samples:
--samples 1,2,3 - Domains:
--domain "Medicine"or--domains "Medicine,Environment"(+--domains-all) - Sources:
--source PLOS - Date range:
--earliest-date YYYY-MM-DD/--latest-date YYYY-MM-DD - Evaluate specific generation models:
--include-models .../--exclude-models ... - Overwrite existing outputs:
--overwrite
Analysis quick refs (default output root: output, override with --out-dir or RECITE_OUTPUT_DIR):
- Explicitness:
recite analyze explicitness - Topology:
recite analyze topology --db recite_results.db - Diversity + clustering:
recite analyze diversitythenrecite analyze cluster-performance - Report artifacts (tables/figures):
recite analyze paper-artifacts --db recite_results.db --out-dir output/paper_artifactsNote: some analysis commands require optional dependencies (e.g.,matplotlib,scipy,sentence-transformers,umap-learn,hdbscan,beautifulsoup4).
Viewing Published Results
The HuggingFace dataset includes pre-computed model responses and evaluations from the study. View them without running any models:
# View LLM-as-a-Judge results (Table 3)
recite results --source hf --fine-grained
# View deterministic results (Table 4 - name-assisted)
recite results --source hf --deterministic --with-nodes
Example LLM-as-a-Judge output:
╭──────────────────────────────┬──────────────┬─────────────┬──────────────┬─────────────┬──────────────┬───────────┬─────────────┬─────╮
│ Model │ Node Prec │ Node Rec │ Edge Prec │ Edge Rec │ F1 │ SHD │ Norm SHD │ N │
├──────────────────────────────┼──────────────┼─────────────┼──────────────┼─────────────┼──────────────┼───────────┼─────────────┼─────┤
│ Claude Opus 4.5 │ 0.972±0.055 │ 0.571±0.215 │ 0.853±0.154 │ 0.305±0.208 │ 0.535±0.202 │ 47.0±36.5 │ 0.108±0.091 │ 292 │
│ GPT 5.2 │ 0.960±0.070 │ 0.524±0.214 │ 0.804±0.160 │ 0.309±0.205 │ 0.510±0.201 │ 58.5±45.1 │ 0.130±0.094 │ 292 │
│ Gemini 3 Pro │ 0.980±0.050 │ 0.527±0.232 │ 0.865±0.152 │ 0.281±0.215 │ 0.501±0.221 │ 48.3±37.2 │ 0.114±0.101 │ 292 │
╰──────────────────────────────┴──────────────┴─────────────┴──────────────┴─────────────┴──────────────┴───────────┴─────────────┴─────╯
To cache the published results locally and make the DB the default source:
recite import-hf --db recite_results.db
export RECITE_RESULTS_SOURCE=recite_results.db
Dataset
The dataset is hosted on HuggingFace: RyanSaklad/ReCITE
| Statistic | Value |
|---|---|
| Samples | 292 |
| Text length | 684–171,141 characters (mean: 40,541) |
| Nodes per graph | 5–140 (mean: 25.0) |
| Edges per graph | 6–205 (mean: 37.4) |
| Explicitness | 0.877 (proportion of nodes mentioned in text) |
| Sources | MDPI, PLOS (open-access) |
| Fields | 17 OpenAlex fields (broad STEM + social science mix) |
Each sample includes:
- input_text: Markdown text from an academic paper
- nodes: Ground truth causal variables
- edges: Ground truth directed causal relationships
- explicitness: Proportion of nodes mentioned in text (explicit or implicit; higher = easier)
- node_explicitness: Per-node explicitness scores (1=explicit, 2=implicit, 3=absent)
Benchmark Results
The benchmark reports two evaluation settings:
LLM-as-a-Judge (Table 3) — fine-grained evaluation against both source text and ground-truth graphs using DeepSeek v3.2 as judge (mean ± std).
| Model | Node Prec | Node Rec | Edge Prec | Edge Rec | F1 | SHD | Norm SHD |
|---|---|---|---|---|---|---|---|
| Claude Opus 4.5 | 0.972±0.055 | 0.571±0.215 | 0.853±0.154 | 0.305±0.208 | 0.535±0.202 | 47.0±36.5 | 0.108±0.091 |
| GPT 5.2 | 0.960±0.070 | 0.524±0.214 | 0.804±0.160 | 0.309±0.205 | 0.510±0.201 | 58.5±45.1 | 0.130±0.094 |
| Gemini 3 Pro | 0.980±0.050 | 0.527±0.232 | 0.865±0.152 | 0.281±0.215 | 0.501±0.221 | 48.3±37.2 | 0.114±0.101 |
| GLM 4.7 | 0.970±0.085 | 0.524±0.226 | 0.808±0.169 | 0.274±0.199 | 0.493±0.210 | 50.8±37.8 | 0.116±0.089 |
| Kimi K2 | 0.950±0.079 | 0.508±0.221 | 0.778±0.177 | 0.253±0.189 | 0.471±0.197 | 54.7±44.6 | 0.126±0.100 |
| Gemini 3 Flash | 0.962±0.096 | 0.501±0.234 | 0.818±0.164 | 0.255±0.210 | 0.469±0.218 | 50.2±36.4 | 0.119±0.098 |
| DeepSeek R1 | 0.955±0.086 | 0.504±0.204 | 0.773±0.181 | 0.231±0.182 | 0.460±0.186 | 50.9±36.0 | 0.122±0.100 |
| QwQ 32B | 0.944±0.088 | 0.481±0.209 | 0.730±0.221 | 0.209±0.167 | 0.432±0.183 | 51.0±37.1 | 0.117±0.090 |
| Qwen 2.5 32B | 0.925±0.109 | 0.431±0.219 | 0.686±0.222 | 0.180±0.161 | 0.386±0.192 | 49.9±31.6 | 0.107±0.085 |
| Llama 3.1 8B | 0.916±0.156 | 0.341±0.205 | 0.615±0.256 | 0.111±0.130 | 0.295±0.171 | 51.8±30.9 | 0.103±0.080 |
Name-assisted deterministic (Table 4) — models are given ground-truth node names; evaluates edge prediction only.
| Model | Precision | Recall | F1 | SHD |
|---|---|---|---|---|
| Claude Opus 4.5 | 0.582 | 0.536 | 0.551 | 35.6 |
| Gemini 3 Pro | 0.567 | 0.531 | 0.538 | 35.8 |
| Gemini 3 Flash | 0.552 | 0.526 | 0.528 | 37.8 |
| GPT 5.2 | 0.537 | 0.521 | 0.520 | 42.6 |
| Kimi K2 | 0.515 | 0.513 | 0.505 | 45.2 |
| GLM 4.7 | 0.517 | 0.514 | 0.505 | 41.0 |
| DeepSeek R1 | 0.513 | 0.512 | 0.502 | 40.6 |
| QwQ 32B | 0.483 | 0.501 | 0.481 | 41.0 |
| Qwen 2.5 32B | 0.290 | 0.435 | 0.332 | 51.0 |
| Llama 3.1 8B | 0.218 | 0.409 | 0.266 | 62.9 |
Project Structure
ReCITE/
├── src/recite/
│ ├── dataset.py # HuggingFace dataset loader
│ ├── helpers.py # Metrics (calculate_graph_metrics, etc.)
│ ├── benchmark.py # Benchmark runner (OpenRouter API)
│ ├── evaluator.py # LLM-as-judge evaluation
│ └── cli.py # Command-line interface
├── src/recite/prompts/ # Prompt templates used to build the dataset
└── tests/
Citation
@misc{saklad2026largelanguagemodelsinfer,
title={Can Large Language Models Infer Causal Relationships from Real-World Text?},
author={Ryan Saklad and Aman Chadha and Oleg Pavlov and Raha Moraffah},
year={2026},
eprint={2505.18931},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2505.18931},
}
License
This work is licensed under CC-BY 4.0.