Sovereign 8s.is mirror of standardgalactic/ReCITE
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
2026-01-25 13:30:59 -05:00
examples Prepare ReCITE for release 2026-01-25 13:30:59 -05:00
src/recite Prepare ReCITE for release 2026-01-25 13:30:59 -05:00
tests Prepare ReCITE for release 2026-01-25 13:30:59 -05:00
.env.example Prepare ReCITE for release 2026-01-25 13:30:59 -05:00
.gitignore Prepare ReCITE for release 2026-01-25 13:30:59 -05:00
LICENSE Initial commit 2025-12-09 15:27:00 -05:00
pyproject.toml Prepare ReCITE for release 2026-01-25 13:30:59 -05:00
README.md Prepare ReCITE for release 2026-01-25 13:30:59 -05:00
uv.lock Prepare ReCITE for release 2026-01-25 13:30:59 -05:00

ReCITE: Real-world CausalIty from Textual Evidence Benchmark

ReCITE (Real-world CausalIty from Textual Evidence) is a benchmark for evaluating LLM causal reasoning on real-world academic text.

Preprint: Can Large Language Models Infer Causal Relationships from Real-World Text?

Overview

ReCITE evaluates whether LLMs can infer causal relationships from authentic academic texts. Unlike synthetic benchmarks with explicit causal statements, ReCITE uses real papers where causal relationships are often implicit, requiring genuine reasoning to extract.

Key findings (from the study):

  • Best model (Claude Opus 4.5) achieves only 0.535 F1 (LLM-as-a-Judge, weighted)
  • Performance degrades significantly as relationships become less explicit
  • Even with ground-truth node names provided, causal reasoning remains the primary bottleneck

Installation

git clone https://github.com/Ryan-Saklad/ReCITE.git
cd ReCITE
uv pip install -e .

Analysis extras (plots + OpenAlex tooling):

uv pip install -e ".[analysis]"

Development + tests:

uv pip install -e ".[dev]"
uv run -m pytest

For API-based benchmarking, set your OpenRouter API key:

export OPENROUTER_API_KEY=your_key_here

Optional defaults (DB + results source + models):

export RECITE_DB_PATH=recite_results.db
export RECITE_RESULTS_SOURCE=recite_results.db  # or "hf"
export RECITE_OUTPUT_DIR=output
export RECITE_VERBOSE=0  # set to 1 for verbose logs
export RECITE_JUDGE_MODEL=deepseek/deepseek-v3.2
export RECITE_CORRECTION_MODEL=openai/gpt-5-mini
export RECITE_DEFAULT_MODEL=qwen/qwq-32b
export RECITE_ABLATION_MODEL=meta-llama/llama-3.1-8b-instruct

Quick Start

from recite import load_dataset

# Load the benchmark dataset from HuggingFace
dataset = load_dataset()
print(f"Loaded {len(dataset)} samples")  # 292 samples

# Access a sample
sample = dataset[0]
print(f"Title: {sample.title}")
print(f"Nodes: {sample.num_nodes}, Edges: {sample.num_edges}")
print(f"Explicitness: {sample.explicitness:.0%}")

# Filter by domain
medical = dataset.filter_by_domain("Medicine")

# Filter by difficulty (lower explicitness = harder)
hard_samples = dataset.filter_by_max_explicitness(0.3)

Running Your Model

from recite import load_dataset, load_prompt, calculate_graph_metrics

dataset = load_dataset()
prompt_template = load_prompt("causal_graph_generation")

for sample in dataset:
    # Prepare prompt
    prompt = prompt_template.replace("NUM_NODES", str(sample.num_nodes))

    # Call your model
    response = your_model(system=prompt, user=sample.input_text)

    # Parse response (should be JSON with "relationships" key)
    predicted_edges = parse_response(response)

    # Evaluate
    precision, recall, f1, shd, norm_shd = calculate_graph_metrics(
        sample.edges,
        predicted_edges
    )
    print(f"Sample {sample.id}: F1={f1:.3f}, SHD={shd:.0f}")

See examples/ for complete working examples.

CLI Usage

Run the full benchmark on any model via OpenRouter:

# Run benchmark on all 292 samples
recite run qwen/qwq-32b

# Run with node names provided (easier mode)
recite run qwen/qwq-32b --with-nodes

# Run on specific samples
recite run qwen/qwq-32b --samples 1,2,3,4,5

# Run multiple models concurrently
recite run --models qwen/qwq-32b,openai/gpt-5.2 --max-model-concurrent 2

# Overwrite existing responses
recite run qwen/qwq-32b --overwrite

# View your local results (deterministic evaluation)
recite results

# Export results to JSON
recite export -o results.json

Filtering quick refs:

  • Samples: --samples 1,2,3
  • Domains: --domain "Medicine" or --domains "Medicine,Environment" (+ --domains-all)
  • Sources: --source PLOS
  • Date range: --earliest-date YYYY-MM-DD / --latest-date YYYY-MM-DD
  • Evaluate specific generation models: --include-models ... / --exclude-models ...
  • Overwrite existing outputs: --overwrite

Analysis quick refs (default output root: output, override with --out-dir or RECITE_OUTPUT_DIR):

  • Explicitness: recite analyze explicitness
  • Topology: recite analyze topology --db recite_results.db
  • Diversity + clustering: recite analyze diversity then recite analyze cluster-performance
  • Report artifacts (tables/figures): recite analyze paper-artifacts --db recite_results.db --out-dir output/paper_artifacts Note: some analysis commands require optional dependencies (e.g., matplotlib, scipy, sentence-transformers, umap-learn, hdbscan, beautifulsoup4).

Viewing Published Results

The HuggingFace dataset includes pre-computed model responses and evaluations from the study. View them without running any models:

# View LLM-as-a-Judge results (Table 3)
recite results --source hf --fine-grained

# View deterministic results (Table 4 - name-assisted)
recite results --source hf --deterministic --with-nodes

Example LLM-as-a-Judge output:

╭──────────────────────────────┬──────────────┬─────────────┬──────────────┬─────────────┬──────────────┬───────────┬─────────────┬─────╮
│ Model                        │ Node Prec    │ Node Rec    │ Edge Prec    │ Edge Rec    │ F1           │ SHD       │ Norm SHD    │ N   │
├──────────────────────────────┼──────────────┼─────────────┼──────────────┼─────────────┼──────────────┼───────────┼─────────────┼─────┤
│ Claude Opus 4.5              │ 0.972±0.055  │ 0.571±0.215 │ 0.853±0.154  │ 0.305±0.208 │ 0.535±0.202  │ 47.0±36.5 │ 0.108±0.091 │ 292 │
│ GPT 5.2                      │ 0.960±0.070  │ 0.524±0.214 │ 0.804±0.160  │ 0.309±0.205 │ 0.510±0.201  │ 58.5±45.1 │ 0.130±0.094 │ 292 │
│ Gemini 3 Pro                 │ 0.980±0.050  │ 0.527±0.232 │ 0.865±0.152  │ 0.281±0.215 │ 0.501±0.221  │ 48.3±37.2 │ 0.114±0.101 │ 292 │
╰──────────────────────────────┴──────────────┴─────────────┴──────────────┴─────────────┴──────────────┴───────────┴─────────────┴─────╯

To cache the published results locally and make the DB the default source:

recite import-hf --db recite_results.db
export RECITE_RESULTS_SOURCE=recite_results.db

Dataset

The dataset is hosted on HuggingFace: RyanSaklad/ReCITE

Statistic Value
Samples 292
Text length 684–171,141 characters (mean: 40,541)
Nodes per graph 5–140 (mean: 25.0)
Edges per graph 6–205 (mean: 37.4)
Explicitness 0.877 (proportion of nodes mentioned in text)
Sources MDPI, PLOS (open-access)
Fields 17 OpenAlex fields (broad STEM + social science mix)

Each sample includes:

  • input_text: Markdown text from an academic paper
  • nodes: Ground truth causal variables
  • edges: Ground truth directed causal relationships
  • explicitness: Proportion of nodes mentioned in text (explicit or implicit; higher = easier)
  • node_explicitness: Per-node explicitness scores (1=explicit, 2=implicit, 3=absent)

Benchmark Results

The benchmark reports two evaluation settings:

LLM-as-a-Judge (Table 3) — fine-grained evaluation against both source text and ground-truth graphs using DeepSeek v3.2 as judge (mean ± std).

Model Node Prec Node Rec Edge Prec Edge Rec F1 SHD Norm SHD
Claude Opus 4.5 0.972±0.055 0.571±0.215 0.853±0.154 0.305±0.208 0.535±0.202 47.0±36.5 0.108±0.091
GPT 5.2 0.960±0.070 0.524±0.214 0.804±0.160 0.309±0.205 0.510±0.201 58.5±45.1 0.130±0.094
Gemini 3 Pro 0.980±0.050 0.527±0.232 0.865±0.152 0.281±0.215 0.501±0.221 48.3±37.2 0.114±0.101
GLM 4.7 0.970±0.085 0.524±0.226 0.808±0.169 0.274±0.199 0.493±0.210 50.8±37.8 0.116±0.089
Kimi K2 0.950±0.079 0.508±0.221 0.778±0.177 0.253±0.189 0.471±0.197 54.7±44.6 0.126±0.100
Gemini 3 Flash 0.962±0.096 0.501±0.234 0.818±0.164 0.255±0.210 0.469±0.218 50.2±36.4 0.119±0.098
DeepSeek R1 0.955±0.086 0.504±0.204 0.773±0.181 0.231±0.182 0.460±0.186 50.9±36.0 0.122±0.100
QwQ 32B 0.944±0.088 0.481±0.209 0.730±0.221 0.209±0.167 0.432±0.183 51.0±37.1 0.117±0.090
Qwen 2.5 32B 0.925±0.109 0.431±0.219 0.686±0.222 0.180±0.161 0.386±0.192 49.9±31.6 0.107±0.085
Llama 3.1 8B 0.916±0.156 0.341±0.205 0.615±0.256 0.111±0.130 0.295±0.171 51.8±30.9 0.103±0.080

Name-assisted deterministic (Table 4) — models are given ground-truth node names; evaluates edge prediction only.

Model Precision Recall F1 SHD
Claude Opus 4.5 0.582 0.536 0.551 35.6
Gemini 3 Pro 0.567 0.531 0.538 35.8
Gemini 3 Flash 0.552 0.526 0.528 37.8
GPT 5.2 0.537 0.521 0.520 42.6
Kimi K2 0.515 0.513 0.505 45.2
GLM 4.7 0.517 0.514 0.505 41.0
DeepSeek R1 0.513 0.512 0.502 40.6
QwQ 32B 0.483 0.501 0.481 41.0
Qwen 2.5 32B 0.290 0.435 0.332 51.0
Llama 3.1 8B 0.218 0.409 0.266 62.9

Project Structure

ReCITE/
├── src/recite/
│   ├── dataset.py      # HuggingFace dataset loader
│   ├── helpers.py      # Metrics (calculate_graph_metrics, etc.)
│   ├── benchmark.py    # Benchmark runner (OpenRouter API)
│   ├── evaluator.py    # LLM-as-judge evaluation
│   └── cli.py          # Command-line interface
├── src/recite/prompts/ # Prompt templates used to build the dataset
└── tests/

Citation

@misc{saklad2026largelanguagemodelsinfer,
      title={Can Large Language Models Infer Causal Relationships from Real-World Text?},
      author={Ryan Saklad and Aman Chadha and Oleg Pavlov and Raha Moraffah},
      year={2026},
      eprint={2505.18931},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2505.18931},
}

License

This work is licensed under CC-BY 4.0.