- Python 83.2%
- Shell 16.8%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
| configs | ||
| data/input | ||
| examples | ||
| outputs | ||
| scripts | ||
| src/paper_review | ||
| .env.example | ||
| .gitignore | ||
| environment.yml | ||
| pyproject.toml | ||
| README.md | ||
Automatic Review
LLM-assisted literature screening with two open-model voters and GPT arbitration for disagreements.
The pipeline was designed for scoping-review screening where each record has metadata such as title, abstract, publication date, venue, and publication type. It runs two open-source instruction models through vLLM, stores each model's vote as JSONL, and optionally asks a GPT model to adjudicate only the records where the two voters disagree.
What Is Included
- Deterministic publication-date window checking.
- Prompt templates for open-model screening and GPT arbitration.
- Persistent vLLM server scripts for faster repeated inference.
- JSONL outputs for Llama/Qwen votes and GPT arbitration.
- Token-cost estimation for GPT arbitration.
- A synthetic sample record for smoke testing.
This repository intentionally does not include model weights, private datasets, runtime logs, local credentials, or screening outputs.
Screening Scope
The default criteria in configs/screening_criteria.yaml target original
research on generative large language models in medicine:
- Publication date from
2023-01-01through2025-09-30, inclusive. - Transformer-based decoder LLM architecture.
- Methodological contribution related to pretraining, fine-tuning, prompt engineering, benchmarking, or evaluation.
- Medical domain, including clinical medicine, basic biomedical research, medical imaging, audio/video, or multi-omics.
- Original research rather than review, survey, editorial, commentary, or perspective.
Edit the YAML file if your review uses different eligibility criteria.
Installation
Create a Python environment and install the package:
conda create -y -n paper_review python=3.12 pip
conda activate paper_review
python -m pip install -U uv
uv pip install --system -e ".[api]"
Install vLLM for your CUDA/PyTorch stack. One example for CUDA 12.9:
uv pip install \
--system \
--reinstall "https://github.com/vllm-project/vllm/releases/download/v0.22.1/vllm-0.22.1%2Bcu129-cp38-abi3-manylinux_2_28_x86_64.whl" \
--torch-backend=cu129 \
--extra-index-url https://download.pytorch.org/whl/cu129 \
--index-strategy unsafe-best-match
You can also adapt scripts/install_dependencies.sh to your cluster.
Configuration
Copy the environment template and fill in only the credentials you need:
cp .env.example .env
chmod 600 .env
The default open-model voters are configured in configs/models.yaml:
meta-llama/Llama-3.1-8B-InstructQwen/Qwen3-8B
By default, model weights are downloaded under models/, which is ignored by
git. You may edit local_dir, GPU IDs, ports, and vLLM settings in
configs/models.yaml.
Download the configured models:
scripts/download_models.sh --model all
Input Data
The pipeline accepts CSV, Excel, JSON, and JSONL files. Recommended columns:
record_idtitleabstractfull_textpublication_dateyearsourcevenuepublication_typedoicitation_counturl
Common alternative names such as pmid, paper_title, article_type,
journal, and times_cited are also recognized.
Quick Smoke Test
Run a small offline vLLM smoke test against the synthetic example:
scripts/run_voters_in_screen.sh examples/sample_records.jsonl --limit 1 --batch-size 1
Outputs are written to outputs/votes_llama.jsonl and
outputs/votes_qwen.jsonl.
Persistent vLLM Workflow
For larger screening jobs, run each model through a persistent vLLM server so model loading and CUDA warmup happen only once.
Start one voter server:
scripts/start_vllm_server.sh qwen
scripts/wait_vllm_server.sh qwen
Run the matching voter:
scripts/run_voter_against_server.sh \
qwen \
data/input/records.csv \
--max-tokens 512
Stop the server:
scripts/stop_vllm_server.sh qwen
Repeat for llama, or start both servers on separate GPUs if your hardware
allows it.
GPT Arbitration
After both voter files are available, arbitrate only the disagreements:
ENGINE=gpt-5 scripts/run_arbitration.sh \
--llama-votes outputs/votes_llama.jsonl \
--qwen-votes outputs/votes_qwen.jsonl \
--records data/input/records.csv \
--output outputs/arbitration_gpt.jsonl \
--api-mode auto \
--join-key row_index \
--vote-mode compact \
--concurrency 4 \
--max-tokens 768
The GPT client supports either:
- Enterprise gateway mode through
CLIENT_ID,CLIENT_SECRET,APIGEE_BASE,ENGINE, andAPI_VERSION. - Direct OpenAI mode through
OPENAI_API_KEYandOPENAI_MODEL.
Use row_index for joins when your data may contain duplicate record IDs.
Estimate GPT Token Cost
Before running arbitration, estimate token usage:
scripts/estimate_gpt_arbitration_tokens.py \
--llama-votes outputs/votes_llama.jsonl \
--qwen-votes outputs/votes_qwen.jsonl \
--records data/input/records.csv \
--vote-mode compact
Use --input-price-per-1m and --output-price-per-1m to include estimated
costs for a specific GPT model.
Date Window Audit
The date criterion is assessed deterministically before prompting the models:
scripts/check_date_window.py --input data/input/records.csv --output outputs/date_window.jsonl
The computed date assessment is included in each model prompt and is also used
to correct the date_in_scope field in parsed voter outputs.
Repository Hygiene
Do not commit:
.envor any credentials.- model weights under
models/. - input datasets under
data/. - outputs under
outputs/. - runtime logs under
logs/.
The .gitignore file is configured for these defaults.