Sovereign 8s.is mirror of standardgalactic/Automatic_Review
  • Python 83.2%
  • Shell 16.8%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
2026-06-16 11:11:16 -04:00
configs Replace legacy scripts with LLM screening pipeline 2026-06-16 11:11:16 -04:00
data/input Replace legacy scripts with LLM screening pipeline 2026-06-16 11:11:16 -04:00
examples Replace legacy scripts with LLM screening pipeline 2026-06-16 11:11:16 -04:00
outputs Replace legacy scripts with LLM screening pipeline 2026-06-16 11:11:16 -04:00
scripts Replace legacy scripts with LLM screening pipeline 2026-06-16 11:11:16 -04:00
src/paper_review Replace legacy scripts with LLM screening pipeline 2026-06-16 11:11:16 -04:00
.env.example Replace legacy scripts with LLM screening pipeline 2026-06-16 11:11:16 -04:00
.gitignore Replace legacy scripts with LLM screening pipeline 2026-06-16 11:11:16 -04:00
environment.yml Replace legacy scripts with LLM screening pipeline 2026-06-16 11:11:16 -04:00
pyproject.toml Replace legacy scripts with LLM screening pipeline 2026-06-16 11:11:16 -04:00
README.md Replace legacy scripts with LLM screening pipeline 2026-06-16 11:11:16 -04:00

Automatic Review

LLM-assisted literature screening with two open-model voters and GPT arbitration for disagreements.

The pipeline was designed for scoping-review screening where each record has metadata such as title, abstract, publication date, venue, and publication type. It runs two open-source instruction models through vLLM, stores each model's vote as JSONL, and optionally asks a GPT model to adjudicate only the records where the two voters disagree.

What Is Included

  • Deterministic publication-date window checking.
  • Prompt templates for open-model screening and GPT arbitration.
  • Persistent vLLM server scripts for faster repeated inference.
  • JSONL outputs for Llama/Qwen votes and GPT arbitration.
  • Token-cost estimation for GPT arbitration.
  • A synthetic sample record for smoke testing.

This repository intentionally does not include model weights, private datasets, runtime logs, local credentials, or screening outputs.

Screening Scope

The default criteria in configs/screening_criteria.yaml target original research on generative large language models in medicine:

  • Publication date from 2023-01-01 through 2025-09-30, inclusive.
  • Transformer-based decoder LLM architecture.
  • Methodological contribution related to pretraining, fine-tuning, prompt engineering, benchmarking, or evaluation.
  • Medical domain, including clinical medicine, basic biomedical research, medical imaging, audio/video, or multi-omics.
  • Original research rather than review, survey, editorial, commentary, or perspective.

Edit the YAML file if your review uses different eligibility criteria.

Installation

Create a Python environment and install the package:

conda create -y -n paper_review python=3.12 pip
conda activate paper_review
python -m pip install -U uv
uv pip install --system -e ".[api]"

Install vLLM for your CUDA/PyTorch stack. One example for CUDA 12.9:

uv pip install \
  --system \
  --reinstall "https://github.com/vllm-project/vllm/releases/download/v0.22.1/vllm-0.22.1%2Bcu129-cp38-abi3-manylinux_2_28_x86_64.whl" \
  --torch-backend=cu129 \
  --extra-index-url https://download.pytorch.org/whl/cu129 \
  --index-strategy unsafe-best-match

You can also adapt scripts/install_dependencies.sh to your cluster.

Configuration

Copy the environment template and fill in only the credentials you need:

cp .env.example .env
chmod 600 .env

The default open-model voters are configured in configs/models.yaml:

  • meta-llama/Llama-3.1-8B-Instruct
  • Qwen/Qwen3-8B

By default, model weights are downloaded under models/, which is ignored by git. You may edit local_dir, GPU IDs, ports, and vLLM settings in configs/models.yaml.

Download the configured models:

scripts/download_models.sh --model all

Input Data

The pipeline accepts CSV, Excel, JSON, and JSONL files. Recommended columns:

  • record_id
  • title
  • abstract
  • full_text
  • publication_date
  • year
  • source
  • venue
  • publication_type
  • doi
  • citation_count
  • url

Common alternative names such as pmid, paper_title, article_type, journal, and times_cited are also recognized.

Quick Smoke Test

Run a small offline vLLM smoke test against the synthetic example:

scripts/run_voters_in_screen.sh examples/sample_records.jsonl --limit 1 --batch-size 1

Outputs are written to outputs/votes_llama.jsonl and outputs/votes_qwen.jsonl.

Persistent vLLM Workflow

For larger screening jobs, run each model through a persistent vLLM server so model loading and CUDA warmup happen only once.

Start one voter server:

scripts/start_vllm_server.sh qwen
scripts/wait_vllm_server.sh qwen

Run the matching voter:

scripts/run_voter_against_server.sh \
  qwen \
  data/input/records.csv \
  --max-tokens 512

Stop the server:

scripts/stop_vllm_server.sh qwen

Repeat for llama, or start both servers on separate GPUs if your hardware allows it.

GPT Arbitration

After both voter files are available, arbitrate only the disagreements:

ENGINE=gpt-5 scripts/run_arbitration.sh \
  --llama-votes outputs/votes_llama.jsonl \
  --qwen-votes outputs/votes_qwen.jsonl \
  --records data/input/records.csv \
  --output outputs/arbitration_gpt.jsonl \
  --api-mode auto \
  --join-key row_index \
  --vote-mode compact \
  --concurrency 4 \
  --max-tokens 768

The GPT client supports either:

  • Enterprise gateway mode through CLIENT_ID, CLIENT_SECRET, APIGEE_BASE, ENGINE, and API_VERSION.
  • Direct OpenAI mode through OPENAI_API_KEY and OPENAI_MODEL.

Use row_index for joins when your data may contain duplicate record IDs.

Estimate GPT Token Cost

Before running arbitration, estimate token usage:

scripts/estimate_gpt_arbitration_tokens.py \
  --llama-votes outputs/votes_llama.jsonl \
  --qwen-votes outputs/votes_qwen.jsonl \
  --records data/input/records.csv \
  --vote-mode compact

Use --input-price-per-1m and --output-price-per-1m to include estimated costs for a specific GPT model.

Date Window Audit

The date criterion is assessed deterministically before prompting the models:

scripts/check_date_window.py --input data/input/records.csv --output outputs/date_window.jsonl

The computed date assessment is included in each model prompt and is also used to correct the date_in_scope field in parsed voter outputs.

Repository Hygiene

Do not commit:

  • .env or any credentials.
  • model weights under models/.
  • input datasets under data/.
  • outputs under outputs/.
  • runtime logs under logs/.

The .gitignore file is configured for these defaults.