Sovereign 8s.is mirror of standardgalactic/RAG-based-Architectures-for-Drug-Side-Effect-Retrieval-using-compact-LLMs
  • Python 99.6%
  • Shell 0.4%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
2025-11-29 20:54:59 +03:00
data/processed Cleanup codebase for November 2025 experiments: remove unused files, clean code, update README with results 2025-11-29 20:49:29 +03:00
experiments Cleanup codebase for November 2025 experiments: remove unused files, clean code, update README with results 2025-11-29 20:49:29 +03:00
results Cleanup codebase for November 2025 experiments: remove unused files, clean code, update README with results 2025-11-29 20:49:29 +03:00
scripts Cleanup codebase for November 2025 experiments: remove unused files, clean code, update README with results 2025-11-29 20:49:29 +03:00
src Cleanup codebase for November 2025 experiments: remove unused files, clean code, update README with results 2025-11-29 20:49:29 +03:00
.gitignore Add Jupyter notebooks 2025-11-29 19:33:41 +03:00
.python-version Clean DrugRAG implementation with evaluation results 2025-09-20 21:16:51 +03:00
pyproject.toml Initial commit - DrugRAG project 2025-11-29 18:05:40 +03:00
README.md Update README.md 2025-11-29 20:54:59 +03:00
run_benchmark.sh Cleanup codebase for November 2025 experiments: remove unused files, clean code, update README with results 2025-11-29 20:49:29 +03:00
uv.lock Initial commit - DrugRAG project 2025-11-29 18:05:40 +03:00

DrugRAG: Drug Side Effect Retrieval System

A production-ready RAG system for identifying drug side effects using multiple retrieval architectures. This repository contains the November 2025 experiments evaluating Pure LLM (baseline), GraphRAG, Format A RAG, and Format B RAG on reverse queries, binary classification, and misspelling robustness.

Key Features

  • Pure LLM Baseline: Direct LLM inference without retrieval augmentation for baseline comparison
  • GraphRAG: Graph-based retrieval using Neo4j for structured drug-side effect relationships
  • Format A RAG: Vector-based retrieval with Pinecone for semantic search
  • Format B RAG: Chunked retrieval strategy optimized for large result sets
  • vLLM Integration: Ultra-fast inference using tensor parallelism (Qwen and Llama3 models)
  • Misspelling Robustness: Evaluation of system performance with misspelled drug names
  • Spell Correction Recovery: Assessment of spell correction impact on retrieval accuracy

Installation

Prerequisites

  • Python 3.9+
  • Neo4j (for GraphRAG)
  • vLLM server (for inference)
  • Pinecone account (for vector storage)
  • OpenAI API key (for embeddings)

Install Dependencies

Using pip:

pip install -e .

Using uv (recommended):

uv sync

Configuration

Create a config.json file in the experiments/ directory:

{
  "neo4j_uri": "bolt://localhost:7687",
  "neo4j_user": "neo4j",
  "neo4j_password": "your_password",
  "pinecone_api_key": "your_pinecone_key",
  "pinecone_index_name": "drug-side-effects",
  "openai_api_key": "your_openai_key"
}

Usage

1. Reverse Query Benchmark

Evaluates the system's ability to list all drugs causing a specific side effect.

./run_benchmark.sh reverse

2. Binary Query Benchmark (vLLM)

Evaluates "Does Drug X cause Side Effect Y?" using vLLM for fast inference.

# GraphRAG with Qwen
./run_benchmark.sh binary --architecture graphrag_qwen --test_size 100

# Format B with Llama3
./run_benchmark.sh binary --architecture format_b_llama3 --test_size 100

# Pure LLM baseline
./run_benchmark.sh binary --architecture pure_llm_qwen --test_size 100

3. Misspelling Evaluation

Tests robustness against misspelled drug names.

./run_benchmark.sh misspelling

November 2025 Experiment Results

Binary Classification Performance (19,520 Queries)

Performance on binary queries ("Is [side effect] an adverse effect of [drug]?") using the full evaluation dataset.

Source: experiments/results_vllm_*_19520_*.json files (Nov 25-28, 2025)

Architecture Model Accuracy F1 Score Precision Sensitivity Specificity
GraphRAG Qwen 99.95% 0.999 0.999 0.999 0.999
GraphRAG Llama3 99.96% 1.000 0.999 1.000 0.999
Format B Qwen 99.71% 0.997 0.997 0.997 0.997
Format B Llama3 98.36% 0.983 0.998 0.970 0.998
Format A Qwen 90.86% 0.900 0.997 0.820 0.997
Format A Llama3 86.58% 0.845 0.998 0.733 0.998
Pure LLM Qwen 62.93% 0.494 0.777 0.363 0.896
Pure LLM Llama3 63.19% 0.535 0.726 0.423 0.840

Key Findings:

  • GraphRAG achieves near-perfect accuracy with both models (99.95% Qwen, 99.96% Llama3)
  • Format B achieves excellent performance (99.71% Qwen, 98.36% Llama3)
  • Format A shows good accuracy (90.86% Qwen, 86.58% Llama3)
  • Pure LLM baseline performs at 62-63% accuracy, demonstrating the critical need for retrieval augmentation
  • RAG architectures provide 37-38% absolute improvement over Pure LLM baseline

Reverse Query Performance

Comparison of architectures on the reverse query task (side effect → list of drugs).

Source: experiments/results_reverse_query_benchmark_20251128_235601.json

Architecture Avg Recall Avg Precision Avg F1 Avg Latency Queries
GraphRAG 100.00% 100.00% 100.00% 0.09s 121
Format B (Chunked) 98.59% 99.93% 99.18% 84.63s 121
Format A 7.97% 81.03% 11.79% 23.32s 121

Key Findings:

  • GraphRAG achieves perfect performance (100% recall, 100% precision) with ultra-fast 0.09s latency
  • Format B achieves excellent 98.59% recall with chunked retrieval strategy
  • Format A fails on reverse queries (only 7.97% recall) - not suitable for this task
  • GraphRAG is ~900x faster than Format B for reverse queries (0.09s vs 84.6s)

Misspelling Robustness (180-Query Test Set)

System performance degradation with misspelled drug names.

Source: results/misspelling_experiment/summary_report_20251129_155639.txt

Architecture Correct Spelling Misspelled Accuracy Drop F1 Degradation
GraphRAG 100.00% 50.00% -50.00% -100.00%
Format B 100.00% 50.00% -50.00% -100.00%
Format A 90.00% 50.00% -44.44% -100.00%
Pure LLM 60.56% 62.78% +3.67% +8.66%

Key Findings:

  • RAG architectures fail to retrieve with misspelled drug names (drop to 50% - random guessing)
  • Pure LLM is robust to misspellings (slight improvement from 60.56% to 62.78%) as it relies on learned knowledge rather than exact retrieval matching
  • Misspelling severely impacts retrieval-based systems due to embedding/graph lookup failures
  • This highlights critical need for spell correction in production RAG systems

Spell Correction Recovery (180-Query Test Set)

Impact of LLM-based spell correction on misspelled queries.

Source: results/spell_correction_recovery/recovery_summary_20251129_163320.txt

Architecture Perfect F1 Raw Misspelled F1 With Spell Correction F1 Recovery Rate
GraphRAG 1.000 0.000 0.875 87.5%
Format B 1.000 0.000 0.875 87.5%
Format A 0.889 0.000 0.732 82.4%
Pure LLM 0.493 0.433 0.485 87.6%

Key Findings:

  • Spell correction recovers 82-88% of performance for RAG systems
  • GraphRAG and Format B both achieve 87.5% F1 score after correction
  • Format A recovers to 73.2% F1 (82.4% recovery rate)
  • Pure LLM maintains consistent performance (~0.49 F1) regardless of spelling, as it doesn't rely on retrieval
  • Spell correction is highly effective for production RAG deployment

Repository Structure

drugRAG/
├── README.md                    # This file
├── pyproject.toml              # Dependencies and project metadata
├── run_evaluations.sh          # Unified benchmark runner
├── data/
│   └── processed/              # Datasets and ground truth
├── experiments/
│   ├── reverse_query_benchmark.py      # Reverse query evaluation
│   ├── evaluate_vllm.py                # Binary classification with vLLM
│   ├── evaluate_misspelling.py         # Misspelling robustness test
│   └── evaluate_spell_correction.py    # Spell correction recovery
├── results/
│   ├── misspelling_experiment/         # Misspelling results (JSON/CSV/TXT)
│   ├── spell_correction_experiment/    # Spell correction results
│   └── spell_correction_recovery/      # Recovery analysis results
├── scripts/
│   ├── generate_comprehensive_dataset.py         # Dataset generation
│   ├── generate_comprehensive_dataset_parallel.py
│   ├── generate_ground_truth_neo4j.py            # Neo4j ground truth
│   └── fix_dataset_case_sensitivity.py
└── src/
    ├── architectures/
    │   ├── graphrag.py          # GraphRAG implementation
    │   ├── rag_format_a.py      # Format A RAG
    │   └── rag_format_b.py      # Format B RAG (chunked)
    ├── models/
    │   └── vllm_model.py        # vLLM client for Qwen/Llama3
    ├── utils/
    │   ├── database.py          # Neo4j database utilities
    │   ├── embedding_client.py  # OpenAI embedding client
    │   ├── spell_corrector.py   # Spell correction utilities
    │   └── ...                  # Other utilities
    └── evaluation/
        ├── metrics.py           # Evaluation metrics
        └── advanced_metrics.py  # Advanced analysis

Reproducibility

All experiments use the following settings for reproducibility:

  • Random Seed: 42 (set in all benchmark scripts)
  • Dataset: Comprehensive reverse queries (case-corrected, November 2025)
  • Ground Truth: Neo4j-based ground truth with drug-side effect pairs
  • Binary Test Set: 19,520 balanced queries (9,760 positive, 9,760 negative examples)
  • Misspelling Test Set: 180 balanced queries (90 positive, 90 negative examples)
  • Models: Qwen2.5-7B-Instruct and Llama3-8B via vLLM

Expected Performance Metrics

Based on November 2025 experiments (latest results from Nov 25-28, 2025):

Binary Classification (19,520 Queries):

  • GraphRAG Qwen: 99.95% accuracy, 0.999 F1 score
  • GraphRAG Llama3: 99.96% accuracy, 1.000 F1 score
  • Format B Qwen: 99.71% accuracy, 0.997 F1 score
  • Format B Llama3: 98.36% accuracy, 0.983 F1 score
  • Format A Qwen: 90.86% accuracy, 0.900 F1 score
  • Format A Llama3: 86.58% accuracy, 0.845 F1 score
  • Pure LLM Qwen: 62.93% accuracy, 0.494 F1 score
  • Pure LLM Llama3: 63.19% accuracy, 0.535 F1 score

Misspelling Robustness (180 Queries):

  • RAG systems drop to 50% accuracy (random guessing) with misspelled drug names
  • Spell correction recovers 82-88% of original performance
  • GraphRAG/Format B achieve 87.5% F1 after spell correction

Citation

If you use this code or findings in your research, please cite:

@software{drugrag2025,
  title={DrugRAG: Drug Side Effect Retrieval with RAG and GraphRAG},
  author={DrugRAG Team},
  year={2025},
  month={November},
  url={https://github.com/apicurius/drugRAG}
}

License

MIT License - see LICENSE file for details.