Sovereign 8s.is mirror of standardgalactic/sdoh_annotation
- Jupyter Notebook 97.8%
- Python 2.2%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
| .env.example | ||
| .gitignore | ||
| augmenting.py | ||
| definition.json | ||
| optimize_prompt.py | ||
| prompts.py | ||
| README.md | ||
| requirements.txt | ||
| SHARE_bert_ft.ipynb | ||
| SHARE_dspy_load.ipynb | ||
| SHARE_dspy_train.ipynb | ||
| SHARE_step2_Qwen_2_5_+_Unsloth_2x_faster_finetuning.ipynb | ||
SDOH Annotation Project
This project implements a complete Social Determinants of Health (SDOH) annotation pipeline, including data augmentation, prompt optimization, and model training phases.
Project Structure
project/
├── augmenting.py # Data augmentation script
├── prompts.py # Prompt templates
├── optimize_prompt.py # Prompt optimization script
├── definition.json # Label definition file
├── baseline_llm.ipynb # Baseline model implementation
├── dspy_train.ipynb # DSPy training workflow
├── dspy_load.ipynb # DSPy model loading
├── bert_ft.ipynb # BERT fine-tuning
└── requirements.txt # Dependency list
1. Data Preparation and Augmentation
1.1 Label Definitions (definition.json)
The project uses a structured JSON file to define the SDOH label system:
- t1_Homelessness
- t1_InadequateHousing
- t1_LackOfAdequateFood
- t2_FinancialInsecurity
- t2_HousingInstability
- t2_MaterialHardship
- t2_TransportationInsecurity
- t3_Eviction_absent
- t3_Eviction_present_history
- t3_Eviction_present_current
- t3_Eviction_pending
- t3_Eviction_mr_history
- t3_Eviction_mr_current
- t3_Eviction_hypothetical
Each label includes:
- Detailed definition
- Example sentences
- Usage guidelines
1.2 Data Augmentation (augmenting.py)
Two modes are used for data augmentation:
-
MIMIC Mode:
- Complete rewriting of clinical notes
- Integration of SDOH label information
- Maintaining medical accuracy
- Output limited to 100 words
-
PMC Mode:
- Insertion of 1-2 relevant sentences into existing notes
- Ensuring natural integration with context
- Maintaining narrative coherence
1.3 Prompt Optimization (prompts.py & optimize_prompt.py)
Contains three core prompt templates:
- MIMIC prompts: For complete note rewriting
- PMC prompts: For contextual insertion
- Optimization prompts: For refining definitions based on expert feedback
Optimization process:
- Analysis of error case patterns
- Adjustment of definitions based on expert feedback
- Addition of new examples
- Maintaining clarity and accuracy of definitions
2. Model Training
2.1 DSPy Training (dspy_train.ipynb)
Using the DSPy framework for model training:
-
Data Preparation:
- Preparing train set and dev set
- Data preprocessing and formatting
-
Training Process:
- Configuring DSPy training parameters
- Implementing custom training logic
- Model evaluation and optimization
-
Output:
- Trained model
- Performance evaluation report
- Optimized prompts
2.2 Open Source Model Fine-tuning
BERT Fine-tuning (bert_ft.ipynb)
BERT model fine-tuning for SDOH classification:
-
Data Preparation:
- Text preprocessing
- Label encoding
- Dataset splitting
-
Training Configuration:
- Learning rate settings
- Batch size optimization
- Training epochs adjustment
-
Evaluation Metrics:
- Accuracy
- F1 score
- Confusion matrix
Qwen/llama Fine-tuning
Model fine-tuning using Unsloth acceleration:
-
Optimization Features:
- 2x training speed
- Reduced memory usage
- Improved inference efficiency
-
Training Steps:
- Configuring Unsloth optimizer
- Implementing LoRA fine-tuning
- Model evaluation and export
Usage Instructions
Environment Setup
pip install -r requirements.txt
API Key Configuration
Create a .env file and configure:
OPENAI_API_KEY=your_key
LANGFUSE_PUBLIC_KEY=your_key
LANGFUSE_SECRET_KEY=your_key
LANGFUSE_HOST=your_host
Running Data Augmentation
python augmenting.py
Running Training
- Open the corresponding training file in Jupyter Notebook
- Follow the instructions in the notebook step by step