- Python 100%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
| bert_trainer.py | ||
| config.py | ||
| data_processor.py | ||
| evaluate.py | ||
| LICENSE | ||
| main.py | ||
| predictor.py | ||
| README.md | ||
| requirements.txt | ||
| run_all_scenarios.py | ||
| tfidf_baseline.py | ||
| utils.py | ||
BERT Multi-Label Text Classification for UXPID dataset
An implementation of BERT-based multi-label text classification for forum thread categorization.
Dataset: UXPID on Zenodo | Paper: arXiv:2509.11777
📁 Project Structure
Standard Structure
bert_classification/
├── dataset/ # Raw data (JSON files)
├── splits/ # Official Zenodo train/test split
│ ├── train_branches.txt
│ └── test_branches.txt
├── processed_data/ # Processed training/test data
├── data_processor.py # Data preprocessing module
├── bert_trainer.py # BERT training module
├── tfidf_baseline.py # TF-IDF + Logistic Regression baseline
├── predictor.py # Inference module
├── evaluate.py # Evaluation module
├── main.py # Main training script
├── config.py # Configuration settings
├── utils.py # Utility functions for timestamp folders
├── requirements.txt # Python dependencies
└── README.md # This file
Timestamp-Based Output Structure (Default)
When using timestamp folders (enabled by default), all outputs are organized as:
outputs/
└── YYYYMMDD_HHMMSS/
├── models/
│ └── best_model.pth
├── logs/
│ └── training_YYYYMMDD_HHMMSS.log
├── plots/
│ ├── training_history.png
│ └── data_visualization.png
├── results/
│ ├── final_results.json
│ ├── training_config.json
│ └── data_analysis.json
├── evaluation_results/
│ ├── evaluation_report.json
│ └── plots/
└── run_summary.json
🛠️ Installation
-
Clone the repository
git clone <repository-url> cd bert_classification -
Create a virtual environment (recommended)
python -m venv venv source venv/bin/activate -
Install dependencies
pip install -r requirements.txt
📊 Data Format
Download the dataset from Zenodo and place the JSON files in the dataset/ directory. Each JSON file represents a forum thread with the following structure:
{
"metadata": {
"branch_id": "1000153386",
"thread_id": "9520646325",
"publication_year": 2021,
"branch_status": "Unsolved",
"branch_type": "question"
},
"content": [
{
"comment_id": 341703487007721,
"user_name": "user123",
"comment_position": 0,
"is_reply": false,
"comment_year": 2021,
"comment_month": 9,
"comment_body": "Forum post text content..."
}
],
"topics": {
"21": "Bug Reports",
"13": "Network Configuration Challenges"
}
}
🚀 Quick Start
Run the complete BERT training pipeline with automatic timestamp folders:
python main.py
Run the TF-IDF + Logistic Regression baseline as a separate pipeline:
python tfidf_baseline.py
Both scripts share the same data, config, and output folder structure. Key CLI options for the baseline:
| Flag | Default | Description |
|---|---|---|
--text-field |
from config.py |
text or insight_summary |
--target-field |
from config.py |
topics, branch_status, branch_type, overall_branch_sentiment |
--max-features |
50000 |
TF-IDF vocabulary size |
--ngram-max |
2 |
Upper n-gram bound (1 = unigrams, 2 = unigrams+bigrams) |
--C |
1.0 |
Logistic Regression regularisation strength |
--skip-data-processing |
off | Load existing processed_data/ without re-processing |
--data-split |
off | Create a new train/test split |
--use-official-split |
off | Use the official Zenodo split (requires --data-split) |
--splits-dir |
splits |
Path to train_branches.txt / test_branches.txt |
To use the official Zenodo split (recommended for reproducibility):
python tfidf_baseline.py --data-split --use-official-split
python main.py --data-split --use-official-split
To compare TF-IDF and BERT side by side, use TFIDFBaseline.print_comparison() / plot_comparison() programmatically after both runs.
This will:
- Create a timestamp folder (e.g.,
outputs/20250118_143052/) - Process the raw forum data
- Create train/validation/test splits
- Train the BERT model
- Save all outputs (models, logs, plots, results) in organized subfolders
Making Predictions
from predictor import BERTPredictor
# Initialize predictor
predictor = BERTPredictor("best_model.pth")
# Single prediction
text = "I'm having network connectivity issues with my device."
result = predictor.predict_single(text)
print(f"Predicted labels: {result['predicted_labels']}")
# Batch predictions
texts = ["Text 1", "Text 2", "Text 3"]
results = predictor.predict_batch(texts)
# Get top-k predictions
top_predictions = predictor.get_top_predictions(text, top_k=3)
Model Evaluation
python evaluate.py --model-path best_model.pth --output-dir evaluation_results
This generates:
- Comprehensive evaluation metrics
- Confusion matrices
- Precision-Recall curves
- ROC curves
- Error analysis
⚙️ Configuration
Edit config.py to customize:
- Model settings: BERT model type, sequence length, etc.
- Training parameters: Learning rate, batch size, epochs
- Data processing: Text cleaning, filtering options
- Evaluation metrics: Thresholds, metrics to track
Key configuration sections:
MODEL_CONFIG = {
"model_name": "bert-base-uncased",
"max_length": 512,
"dropout_rate": 0.3,
}
TRAINING_CONFIG = {
"batch_size": 8,
"learning_rate": 2e-5,
"num_epochs": 3,
"warmup_steps": 500,
}
AI Use Disclosure
The UXPID corpus was created with the assistance of AI tools. For full details, see the paper: arXiv:2509.11777. Specifically, the analysis fields present in each JSON record — including insight_summary, user_expectations, severity_expectation_level, gain_keywords, pain_keywords, feature_keywords, and overall_branch_sentiment — were generated using a large language model (LLM). These AI-generated annotations were produced by prompting the model to analyse the raw forum thread content and extract structured UX-relevant insights.
The raw forum content (content field) was artificially synthesized and anonymized from branches originally extracted from a public industrial automation forum. Thread metadata (metadata field) was derived from the same source.
📝 Requirements
- Python 3.8+
- PyTorch 2.0+
- Transformers 4.30+
- CUDA-capable GPU (recommended)
- 8GB+ RAM
- 2GB+ GPU memory
📄 Citation
If you use this dataset or code in your research, please cite the paper: arXiv:2509.11777
@article{kulyabin2025user,
title={User eXperience Perception Insights Dataset (UXPID): Synthetic User Feedback from Public Industrial Forums},
author={Kulyabin, Mikhail and Joosten, Jan and Pacheco, Nuno Miguel Martins and Ries, Fabian and Petridis, Filippos and Bosch, Jan and Olsson, Helena Holmstr{\"o}m and others},
journal={arXiv preprint arXiv:2509.11777},
year={2025}
}