Sovereign 8s.is mirror of standardgalactic/babel-ai
  • Jupyter Notebook 60.1%
  • Python 39.9%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
2025-10-02 15:49:15 +02:00
.vscode Initializes vsc test settings 2025-03-17 16:12:34 +01:00
analysis pdf generation from output file (prelim) 2025-07-18 10:34:41 +02:00
configs Update the fetcher and their configs and adapt all code and tests 2025-07-09 15:00:29 +02:00
doc Add documentation for the repository 2025-07-29 12:00:41 +02:00
notebooks beautify graph/agg only/purge out 2025-10-02 13:54:56 +02:00
src New model names and adapted config 2025-10-02 15:49:15 +02:00
tests feat: Add graphical analysis utilities and notebook 2025-10-02 09:59:58 +00:00
.env_example Introduce Anthropic API access 2025-07-24 16:31:29 +02:00
.gitignore ignore logs 2025-07-24 15:00:20 +02:00
.pre-commit-config.yaml initiate pyproject for repository 2025-03-03 13:44:24 +01:00
LICENSE Initial commit 2025-03-03 13:23:23 +01:00
poetry.lock Introduce Anthropic API access 2025-07-24 16:31:29 +02:00
pyproject.toml feat: Add graphical analysis utilities and notebook 2025-10-02 09:59:58 +00:00
README.md Add documentation for the repository 2025-07-29 12:00:41 +02:00

babel-ai

A research framework for analyzing long-term behavior and drift patterns in Large Language Models (LLMs) during self-loop conversations without external input.

Overview

This project investigates whether AIs need external input to create sensible output and explores the importance of human interaction for AI performance. It analyzes how AI interaction can be made more diverse without external input and what productive input patterns look like.

Quick Start

Installation

  1. Prerequisites: Python 3.13+ and Poetry

    Install Poetry if needed:

    # Official installer (recommended)
    curl -sSL https://install.python-poetry.org | python3 -
    
    # Or via pip
    pip install --user poetry
    
    # Or via Homebrew (macOS)
    brew install poetry
    
  2. Clone and install:

    # Clone the repository
    git clone <repository-url>
    cd babel_ai
    
    # Install dependencies using Poetry
    poetry install
    
    # Activate the virtual environment
    poetry shell
    
  3. Set up environment (optional):

    # Create .env file for API keys if using external LLM providers
    cp .env.example .env  # Edit with your API keys
    

Running Tests

# Run all tests
poetry run pytest

# Run with coverage
poetry run coverage run --source=src -m pytest
poetry run coverage report

# Run specific test categories
poetry run pytest tests/unit_tests/     # Unit tests only
poetry run pytest tests/integration/   # Integration tests only

Quick Experiment

  1. Basic experiment:

    # Run with the provided test configuration
    poetry run python src/main.py configs/test_config.yaml
    
  2. Custom experiment:

    # Create your own config file (see configs/test_config.yaml as template)
    poetry run python src/main.py your_config.yaml --debug
    
  3. View results:

    • Results saved to results/ directory as CSV and JSON files
    • Use notebooks in notebooks/ for analysis and visualization

For detailed experiment configuration, multi-agent setups, and analysis workflows, see doc/experiments.md.

Key Features

  • Multi-Provider Support: OpenAI, Anthropic, Azure OpenAI, Ollama
  • Drift Analysis: Semantic similarity, lexical analysis, perplexity metrics
  • Data Sources: ShareGPT, Topical Chat, Infinite Conversation datasets
  • Configurable Experiments: YAML-based configuration system
  • Analysis Notebooks: Jupyter notebooks for result visualization
  • Comprehensive Testing: Unit and integration test suites

Project Structure

babel_ai/
├── src/
│   ├── babel_ai/          # Core experiment framework
│   ├── api/               # LLM provider interfaces
│   ├── models/            # Data models and configurations
│   └── main.py            # Main experiment runner
├── tests/                 # Test suites
├── notebooks/             # Analysis and visualization
├── configs/               # Experiment configurations
├── doc/                   # Detailed documentation
└── data/                  # Datasets (not included)

Contributing

We welcome contributions! Please see doc/contributing.md for detailed guidelines including:

  • Code style and formatting (Black, 79-char limit)
  • Testing requirements (pytest, coverage)
  • Development workflow
  • Adding new analyzers, fetchers, or providers

Documentation

License

LICENSE

Research Context

This project explores fundamental questions about AI behavior:

  • Do AIs require external input for coherent output?
  • How important is human interaction for AI performance?
  • Can AI interaction diversity be improved without external input?
  • What input patterns are most productive for AI systems?