Sovereign 8s.is mirror of standardgalactic/carchive
  • Python 85.7%
  • HTML 11.7%
  • Shell 2.3%
  • Mako 0.1%
  • JavaScript 0.1%
  • Other 0.1%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Tem Noon 90c10c277d Improve title generation prompt with stricter formatting instructions
- Modified the title generation prompt to enforce stricter output constraints
- Added numbered list format for clear instructions
- Added explicit guidance to prevent model from responding to content requests
- Emphasized single-line response requirement and maximum word count

🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
2025-03-02 17:25:29 -05:00
alembic Add buffer system for chunk management 2025-03-02 15:11:07 -05:00
alembic-new Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
chat Update .gitignore, remove large files and add chat/ directory to .gitignore 2025-02-26 16:14:44 -05:00
docs Update gencom_guide.md with latest features 2025-03-01 22:40:23 -05:00
my_output Update .gitignore and add .gitkeep files to maintain directory structure 2025-02-26 14:56:45 -05:00
my_outputs Update .gitignore and add .gitkeep files to maintain directory structure 2025-02-26 14:56:45 -05:00
old_alembic_problems Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
output Update .gitignore and add .gitkeep files to maintain directory structure 2025-02-26 14:56:45 -05:00
output_tests Update .gitignore and add .gitkeep files to maintain directory structure 2025-02-26 14:56:45 -05:00
scripts Add script for semantic search and summarization 2025-03-01 11:38:58 -05:00
src/carchive Improve title generation prompt with stricter formatting instructions 2025-03-02 17:25:29 -05:00
tests Add tests for chunk management system 2025-03-02 15:28:25 -05:00
.gitignore Add provider-based filtering and improved CLI for conversations 2025-02-28 21:45:50 -05:00
alembic.ini Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
api_server.py Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
apply_media_message_fix.sh Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
CLAUDE.md Add provider-based filtering and improved CLI for conversations 2025-02-28 21:45:50 -05:00
comprehensive_fix.sh Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
conversation_render.html Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
criteria_Agent_Theory_20250120_012237.json Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
criteria_Agent_Theory_20250122_001721.json Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
criteria_article_20250121_233826.json Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
criteria_Edmund_Husserl_and_Phenomenology_20250120_010811.json Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
criteria_heart_sutra_science_20250120_012041.json Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
criteria_phenomenology_20250120_161336.json Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
disable_conda_auto.sh Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
disable_conda_completely.sh Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
environment.yml Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
fix_conda_base.sh Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
fix_conda_conflict.sh Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
fix_cors.sh Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
fix_dependencies.sh Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
gui_server.py Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
install_missing_deps.sh Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
install_missing_markdown_ext.sh Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
LICENSE Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
migrate_to_carchive.py Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
poetry.lock Update poetry dependencies and lock file 2025-03-02 12:46:04 -05:00
pyproject.toml Update poetry dependencies and lock file 2025-03-02 12:46:04 -05:00
pytest.ini Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
README.md Add cluster visualization to README 2025-03-01 22:22:22 -05:00
requirements.txt Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
restart_servers.sh Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
restart_servers_alt_ports.sh Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
run_api.sh Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
run_api_fixed.sh Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
run_api_mac.sh Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
run_api_venv.sh Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
run_api_with_cors.sh Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
run_gui.sh Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
run_gui_fixed.sh Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
run_gui_mac.sh Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
run_gui_standalone.sh Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
run_gui_venv.sh Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
run_gui_with_cors.sh Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
search_criteria.json Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
setup_carchive04_db.sh Enhance setup_carchive04_db.sh with macOS Keychain integration 2025-03-01 23:03:01 -05:00
setup_carchive_env.sh Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
setup_environment.sh Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
setup_mac_optimized_env.sh Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
setup_venv.sh Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00
tree.txt Initial commit for carchive project with corrected paths and headers 2025-02-26 14:41:09 -05:00

Carchive

Carchive is a modern conversation archiving and analysis system designed to store, organize, search, and analyze text-based conversations using vector embeddings and AI. It supports importing conversations from multiple providers including ChatGPT, Claude and eventually others. Visualization of clusters:

Cluster Visualization

Features

  • Multi-Provider Support: Import from ChatGPT, Claude, and other LLM providers
  • Conversation Archiving: Store and organize text-based conversations with metadata
  • Vector Search: Find semantically similar content using pgvector
  • AI Analysis: Generate summaries, insights, and connections between conversations
  • Media Management: Link media files with conversation messages
  • Collections: Organize conversations and messages into custom collections
  • Statistics: View usage patterns and distribution by provider

Prerequisites

  • Python 3.10+
  • PostgreSQL with pgvector extension
  • Poetry for dependency management

Installation

  1. Clone this repository:

    git clone https://github.com/temnoon/carchive.git
    cd carchive
    
  2. Install dependencies:

    poetry install
    
  3. Set up environment variables (create a .env file):

    DATABASE_URL=postgresql://username:password@localhost:5432/carchive04_db
    ANTHROPIC_API_KEY=your_anthropic_api_key  # Optional for AI features
    OPENAI_API_KEY=your_openai_api_key  # Optional for AI features
    OLLAMA_URL=http://localhost:11434  # Optional for local embeddings
    
  4. Create and initialize the database:

    # Create the database with pgvector support
    ./setup_carchive04_db.sh
    
    # Or run the migrations manually
    poetry run alembic upgrade head
    

CLI Commands

Carchive provides a comprehensive CLI interface for managing conversations and performing analysis.

Main CLI

poetry run carchive [OPTIONS] COMMAND [ARGS]...

Available Commands:

  • api - Commands for managing the API server
  • collection - Commands to manage or create collections
  • conv2 - Commands for managing conversations
  • conversation - Legacy conversation commands
  • cluster - Commands for clustering and visualizing embeddings by topic
  • embed - Commands for generating and storing embeddings
  • gencom - Commands for generating AI comments on content (see Gencom Guide)
  • ingest - Commands to ingest chat data into the database
  • media - Commands for managing media files
  • migrate - Migration tools for carchive
  • search - Commands to search conversations/messages/chunks
  • summarize - Commands for generating summaries using content agents

Migration from LLM Services

# Migrate ChatGPT archive
poetry run carchive migrate chatgpt --db-name=carchive04_db --db-user=carchive_app path/to/conversations.json

# Migrate Claude archive
poetry run carchive migrate claude --db-name=carchive04_db --db-user=carchive_app path/to/claude_archive

Conversation Management

# List all conversations
poetry run carchive conv2 list

# Filter conversations by provider
poetry run carchive conv2 list --provider claude

# Get conversation details
poetry run carchive conv2 get <conversation-id> --with-messages

# View conversation statistics by provider
poetry run carchive conv2 stats

# List available providers
poetry run carchive conv2 providers

# Export a conversation to JSON
poetry run carchive conv2 export <conversation-id> output.json
# Basic search
poetry run carchive search messages "your search term"

# Detailed search with context
poetry run carchive search detailed "your search term"

# Advanced search using saved criteria
poetry run carchive search advanced --criteria-file criteria.json --output-file results.html

Media Management

# Scan and link media files
poetry run carchive media scan-and-link --media-dir /path/to/media --recursive

# Analyze media distribution
poetry run carchive media analyze-media-distribution

Collections

# Create a new collection
poetry run carchive collection create "Collection Name" --description "Description"

# List existing collections
poetry run carchive collection list

# Render a collection to HTML
poetry run carchive collection render <collection-id> --output-file collection.html

Embeddings

# Generate embeddings for all messages
poetry run carchive embed all --min-word-count 10

# Generate embeddings for specific roles
poetry run carchive embed all --min-word-count 10 --include-roles user assistant

# Generate embeddings with custom model
poetry run carchive embed all --min-word-count 10 --provider openai --model-version text-embedding-ada-002

# Generate embeddings for a specific collection
poetry run carchive embed collection <collection-id>

# Generate summaries for messages with embeddings
poetry run carchive embed summarize --provider llama3.2

Clustering and Topic Analysis

# Analyze embeddings to find optimal cluster count
poetry run carchive cluster analyze

# Create clusters with KMeans algorithm and visualize
poetry run carchive cluster kmeans --n-clusters=20 --visualization

# Use density-based clustering with DBSCAN
poetry run carchive cluster dbscan --visualization

# Create hierarchical clusters with topic and subtopic levels
poetry run carchive cluster hierarchical --primary-clusters=5 --secondary-clusters=3

For detailed documentation on clustering capabilities, see Embedding Clusters Guide.

API Server

# Run the API server
poetry run carchive api serve --host 0.0.0.0 --port 8000

Development

# Run tests
poetry run pytest

# Add dependency
poetry add package_name

# Set up environment
./setup_carchive_env.sh

Database Structure

The carchive04_db uses a normalized structure with the following key tables:

  • providers - Information about LLM providers (ChatGPT, Claude, etc.)
  • conversations - Main conversation metadata
  • messages - Individual messages with content and metadata
  • media - Media file references and metadata
  • message_media - Links between messages and media
  • collections/collection_items - For organizing conversations and messages
  • embeddings - Vector embeddings for semantic search

License

MIT License