Sovereign 8s.is mirror of standardgalactic/neuro-san-benchmarking
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Sid's Cognizant Account 73b22fe712
Merge pull request #4 from cognizant-ai-lab/goPublic
Adding information files.
2025-11-13 08:46:15 -08:00
coded_tools Updated to meet Cognizant standards for public repository. 2025-11-12 09:29:16 -08:00
data Added 6x6 multiplication bechmark samples 2025-11-11 07:15:03 -08:00
.gitignore Initial commit: Extract benchmarking app from neuro-san-studio 2025-11-04 21:47:13 +00:00
__init__.py Initial commit: Extract benchmarking app from neuro-san-studio 2025-11-04 21:47:13 +00:00
agent_benchmark_runner.py Updated to meet Cognizant standards for public repository. 2025-11-12 09:29:16 -08:00
CODE_OF_CONDUCT.md Adding information files. 2025-11-13 08:43:38 -08:00
composition_discriminator.hocon Initial commit: Extract benchmarking app from neuro-san-studio 2025-11-04 21:47:13 +00:00
CONTRIBUTING.md Adding information files. 2025-11-13 08:43:38 -08:00
decomposer.hocon Better agent prompts 2025-11-10 07:51:59 -08:00
INSTRUMENTATION_GUIDE.md Initial commit: Extract benchmarking app from neuro-san-studio 2025-11-04 21:47:13 +00:00
LICENSE.txt Adding information files. 2025-11-13 08:43:38 -08:00
linear_multiagent_reasoner.py Updated to meet Cognizant standards for public repository. 2025-11-12 09:29:16 -08:00
make_synthetic_benchmarks.py Updated to meet Cognizant standards for public repository. 2025-11-12 09:29:16 -08:00
manifest_solver.hocon Initial commit: Extract benchmarking app from neuro-san-studio 2025-11-04 21:47:13 +00:00
multi_step_decomposer.hocon Initial commit: Extract benchmarking app from neuro-san-studio 2025-11-04 21:47:13 +00:00
multiagent_reasoner.py Updated to meet Cognizant standards for public repository. 2025-11-12 09:29:16 -08:00
problem_solver.hocon Best prompts so far plus 0.5 for temperature 2025-11-08 07:46:18 -08:00
README.md Initial commit: Extract benchmarking app from neuro-san-studio 2025-11-04 21:47:13 +00:00
requirements.txt Initial commit: Extract benchmarking app from neuro-san-studio 2025-11-04 21:47:13 +00:00
SECURITY.md Adding information files. 2025-11-13 08:43:38 -08:00
single_test.txt Initial commit: Extract benchmarking app from neuro-san-studio 2025-11-04 21:47:13 +00:00
solution_discriminator.hocon Better agent prompts 2025-11-10 07:51:59 -08:00
SUPPORT.md Adding information files. 2025-11-13 08:43:38 -08:00
synthetic_long_mul_6_6__200.jsonl Initial commit: Extract benchmarking app from neuro-san-studio 2025-11-04 21:47:13 +00:00
synthetic_sort_len_5__200.jsonl Initial commit: Extract benchmarking app from neuro-san-studio 2025-11-04 21:47:13 +00:00

Neuro-SAN Benchmarking

A benchmarking suite for evaluating multi-agent reasoning systems on arithmetic problems using Neuro-SAN.

Overview

This repository contains tools for benchmarking multi-agent systems on multiplication and other arithmetic tasks. The main components are:

  • multiagent_reasoner.py - Multi-agent reasoning system with decomposition, voting, and composition
  • linear_multiagent_reasoner.py - Linear variant of the multi-agent reasoner
  • agent_benchmark_runner.py - Generic benchmark runner supporting parallel execution
  • make_synthetic_benchmarks.py - Tool for generating synthetic benchmark datasets

Installation

  1. Clone this repository
  2. Create a virtual environment:
    python3 -m venv venv
    source venv/bin/activate
    
  3. Install dependencies:
    pip install -r requirements.txt
    
  4. Set up your OpenAI API key:
    export OPENAI_API_KEY="your-api-key-here"
    

Usage

Single Problem

Test the multi-agent reasoner on a single multiplication problem:

LOG_LEVEL=INFO echo "What is 46048 × 42098?" | python multiagent_reasoner.py

Benchmark Suite

Run the benchmark on a dataset:

python agent_benchmark_runner.py \
  --python-prog multiagent_reasoner.py \
  --task mul_5x5 \
  --local-jsonl data/bench_long_mul_5_5__200.jsonl \
  --answer-format number \
  --final-token '>>>>' \
  --num-workers 10 \
  --limit 10

Configuration

The multi-agent reasoner supports several environment variables:

  • WINNING_VOTE_COUNT - Number of votes required for consensus (default: 2)
  • MAX_DEPTH - Maximum recursion depth for decomposition (default: 5)
  • LOG_LEVEL - Logging level (default: INFO)
  • LOG_DIR - Directory for detailed per-run logs (optional)
  • LOG_FAILURES_JSONL - Path to log failure cases for analysis (optional)

Example with custom configuration:

export WINNING_VOTE_COUNT=8
export MAX_DEPTH=3
export LOG_FAILURES_JSONL="./failures.jsonl"
export LOG_DIR="./logs"

python agent_benchmark_runner.py \
  --python-prog multiagent_reasoner.py \
  --task mul_5x5 \
  --local-jsonl data/bench_long_mul_5_5__200.jsonl \
  --answer-format number \
  --final-token '>>>>' \
  --num-workers 50 \
  --limit 200 \
  --timeout-ms 180000

Benchmark Runner Options

  • --python-prog - Path to the Python program to benchmark
  • --task - Task name (e.g., mul_5x5, gsm8k)
  • --local-jsonl - Path to local JSONL benchmark file
  • --answer-format - How to parse answers (number, list-json)
  • --final-token - Token marking the final answer (default: ####)
  • --num-workers - Number of parallel workers (default: 1)
  • --limit - Limit number of problems to evaluate
  • --timeout-ms - Per-item timeout in milliseconds (default: 120000)
  • --sample-retries - Max retries per sample on failure (default: 2)

Output

The benchmark runner produces:

  • results_{task}_{timestamp}.csv - Detailed results for each problem
  • results_{task}_{timestamp}.jsonl - Results in JSONL format
  • results_progress_{timestamp}.csv - Real-time progress tracking

If failure logging is enabled:

  • Failure logs in the specified JSONL file
  • Per-run detailed logs in the LOG_DIR directory

Instrumentation

See INSTRUMENTATION_GUIDE.md for details on:

  • Failure pattern analysis
  • Trace data collection
  • Debugging techniques

Benchmark Data

The data/ directory contains sample benchmark datasets:

  • bench_long_mul_5_5__200.jsonl - 200 5×5 digit multiplication problems
  • bench_long_mul_10_10__200.jsonl - 200 10×10 digit multiplication problems
  • bench_sort_len_500__50.jsonl - 50 sorting problems

License

See LICENSE.txt in the parent repository.