Sovereign 8s.is mirror of standardgalactic/discover
  • Python 99.1%
  • Shell 0.9%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
2026-08-26 00:58:41 -04:00
complex Initial commit 2026-08-26 00:53:56 -04:00
conj Initial commit 2026-08-26 00:53:56 -04:00
letter_sequence Initial commit 2026-08-26 00:53:56 -04:00
list_3_5 Initial commit 2026-08-26 00:53:56 -04:00
list_5_5 Initial commit 2026-08-26 00:53:56 -04:00
svo Initial commit 2026-08-26 00:53:56 -04:00
.gitignore Initial commit 2026-08-26 00:53:56 -04:00
analogies_interventions.py Initial commit 2026-08-26 00:53:56 -04:00
compute_standardization_vectors.py Initial commit 2026-08-26 00:53:56 -04:00
config.py Initial commit 2026-08-26 00:53:56 -04:00
decompose.py Initial commit 2026-08-26 00:53:56 -04:00
decompose_ensemble.py Initial commit 2026-08-26 00:53:56 -04:00
dep_parse.py Initial commit 2026-08-26 00:53:56 -04:00
discover_argparser.py Initial commit 2026-08-26 00:53:56 -04:00
discover_supplement.pdf Create discover_supplement.pdf 2026-08-26 00:58:41 -04:00
evaluation.py Initial commit 2026-08-26 00:53:56 -04:00
generate_encodings.py Initial commit 2026-08-26 00:53:56 -04:00
generate_encodings_llm.py Initial commit 2026-08-26 00:53:56 -04:00
LICENSE Initial commit 2026-08-26 00:39:03 -04:00
models.py Initial commit 2026-08-26 00:53:56 -04:00
README.md Initial commit 2026-08-26 00:53:56 -04:00
role_assignment_functions.py Initial commit 2026-08-26 00:53:56 -04:00
sh_to_scrs.py Initial commit 2026-08-26 00:53:56 -04:00
systematic_replace_ensemble.py Initial commit 2026-08-26 00:53:56 -04:00
token_dictionary.py Initial commit 2026-08-26 00:53:56 -04:00
train_decoder.py Initial commit 2026-08-26 00:53:56 -04:00
train_seq2seq.py Initial commit 2026-08-26 00:53:56 -04:00
training.py Initial commit 2026-08-26 00:53:56 -04:00
utils.py Initial commit 2026-08-26 00:53:56 -04:00

DISCOVER

DISsecting COmpositionality in VEctor Representations

Requirements

  1. Create a python venv to run code in:
python -m venv .venv
  1. Activate the venv
source .venv/bin/activate
  1. Install requirements
pip install -U pip setuptools wheel
pip install torch
pip install numpy

pip install transformers
pip install accelerate

Letter sequence experiments

Basic DISCOVER

  1. Generate data
cd letter_sequence/
source dataset_creation.sh
  1. Train models
python train_target_models_scriptwriter.py 
cd ..

# Run this 10 times to get 10 reruns of each
source letter_sequence/train_target_models.sh

# Collect the results; they will be in logs/aggregated_target_model_acc.txt
cd letter_sequence/
python aggregate_target_model_accs.py
cd ..
  1. Run DISCOVER
cd letter_sequence/
python decompose_scriptwriter.py
cd ..
source letter_sequence/decompose.sh

# Collect the results; they will be in logs/aggregated_tpe_accs.txt 
cd letter_sequence/
python aggregate_tpe_accs.py 
cd ..

DISCOVER with OOD generalization

  1. Generate data
cd letter_sequence/
source dataset_creation_ood.sh
  1. Do hyperparameter search for the weight for the L1 loss term
python decompose_scriptwriter_ood_hypsearch.py
cd ..
source letter_sequence/decompose_ood_hypsearch.sh 
cd letter_sequence/
python aggregate_tpe_accs_ood_hypsearch.py
  1. Run DISCOVER
python decompose_scriptwriter_ood.py 
cd ..
source letter_sequence/decompose_ood.sh

# Collect the results; they will be in logs/aggregated_tpe_accs_ood.txt
cd letter_sequence/
python aggregate_tpe_accs_ood.py

Constituent surgery / causal interventions

  1. Generate interventions
cd letter_sequence/
source intervention_generation.sh 
  1. Run the interventions
python intervention_scriptwriter.py 
cd ..
source letter_sequence/interventions.sh
  1. Aggregate the results; they will be in logs/aggregated_intervention_acc.txt
cd letter_sequence/
python aggregate_interventions.py

Analogies

  1. Generate analogies
cd letter_sequence/
source analogy_generation.sh
  1. Run the analogies
python analogy_scriptwriter.py
cd ..
source letter_sequence/analogies.sh
  1. Aggregate the results; they will be in logs/aggregated_analogy_acc.txt
cd letter_sequence/
python aggregate_analogies.py

Period encoding experiments

SVO sentences

  1. Generate the sentences:
cd svo
python svo_sentence_generator.py
  1. Encode the sentences
python encoding_scriptwriter.py
cd ..
source svo/encodings.sh
  1. Compute means and standard deviations for z-scoring
cd svo
python standardization_scriptwriter.py
cd ..
source svo/standardization.sh
  1. Train decoder
cd svo
python decoder_scriptwriter.py
cd ..
source svo/hidden_decoder.sh
cd svo

# Results will be in logs/aggregated_hidden_decoder_accs.txt 
python aggregate_decoder_acc.py 
  1. Run DISCOVER
cd svo/
python decompose_scriptwriter.py 
cd ..
source svo/decompose_llms.sh
cd svo

# Results will be in:
#   logs/aggregated_tpe_acc.txt
#   logs/aggregated_tpe_agr.txt
#   logs/aggregated_tpe_r2.txt
python aggregate_tpe_results.py 

Lists

  1. Generate the lists
cd list_3_5
python list_generator.py
  1. Encode the lists
python encoding_scriptwriter.py
cd ..
source list_3_5/encodings.sh
  1. Compute means and standard deviations for z-scoring
cd list_3_5
python standardization_scriptwriter.py
cd ..
source list_3_5/standardization.sh
  1. Train decoder
cd list_3_5
python decoder_scriptwriter.py
cd ..
source list_3_5/hidden_decoder.sh


# Results will be in logs/aggregated_hidden_decoder_accs.txt 
python aggregate_decoder_acc.py 
  1. Run DISCOVER
cd list_3_5/
python decompose_scriptwriter.py 
cd ..
source list_3_5/decompose_llms.sh
cd list_3_5/

# Results will be in:
#   logs/aggregated_tpe_acc.txt
#   logs/aggregated_tpe_agr.txt
#   logs/aggregated_tpe_r2.txt
python aggregate_tpe_results.py

Complex sentences

  1. Generate the sentences:
cd complex
python sentence_generator.py
  1. Encode the sentences
python encoding_scriptwriter.py
cd ..
source complex/encodings.sh
  1. Compute means and standard deviations for z-scoring
cd complex
python standardization_scriptwriter.py
cd ..
source complex/standardization.sh
  1. Train decoder
cd complex
python decoder_scriptwriter.py
cd ..
source complex/hidden_decoder.sh


# Results will be in logs/aggregated_hidden_decoder_accs.txt 
python aggregate_decoder_acc.py 
  1. Run DISCOVER
cd complex/
python decompose_scriptwriter.py
cd ..
source complex/decompose_llms.sh
cd complex/

# Results will be in:
#   logs/aggregated_tpe_acc.txt
#   logs/aggregated_tpe_agr.txt
#   logs/aggregated_tpe_r2.txt
python aggregate_tpe_results.py

OOD: Lists

  1. Generate the sentences
cd list_5_5
python list_generator.py
python list_generator_ood.py 
  1. Encode the sentences
python encoding_scriptwriter.py 
python encoding_scriptwriter_ood_split.py
cd ..
source list_5_5/encodings.sh
source list_5_5/encodings_ood_split.sh
  1. Compute means and standard deviations for z-scoring
cd list_5_5
python standardization_scriptwriter.py
python standardization_scriptwriter_ood_split.py
cd ..
source list_5_5/standardization.sh
source list_5_5/standardization_ood_split.sh
  1. Train decoder
cd list_5_5/
python decoder_scriptwriter.py
cd ..
source list_5_5/hidden_decoder.sh

# Results will be in logs/aggregated_hidden_decoder_accs.txt 
python aggregate_decoder_acc.py
  1. Run a hypothesis search for the L1 lambda hyperparameter
cd list_5_5
python decompose_scriptwriter_l1.py
cd ..
source list_5_5/decompose_llms_l1_hypsearch.sh
  1. Run DISCOVER
cd list_5_5/
python decompose_scriptwriter_ood.py
cd ..
source list_5_5/decompose_llms_ood.sh
cd list_5_5/

# Results will be in:
#   logs/aggregated_tpe_acc.txt
#   logs/aggregated_tpe_agr.txt
#   logs/aggregated_tpe_r2.txt
python aggregate_tpe_results_ood.py

OOD: Sentences

  1. Generate the sentences
cd conj
python sentence_generator.py 
python sentence_generator_ood_sameverb.py  
  1. Encode the sentences
python encoding_scriptwriter.py 
python encoding_scriptwriter_oodsameverb.py 
cd ..
source conj/encodings.sh
source conj/encodings_ood_sameverb.sh
  1. Compute means and standard deviations for z-scoring
cd conj
python standardization_scriptwriter.py
python standardization_scriptwriter_oodsameverb.py
cd ..
source conj/standardization.sh
source conj/standardization_oodsameverb.sh
  1. Train decoder
cd conj/
python decoder_scriptwriter.py
cd ..
source conj/hidden_decoder.sh
  1. Run DISCOVER
cd conj/
python decompose_scriptwriter_ood.py
cd ..
source conj/decompose_llms_ood_sameverb.sh
cd conj/

# Results will be in:
#   logs/aggregated_tpe_acc.txt
#   logs/aggregated_tpe_agr.txt
#   logs/aggregated_tpe_r2.txt
python aggregate_tpe_results_ood.py

Analogies: Lists

  1. Generate the analogies
cd list_5_5
python analogy_generator.py
  1. Encode the analogy sentences
python get_analogy_sentences.py
python encoding_scriptwriter_analogies.py

cd ..
source list_5_5/encodings_analogies.sh
  1. Run analogies
cd list_5_5
python analogy_scriptwriter.py
cd ..
source list_5_5/analogies.sh
cd list_5_5

# Results will be in logs/aggregated_analogy_acc.txt 
python aggregate_analogy_acc.py

Analogies: Sentences

  1. Generate the analogies
cd conj
python analogy_generator.py
  1. Encode the analogy sentences
python get_analogy_sentences.py
python encoding_scriptwriter_analogies.py 
cd ../
source conj/encodings_analogies.sh 
  1. Run analogies
cd conj
python analogy_scriptwriter.py
cd ..
source conj/analogies.sh
cd conj

# Results will be in logs/aggregated_analogy_acc.txt 
python aggregate_analogy_acc.py 

Interventions: Lists

  1. Generate the intervention data
cd list_5_5
python intervention_generator.py
  1. Encode the intervention sentences
python get_intervention_sentences.py
python encoding_scriptwriter_interventions.py
source encodings_interventions.sh
  1. Run DISCOVER
cd list_5_5/
python decompose_scriptwriter.py
cd ..
source list_5_5/decompose_llms.sh
cd list_5_5/

# Results will be in:
#   logs/aggregated_tpe_acc.txt
#   logs/aggregated_tpe_agr.txt
#   logs/aggregated_tpe_r2.txt
python aggregate_tpe_results.py
  1. Run interventions
python intervention_scriptwriter.py
cd ../
source list_5_5/interventions.sh
cd list_5_5

# Results will be in logs/aggregated_intervention_acc.txt 
python aggregate_intervention_acc.py 

Interventions: Sentences

  1. Generate the intervention data
cd conj
python intervention_generator.py
  1. Encode the intervention sentences
python get_intervention_sentences.py 
python encoding_scriptwriter_interventions.py 
source conj/encodings_interventions.sh
  1. Run DISCOVER
cd conj/
python decompose_scriptwriter.py
cd ..
source conj/decompose_llms.sh
cd conj/

# Results will be in:
#   logs/aggregated_tpe_acc.txt
#   logs/aggregated_tpe_agr.txt
#   logs/aggregated_tpe_r2.txt
python aggregate_tpe_results.py
  1. Run interventions
python intervention_scriptwriter.py
cd ../
source conj/interventions.sh
cd conj

# Results will be in logs/aggregated_intervention_acc.txt 
python aggregate_intervention_acc.py