Sovereign 8s.is mirror of standardgalactic/discover
- Python 99.1%
- Shell 0.9%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
| complex | ||
| conj | ||
| letter_sequence | ||
| list_3_5 | ||
| list_5_5 | ||
| svo | ||
| .gitignore | ||
| analogies_interventions.py | ||
| compute_standardization_vectors.py | ||
| config.py | ||
| decompose.py | ||
| decompose_ensemble.py | ||
| dep_parse.py | ||
| discover_argparser.py | ||
| discover_supplement.pdf | ||
| evaluation.py | ||
| generate_encodings.py | ||
| generate_encodings_llm.py | ||
| LICENSE | ||
| models.py | ||
| README.md | ||
| role_assignment_functions.py | ||
| sh_to_scrs.py | ||
| systematic_replace_ensemble.py | ||
| token_dictionary.py | ||
| train_decoder.py | ||
| train_seq2seq.py | ||
| training.py | ||
| utils.py | ||
DISCOVER
DISsecting COmpositionality in VEctor Representations
Requirements
- Create a python venv to run code in:
python -m venv .venv
- Activate the venv
source .venv/bin/activate
- Install requirements
pip install -U pip setuptools wheel
pip install torch
pip install numpy
pip install transformers
pip install accelerate
Letter sequence experiments
Basic DISCOVER
- Generate data
cd letter_sequence/
source dataset_creation.sh
- Train models
python train_target_models_scriptwriter.py
cd ..
# Run this 10 times to get 10 reruns of each
source letter_sequence/train_target_models.sh
# Collect the results; they will be in logs/aggregated_target_model_acc.txt
cd letter_sequence/
python aggregate_target_model_accs.py
cd ..
- Run DISCOVER
cd letter_sequence/
python decompose_scriptwriter.py
cd ..
source letter_sequence/decompose.sh
# Collect the results; they will be in logs/aggregated_tpe_accs.txt
cd letter_sequence/
python aggregate_tpe_accs.py
cd ..
DISCOVER with OOD generalization
- Generate data
cd letter_sequence/
source dataset_creation_ood.sh
- Do hyperparameter search for the weight for the L1 loss term
python decompose_scriptwriter_ood_hypsearch.py
cd ..
source letter_sequence/decompose_ood_hypsearch.sh
cd letter_sequence/
python aggregate_tpe_accs_ood_hypsearch.py
- Run DISCOVER
python decompose_scriptwriter_ood.py
cd ..
source letter_sequence/decompose_ood.sh
# Collect the results; they will be in logs/aggregated_tpe_accs_ood.txt
cd letter_sequence/
python aggregate_tpe_accs_ood.py
Constituent surgery / causal interventions
- Generate interventions
cd letter_sequence/
source intervention_generation.sh
- Run the interventions
python intervention_scriptwriter.py
cd ..
source letter_sequence/interventions.sh
- Aggregate the results; they will be in logs/aggregated_intervention_acc.txt
cd letter_sequence/
python aggregate_interventions.py
Analogies
- Generate analogies
cd letter_sequence/
source analogy_generation.sh
- Run the analogies
python analogy_scriptwriter.py
cd ..
source letter_sequence/analogies.sh
- Aggregate the results; they will be in logs/aggregated_analogy_acc.txt
cd letter_sequence/
python aggregate_analogies.py
Period encoding experiments
SVO sentences
- Generate the sentences:
cd svo
python svo_sentence_generator.py
- Encode the sentences
python encoding_scriptwriter.py
cd ..
source svo/encodings.sh
- Compute means and standard deviations for z-scoring
cd svo
python standardization_scriptwriter.py
cd ..
source svo/standardization.sh
- Train decoder
cd svo
python decoder_scriptwriter.py
cd ..
source svo/hidden_decoder.sh
cd svo
# Results will be in logs/aggregated_hidden_decoder_accs.txt
python aggregate_decoder_acc.py
- Run DISCOVER
cd svo/
python decompose_scriptwriter.py
cd ..
source svo/decompose_llms.sh
cd svo
# Results will be in:
# logs/aggregated_tpe_acc.txt
# logs/aggregated_tpe_agr.txt
# logs/aggregated_tpe_r2.txt
python aggregate_tpe_results.py
Lists
- Generate the lists
cd list_3_5
python list_generator.py
- Encode the lists
python encoding_scriptwriter.py
cd ..
source list_3_5/encodings.sh
- Compute means and standard deviations for z-scoring
cd list_3_5
python standardization_scriptwriter.py
cd ..
source list_3_5/standardization.sh
- Train decoder
cd list_3_5
python decoder_scriptwriter.py
cd ..
source list_3_5/hidden_decoder.sh
# Results will be in logs/aggregated_hidden_decoder_accs.txt
python aggregate_decoder_acc.py
- Run DISCOVER
cd list_3_5/
python decompose_scriptwriter.py
cd ..
source list_3_5/decompose_llms.sh
cd list_3_5/
# Results will be in:
# logs/aggregated_tpe_acc.txt
# logs/aggregated_tpe_agr.txt
# logs/aggregated_tpe_r2.txt
python aggregate_tpe_results.py
Complex sentences
- Generate the sentences:
cd complex
python sentence_generator.py
- Encode the sentences
python encoding_scriptwriter.py
cd ..
source complex/encodings.sh
- Compute means and standard deviations for z-scoring
cd complex
python standardization_scriptwriter.py
cd ..
source complex/standardization.sh
- Train decoder
cd complex
python decoder_scriptwriter.py
cd ..
source complex/hidden_decoder.sh
# Results will be in logs/aggregated_hidden_decoder_accs.txt
python aggregate_decoder_acc.py
- Run DISCOVER
cd complex/
python decompose_scriptwriter.py
cd ..
source complex/decompose_llms.sh
cd complex/
# Results will be in:
# logs/aggregated_tpe_acc.txt
# logs/aggregated_tpe_agr.txt
# logs/aggregated_tpe_r2.txt
python aggregate_tpe_results.py
OOD: Lists
- Generate the sentences
cd list_5_5
python list_generator.py
python list_generator_ood.py
- Encode the sentences
python encoding_scriptwriter.py
python encoding_scriptwriter_ood_split.py
cd ..
source list_5_5/encodings.sh
source list_5_5/encodings_ood_split.sh
- Compute means and standard deviations for z-scoring
cd list_5_5
python standardization_scriptwriter.py
python standardization_scriptwriter_ood_split.py
cd ..
source list_5_5/standardization.sh
source list_5_5/standardization_ood_split.sh
- Train decoder
cd list_5_5/
python decoder_scriptwriter.py
cd ..
source list_5_5/hidden_decoder.sh
# Results will be in logs/aggregated_hidden_decoder_accs.txt
python aggregate_decoder_acc.py
- Run a hypothesis search for the L1 lambda hyperparameter
cd list_5_5
python decompose_scriptwriter_l1.py
cd ..
source list_5_5/decompose_llms_l1_hypsearch.sh
- Run DISCOVER
cd list_5_5/
python decompose_scriptwriter_ood.py
cd ..
source list_5_5/decompose_llms_ood.sh
cd list_5_5/
# Results will be in:
# logs/aggregated_tpe_acc.txt
# logs/aggregated_tpe_agr.txt
# logs/aggregated_tpe_r2.txt
python aggregate_tpe_results_ood.py
OOD: Sentences
- Generate the sentences
cd conj
python sentence_generator.py
python sentence_generator_ood_sameverb.py
- Encode the sentences
python encoding_scriptwriter.py
python encoding_scriptwriter_oodsameverb.py
cd ..
source conj/encodings.sh
source conj/encodings_ood_sameverb.sh
- Compute means and standard deviations for z-scoring
cd conj
python standardization_scriptwriter.py
python standardization_scriptwriter_oodsameverb.py
cd ..
source conj/standardization.sh
source conj/standardization_oodsameverb.sh
- Train decoder
cd conj/
python decoder_scriptwriter.py
cd ..
source conj/hidden_decoder.sh
- Run DISCOVER
cd conj/
python decompose_scriptwriter_ood.py
cd ..
source conj/decompose_llms_ood_sameverb.sh
cd conj/
# Results will be in:
# logs/aggregated_tpe_acc.txt
# logs/aggregated_tpe_agr.txt
# logs/aggregated_tpe_r2.txt
python aggregate_tpe_results_ood.py
Analogies: Lists
- Generate the analogies
cd list_5_5
python analogy_generator.py
- Encode the analogy sentences
python get_analogy_sentences.py
python encoding_scriptwriter_analogies.py
cd ..
source list_5_5/encodings_analogies.sh
- Run analogies
cd list_5_5
python analogy_scriptwriter.py
cd ..
source list_5_5/analogies.sh
cd list_5_5
# Results will be in logs/aggregated_analogy_acc.txt
python aggregate_analogy_acc.py
Analogies: Sentences
- Generate the analogies
cd conj
python analogy_generator.py
- Encode the analogy sentences
python get_analogy_sentences.py
python encoding_scriptwriter_analogies.py
cd ../
source conj/encodings_analogies.sh
- Run analogies
cd conj
python analogy_scriptwriter.py
cd ..
source conj/analogies.sh
cd conj
# Results will be in logs/aggregated_analogy_acc.txt
python aggregate_analogy_acc.py
Interventions: Lists
- Generate the intervention data
cd list_5_5
python intervention_generator.py
- Encode the intervention sentences
python get_intervention_sentences.py
python encoding_scriptwriter_interventions.py
source encodings_interventions.sh
- Run DISCOVER
cd list_5_5/
python decompose_scriptwriter.py
cd ..
source list_5_5/decompose_llms.sh
cd list_5_5/
# Results will be in:
# logs/aggregated_tpe_acc.txt
# logs/aggregated_tpe_agr.txt
# logs/aggregated_tpe_r2.txt
python aggregate_tpe_results.py
- Run interventions
python intervention_scriptwriter.py
cd ../
source list_5_5/interventions.sh
cd list_5_5
# Results will be in logs/aggregated_intervention_acc.txt
python aggregate_intervention_acc.py
Interventions: Sentences
- Generate the intervention data
cd conj
python intervention_generator.py
- Encode the intervention sentences
python get_intervention_sentences.py
python encoding_scriptwriter_interventions.py
source conj/encodings_interventions.sh
- Run DISCOVER
cd conj/
python decompose_scriptwriter.py
cd ..
source conj/decompose_llms.sh
cd conj/
# Results will be in:
# logs/aggregated_tpe_acc.txt
# logs/aggregated_tpe_agr.txt
# logs/aggregated_tpe_r2.txt
python aggregate_tpe_results.py
- Run interventions
python intervention_scriptwriter.py
cd ../
source conj/interventions.sh
cd conj
# Results will be in logs/aggregated_intervention_acc.txt
python aggregate_intervention_acc.py