- Python 99.7%
- Shell 0.3%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
|
|
||
| callable_genome | ||
| indel_calling | ||
| manifests | ||
| mnm_validation | ||
| phasing | ||
| sex_chromosomes | ||
| snv_calling | ||
| LICENSE | ||
| README.md | ||
Long-read sequencing of trios reveals increased germline and postzygotic mutation rates in repetitive DNA
This repository contains six Snakemake pipelines to call, phase, and validate de novo variants and to calculate callable genome size on the T2T-CHM13 v2.0 reference. Pipelines cover autosomes and sex chromosomes and are intended for use with multi-platform sequencing (HiFi, ONT, Illumina).
For full methodological details, see our arXiv submission: link to arXiv manuscript.
Notes:
- Each pipeline has its own
config.jsonspecifying reference paths and manifests. Update these paths for your environment. - Example manifests are provided under
manifests/.
Pipelines
Pipeline-specific expected outputs and typical runtimes (assuming resources encoded in each Snakefile):
-
callable_genome/
- Expected output: BED files of accessible regions and per-sample callable genome size TSVs under
results/. - Typical runtime: 2–3 weeks.
- Expected output: BED files of accessible regions and per-sample callable genome size TSVs under
-
snv_calling/
- Expected output: validated de novo SNVs under
results/(for example,results/{family}_{sample}.snv_calls.tsv). - Typical runtime: 4–8 hours.
- Expected output: validated de novo SNVs under
-
indel_calling/
- Expected output: validated de novo indels under
validated_calls/. - Typical runtime: 4–8 hours.
- Expected output: validated de novo indels under
-
phasing/
- Expected output: per-sample haplotype assignments and origin (postzygotic/germline) integration under
results/. - Typical runtime: 2–4 hours.
- Expected output: per-sample haplotype assignments and origin (postzygotic/germline) integration under
-
mnm_validation/
- Expected output: validated de novo multinucleotide mutations under
validated_mnms/. - Typical runtime: 1–2 days.
- Expected output: validated de novo multinucleotide mutations under
-
sex_chromosomes/
- Expected output: validated sex-chromosome DNMs for males and females (SNVs and indels) under each subpipeline’s outputs.
- Typical runtime: 2–4 hours.
Notes and tips:
- Some pipelines consume outputs of others. For example,
phasing/expects SNV results fromsnv_calling/(seephasing/Snakefile), andindel_calling/expects prior indel candidates generated bysnv_calling/; ensure upstream outputs exist or paths are adjusted accordingly.
System requirements
- Operating system: Pipelines were written and tested on Ubuntu 22.04. Other operating systems have not been tested.
- Required software:
- Miniconda v4.12.0 (to enable per-rule conda environments).
- Snakemake 4.12.0 (other versions have not been tested with these workflows).
- The pipelines declare software dependencies in their
envs/env.yamlfiles and will create/activate those environments automatically when run with--use-conda.
- Hardware: No non-standard hardware is required. Jobs can be run locally or on a cluster. Example commands below include an SGE/DRMAA submission alias; cluster use is optional.
Installation
The steps below install Miniconda and Snakemake on Ubuntu 22.04 using the Bioconda channels.
# Install Miniconda 4.12.0 (Linux x86_64)
wget https://repo.anaconda.com/miniconda/Miniconda3-py39_4.12.0-Linux-x86_64.sh -O miniconda.sh
bash miniconda.sh -b -p "$HOME/miniconda3"
eval "$($HOME/miniconda3/bin/conda shell.bash hook)"
conda init bash # or 'zsh' if you use zsh
# Configure Bioconda channels
conda config --add channels defaults
conda config --add channels bioconda
conda config --add channels conda-forge
# Create an environment with Snakemake 4.12.0
conda create -y -n snakemake-4.12.0 snakemake=4.12.0
conda activate snakemake-4.12.0
# Optional (only if submitting via DRMAA to an SGE grid):
# conda install -y python-drmaa
- Environment provisioning: When you run any pipeline with
--use-conda, Snakemake will automatically create and manage per-rule environments from theenvs/env.yamlfiles. - Typical install time: ~10–20 minutes on a standard desktop with broadband internet.
Demo Instructions
All commands below are executed from a pipeline directory (for example, snv_calling/ or callable_genome/). Before running, edit that pipeline’s config.json to point to your reference, manifests, and other inputs. Example manifests are in manifests/.
You can run locally or submit to an SGE cluster. To submit via DRMAA on SGE, define this alias (from the repo root or any pipeline directory):
alias snakesub='mkdir -p log; snakemake --ri --jobname "{rulename}.{jobid}" --drmaa " -V -cwd -j y -o ./log -e ./log -l h_rt={resources.hrs}:00:00 -l mfree={resources.mem}G -pe serial {threads} -w n -S /bin/bash" -w 60'
Then submit with:
snakesub -j [number of jobs] --use-conda
- Cluster integration beyond SGE/DRMAA is not provided here. If running on a different scheduler, adapt the submission command accordingly.
Although we cannot provide demo data from our samples, you can test these pipelines using trio data from Genome in a Bottle (GIAB) by following these instructions for the GIAB Ashkenazi trio (HG002—son, HG003—father, HG004—mother), all aligned to GRCh38.
- Download the BAMs
You will need aligned HiFi, ONT, and Illumina BAMs for each memeber of the trio. Run the following commands to fetch the data into your directory:
# HG002 (son)
wget ftp://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data/AshkenazimTrio/HG002_NA24385_son/NIST_NA24385_HG002_HiSeq300x/HG002_HiSeq300x_GRCh38.bam
wget ftp://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data/AshkenazimTrio/HG002_NA24385_son/NIST_NA24385_HG002_HiSeq300x/HG002_HiSeq300x_GRCh38.bam.bai
wget ftp://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data/AshkenazimTrio/HG002_NA24385_son/PacBio_CCS_15kb/HG002_PacBio_CCS_15kb_GRCh38.bam
wget ftp://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data/AshkenazimTrio/HG002_NA24385_son/PacBio_CCS_15kb/HG002_PacBio_CCS_15kb_GRCh38.bam.bai
wget ftp://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data/AshkenazimTrio/HG002_NA24385_son/ONT_UL/HG002_ONT_UL_GRCh38.bam
wget ftp://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data/AshkenazimTrio/HG002_NA24385_son/ONT_UL/HG002_ONT_UL_GRCh38.bam.bai
# HG003 (father)
wget ftp://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data/AshkenazimTrio/HG003_NA24149_father/Illumina300XWGS/HG003_Illumina_300x_GRCh38.bam
wget ftp://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data/AshkenazimTrio/HG003_NA24149_father/Illumina300XWGS/HG003_Illumina_300x_GRCh38.bam.bai
wget ftp://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data/AshkenazimTrio/HG003_NA24149_father/PacBio_CCS_15kb/HG003_PacBio_CCS_15kb_GRCh38.bam
wget ftp://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data/AshkenazimTrio/HG003_NA24149_father/PacBio_CCS_15kb/HG003_PacBio_CCS_15kb_GRCh38.bam.bai
wget ftp://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data/AshkenazimTrio/HG003_NA24149_father/ONT_UL/HG003_ONT_UL_GRCh38.bam
wget ftp://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data/AshkenazimTrio/HG003_NA24149_father/ONT_UL/HG003_ONT_UL_GRCh38.bam.bai
# HG004 (mother)
wget ftp://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data/AshkenazimTrio/HG004_NA24143_mother/Illumina300XWGS/HG004_Illumina_300x_GRCh38.bam
wget ftp://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data/AshkenazimTrio/HG004_NA24143_mother/Illumina300XWGS/HG004_Illumina_300x_GRCh38.bam.bai
wget ftp://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data/AshkenazimTrio/HG004_NA24143_mother/PacBio_CCS_15kb/HG004_PacBio_CCS_15kb_GRCh38.bam
wget ftp://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data/AshkenazimTrio/HG004_NA24143_mother/PacBio_CCS_15kb/HG004_PacBio_CCS_15kb_GRCh38.bam.bai
wget ftp://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data/AshkenazimTrio/HG004_NA24143_mother/ONT_UL/HG004_ONT_UL_GRCh38.bam
wget ftp://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data/AshkenazimTrio/HG004_NA24143_mother/ONT_UL/HG004_ONT_UL_GRCh38.bam.bai
- Download the VCFs
wget ftp://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/release/AshkenazimTrio/HG002_GRCh38/latest/HG002_GRCh38_1_22_v4.2.1_benchmark.vcf.gz
wget ftp://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/release/AshkenazimTrio/HG002_GRCh38/latest/HG002_GRCh38_1_22_v4.2.1_benchmark.vcf.gz.tbi
wget ftp://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/release/AshkenazimTrio/HG003_GRCh38/latest/HG003_GRCh38_1_22_v4.2.1_benchmark.vcf.gz
wget ftp://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/release/AshkenazimTrio/HG003_GRCh38/latest/HG003_GRCh38_1_22_v4.2.1_benchmark.vcf.gz.tbi
wget ftp://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/release/AshkenazimTrio/HG004_GRCh38/latest/HG004_GRCh38_1_22_v4.2.1_benchmark.vcf.gz
wget ftp://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/release/AshkenazimTrio/HG004_GRCh38/latest/HG004_GRCh38_1_22_v4.2.1_benchmark.vcf.gz.tbi
- Combine VCFs into one for the trio
Although GIAB does not provide separate GATK and DeepVariant files like these pipelines expect, you can use just one VCF by listing the same file twice in vcf_manifest.tab. Generate this file by merging variant data from the three VCFs.
bcftools merge -m all -Oz \
-o GIAB_trio_GRCh38_merged.vcf.gz \
HG002.vcf.gz HG003.vcf.gz HG004.vcf.gz
tabix -p vcf GIAB_trio_GRCh38_merged.vcf.gz
- Reference Genome (GRCh38)
Make sure to use GRCh38 as the reference - you will need to update this in your config.json files. You can download the same version used by GIAB:
wget ftp://ftp-trace.ncbi.nlm.nih.gov/giab/release/references/GRCh38_no_alt_analysis_set.fna.gz
gunzip GRCh38_no_alt_analysis_set.fna.gz
samtools faidx GRCh38_no_alt_analysis_set.fna
Once you have obtained these data, create manifests that match those in manifests/, update your config.json files, and you should be good to go! Happy DNM hunting!
Reproduction Instructions
- The original data used for analysis are available in the SFARI Base under accession SFARI_DS714840 and in the NIH NDA Collection ID 3780. Access may require data use agreements through those portals.
- Update each pipeline’s
config.jsonto point to your local copies of the reference (T2T-CHM13v2.fasta), manifests (for example,manifests/bam_manifest.tab,manifests/vcf_manifest.tab,manifests/hifi_sources.tsv), and any other required inputs (for example, TRF/RepeatMasker BEDs).
Contact
Please open an issue or pull request with questions or improvements.