- Shell 65.2%
- R 27.4%
- Python 7.4%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
|
|
||
| assembly_binning_refinement | ||
| mag_gene_prediction | ||
| mapping_public_metagenomes | ||
| sag_gene_prediction | ||
| single_cell_amplified_genome | ||
| singularity_conda_containers | ||
| readme.md | ||
Context
This repository contains scripts and code snippets related to the project MICROSTORE
which aimed at better understanding the micro-eukarayotes from a fresh-water lake,
the lake Pavin in France.
Sampling was conducted a 4 dates in 2018 (April, June, September and November),
in both the oxic and permanently anoxic layer of the lake (9 m and 80 m). Samples
were split according to two size fraction : 0.2-10 µm and 10-50 µm.
Several types of data were generated from these samples:
- Metabarcoding 18S rDNA regions V4 and v9
- Metatranscriptomics of poly-A mRNA
- Metagenomics
- Single-cell Amplified Genomes
This repository focuses of the Metagenomics and Single-cell Amplified Genomes. Script related to the metabarcoding and metatranscriptomics parts are available in our repository MICROSTORE_MetaT-MetaB_Monjot_2023
Data availability
All the sequencing reads have been deposited under the umbrella BioProject PRJEB61527.
Other processed data were deposited on the Zenodo archive: 10.5281/zenodo.16738299 (this archive will be publicly accessible just before the submission of the article)
Script and code snippets
Content :
- Singularity containers recipes
- Metagenomic co-assembly pre-processing
- MAG gene prediction
- SAG gene prediction
- Mapping public metagenomes
- SAG assembly cleaning
Singularity and Conda containers recipes
To build a container, use the following command
sudo singularity build IMAGE.sif IMAGE.def
All recipes, ".def" files, are present in the directory
singularity_conda_containers:
msamtools_coverm.defmostly used to clean mappingmmseqs_avx2.def: latest version of MMseqs, for "modern" CPUsmmseqs.def: latest version of MMseqs, for CPUs that do not support avx2 instructionsaugustus_gmove_genemark.def: Augustus and GeneMark, requires extra steps, see bellow
One conda environment is listed there, conda_env_mapping.yaml. It contains
tools used for the SAGs gene prediction.
Add GeneMark in the dedicated container
This tool requires a license file. It is free to get one but it is so anoying...
- visit http://exon.gatech.edu
- SELECT "GeneMark-ES/ET/EP+", and "LINUX 64 kernel 3.10 - 5"
- fill the form at the bottom of the page. Tips, you can use a random name, affiliation and a disposable e-mail address
Make sure you have files gm_key_64.gz and gmes_linux_64_4.tar.gz in the same
directory as the Singularity augustus_gmove_genemark.def file to built the image.
Metagenomic co-assembly pre-processing
Reference of tools used below:
- Anvi'o v8 => Install it with Conda as all required tools will be present in the same environment
- Seqtk
- Metabat2
Let's start by generating a contigs database for anvi'o with the commands
shown in the code-block bellow. Briefly, these lines take a metagenomic assembly
and removes contigs smaller than 2.5 kb. It is important to provide a Fasta file
not gzipped. Then a command generate the contigs database, this steps also
run a gene prediction with Prodigal. The command anvi-run-hmms search for
genes from 3 sets of single-copy core genes (Bacteria, Archaea and Eukaryote)
present in the sequences:
CPUS=8 # a number of CPU to run steps that uses multi-threading
ASM=assembly.fa # Your metagenomic co-assembly here, MUST ENDs by ".fa"
# Clean the Fasta : rename contigs and drop small those smaller than 2.5 kb
# The report file is a correspondence between the old and new contigs names
anvi-script-reformat-fasta -o ${ASM//.fa}.2500.fa --simplify-names -l 2500 \
--prefix ASM01 --report-file ${ASM//.fa}.2500.report.tsv $ASM
# Prepare the contig database
mkdir contigs_db
ctg=contigs_db/${ASM//.fa}.2500.db
anvi-gen-contigs-database -f ${ASM//.fa}.2500.fa \
-T $CPUS -n ${ASM//.fa}.2500 -o $ctg
# Search single copy core genes sets for Bacteria, Archaea and Eukaryotes
anvi-run-hmms -c $ctg -I Bacteria_71,Protista_83,Archaea_76 -T $CPUS
# Clean the filtered fasta file. It was only used to generate anvi'o contigs db
# and the sequences are all in the contigs db
rm ${ASM//.fa}.2500.fa
Then the backmapping, i.e. mapping of the samples that were co-assembled back
on the assembly. It is a process with several steps. The first consists in generating
the index for bwa:
## Prepare the index of the assembly
bwa index assembly.2500.fa.gz
All remaining sub-steps for this work are described in the
assembly_binning_refinement/backmapping_example.sbatch. This example of script
was used for this step. It was written to run on a HPC system controlled by SLURM.
Then after the backmapping of each samples on the assembly, anvi'o has to
"profile" each sample and then merge these profiles. Let's assume you put all
BAM files in the directory mapping/:
mkdir single_profiles/
for sample in mapping/*.bam
do
NAME=$(basename $fsample | sed "s/\.bam$//") # get the sample name
# profile it
anvi-profile -c $ctg -T $CPUS -i $sample -o single_profiles/${NAME} -S $NAME
done
# Merge individual profiles in a single "merged" profile
mkdir merged_profile
# Linkage will not be performed as there are >25 000 splits (contigs)
anvi-merge -c $ctg -S MyAssemby -o merged_profile \
single_profiles/*/PROFILE.db
## Uncomment the next line to remove single profiles and save disk space
#rm -r single_profiles/*
Everything is ready to run the binning. Just a precision, as anvi'o does not advocate for a binning tool, you have to install Metabat2 by yourself (it is available in the Conda's channel BioConda ) :
# This is a wrapper for Matabat2
anvi-cluster-contigs -p merged_profile/PROFILE.db -c $ctg \
-C METABAT2 -T $CPUS --driver metabat2 --just-do-it
# This will summarise the binning.
anvi-estimate-genome-completeness -c $ctg -p merged_profile/PROFILE.db \
-C METABAT2 -o binning_metabat2.txt
The files above gives a summary of each bin, and estimates to which domain the
bin belongs to. This results is based on single copy core genes sets searched
earlier (anvi-run-hmms). Let's assume you are interested in the bin
METABAT__152, to refine and/or visualise it via the anvi'o interactive
interface, use the following command:
# here we want to visualise the bin "METABAT__152"
anvi-refine -c $ctg -p merged_profile/PROFILE.db -C METABAT2 \
-b METABAT__152
For more details on this step, please visit anvio.org and/or
read the this tutorial about anvi-refine.
MAG gene prediction
This requires two steps. First search protein hints from the public databases to help the gene prediction tools, and then in a second time run the gene prediction.
Scripts mentionned in this section are present in the directory mag_gene_prediction.
Identify proteins from close organisms, in the databases
Get the data from METdb
The data are structured by organisms and seems hard to extract alone. The script
download_metdb.sh will do the job, except for the following
organisms for which the protein were downloaded manualy :
- Fabrice-METDB_00194
- MMETSP-METDB_00388
- MMETSP-metdb_00357 (no protein prediction)
- MMETSP-METDB_00415
- MMETSP-METDB_00015
It is important to know that:
- METDB-00195 and METDB-00336 have redundant protein names, this must be fixed
- RCC-METDB_00336 and RCC-METDB_00485 comes from the same organisms, and like above, some proteins have redundant name, which must be fixed.
Prepare the mmseqs protein database
Create the database for METdb data :
mmseqs createdb data/*/*.fa.gz database/METdbDB
Then download and merge with UniRef90 :
# download the database
mmseqs databases UniRef90 UniRef90DB tmp
# concatenate UniRef90 + METdb
mmseqs concatdbs database/METdbDB uniref90DB \
database/Uniref90andMETdbDB
mmseqs concatdbs database/METdbDB_h uniref90DB_h \
database/Uniref90andMETdbDB_h
# Clean UniRef90 to save disk space, if you want
# rm uniref90DB*
Align proteins vs MAGs
Second step, identify candidate protein that could serve as protein hint.
All steps are described in the script identify_candidate_proteins.sbatch
This script returns a fasta file with proteins that will serve as hints for the gene prediction
Gene prediction : Augustus and GeneMark
Run the script run_single_MAG_gene_prediction.sh.
The Singularity image augustus_gmove_genemark.sif must be in the same directory
as the script! This script get the GeneMark license key from the Singularity
container and copy it in your home directory as $HOME/.gm_key if it is not
present.
The parameters MINCTG ($4) and SPECIES ($5) are respectivelly use by
GeneMark and Augustus. Please check their documentation for more information
All results are stored in the directory gene_call-<NameofTheMAG>.
SAG assembly cleaning
This section presents the work done to clean contaminant from SAG assembly. Scripts
metionned bellow are available in the directory single_cell_amplified_genome/.
The Shell script bellow uses Seqtk and SeqKit, make sure you have access to them. The Python script requires the ETE toolkit package. To install it in a fresh Conda environment called "ete3", use the command
conda create -n ete3 python=3
conda activate ete3
conda install -c etetoolkit ete3 ete_toolchain
ete3 build check
This part was done in 3 steps:
- First a BLASTN of the contigs vs the NT database, using parameters
-evalue 1e-3 -outfmt "6 std staxid. It is important to have thisstaxid(subject taxonomy ID) in the BLASTN output - Second, run the Python script
parse_blastn_taxonomyID.pyas follow to update the ETE toolkit taxonomy database
python parse_blastn_taxonomyID.py --update
- Third, run the script
script_blast_to_taxo.sh. There are two positionnal arguments, the BLASTN results and a sample name
The BASH script calls the Python script, and then sort the contigs according to their affiliation as Bacteria, Archaea, Eukaryota or other. There is also a step to identify Human contamination.
Note: only contigs longer than 2500 bp were kept in the assemblies for gene prediction and publication. Due to diverse reasons this filter was done after the cleaning steps (see just above). However, we highly recommand doing it as the very beginning of the assembly cleaning.
SAG gene prediction
The SAGs gene prediction were done with the same method as for the MAGs with the
difference that only GeneMark-EP+ was used. The scripts are present in the
directory sag_gene_prediction/. These scripts are given as is and may require
modifications about the paths and genomes file names.
01_align_proteins.sh, as for the MAGs, align proteins from UniRef90 and METDB on the genomes02_gene_prediction.sh, run GeneMark-EP+update_genemark_gene_name.py, update the gene names directly in the GFF3 , for safety issue back up the GFF file(s) before using this script!
Mapping public metagenomes
The script mentionned in this section are present in the directory
mapping_public_metagenomes/.
The mapping
Example of script, mapping_public_metagenome.sbatch. It was designed to run on
a HPC cluster with the SLURM scheduler, more precisely with a job array. It
takes as input a file with a list of SRA run identifiers, one per line.
The script requires a Conda environment called fastp, which contains the tool
fastp, and the Singularity container
msamtools_coverm.sif (cf the Singularity container section).
Compute MAG/SAG detection
This script transforms a number of read per contig into a number of read per MAG.
It also compute the MAG detection (breadth of coverage).
All informations used come from the samtools coverage command ran on a BAM file.
This scripts needs :
- a two-columns tab file that link a contig name to its MAG, (with no header!!) e.g.
MAG_AS_0007_000000000003 MAG_AS_0007
MAG_AS_0007_000000000005 MAG_AS_0007
MAG_AS_0005_000000004006 MAG_AS_0005
MAG_AS_0005_000000004007 MAG_AS_0005
MAG_AS_0005_000000004008 MAG_AS_0005
MAG_OS_0050_000000004639 MAG_OS_0050
MAG_OS_0050_000000004640 MAG_OS_0050
- a two-columns tab file with the length of each MAG, as follow (with headers!)
MAG length
mag000 5487655
mag005 25687963
Outputs two files :
detection_per_MAG.tsvsum_read_per_MAG.tsv
UMAP
This is an example script. It requires a table with the number of reads mapped par MAG per metagenome, the MAG taxonomy, the MAG detection per metagenome, and a metadata table of the metagenomes.