- R 75.1%
- Shell 24.9%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
| analysis | ||
| config | ||
| core | ||
| data/templates | ||
| diagnostic | ||
| docs | ||
| modules | ||
| plots | ||
| preparation | ||
| utils | ||
| .gitignore | ||
RNAChromatin Analysis Pipeline
A comprehensive pipeline for analyzing RNA-chromatin sequencing datasets, including ChIRP-seq, CHART-seq, and RAP-seq data. This pipeline processes raw sequencing data through peak calling, motif discovery, and k-mer analysis to characterize RNA-chromatin interactions.
Table of Contents
- Overview
- Requirements
- Directory Structure
- Pipeline Workflow
- Running the Pipeline
- Input Files
- Output Files
- Script Descriptions
Overview
This pipeline performs the following main analyses:
- Read alignment to reference genomes using Bowtie2
- Peak calling using MACS2
- Motif discovery using HOMER and STREME
- K-mer analysis to identify RNA-binding sequences within peaks
- Comparative analysis of even/odd probe pairs
- Antisense/sense probe overlap analysis for control validation
- Visualization of results
The pipeline is designed to run on an LSF (Load Sharing Facility) HPC cluster.
Requirements
HPC Environment
- LSF job scheduler (bsub, bjobs commands)
- Module system for loading software
Software Dependencies
| Software | Version | Purpose |
|---|---|---|
| Bowtie2 | 2.5.1 | Read alignment |
| SAMtools | 1.21 | BAM file processing |
| BEDTools | 2.30.0 | Genomic interval operations |
| MACS2 | 2.2.9.1 | Peak calling |
| HOMER | 4.11 | Motif discovery |
| MEME Suite | 5.5.0 | STREME motif discovery, FIMO |
| deepTools | 3.5.2 | Coverage track generation |
| BLAT | 3.7 | Probe alignment |
| bedops | 2.4.41 | BED/PSL format conversion |
| Picard | 2.25.1 | BAM processing utilities |
| SRA-Toolkit | 3.0.10 | SRA data download |
| R | 4.3.1 | Statistical analysis |
| Java (JRE) | 21.0.2 | Custom k-mer counting and shuffling tools |
R Packages
Data Manipulation:
- stringr, tidyr, dplyr, magrittr, readr, tibble, purrr
Bioinformatics:
- Biostrings, GenomicRanges, IRanges, rtracklayer
Visualization:
- ggplot2, hrbrthemes, patchwork, ggh4x, tidyplot, viridis, geomtextpath, ggseqlogo, ggpattern, ggrepel, cowplot, grid, gridExtra
Other:
- GetoptLong (string interpolation and command-line parsing)
- readxl (Excel file reading)
- XML, reutils (NCBI E-utilities)
External Custom Scripts
The pipeline depends on custom Java tools. Paths are configured in config/paths.sh and config/paths.R:
$KMER_COUNT_TOOL - scripts.lincs.CountKmers (k-mer counting and positions recording)
$ANALYZE_OVERLAPS_TOOL - scripts.lincs.AnalyzePeakOverlaps (peak overlap analysis)
$SHUFFLE_SEQUENCES_TOOL - scripts.lincs.util.ShuffleSequences (sequence shuffling)
Directory Structure
The pipeline expects the following directory structure:
chirp/
├── code/ # This repository
│ ├── core/ # Main pipeline orchestration
│ │ ├── pipeline_orchestrator.R # Main orchestration script
│ │ ├── pipeline_orchestrator_subsample.R # Subsample analysis orchestrator
│ │ └── job_submission.sh # LSF job submission script
│ ├── modules/ # Shell processing modules
│ │ ├── peak_processing_unified.sh # Unified peak processing (MACS2 + published)
│ │ ├── paired_peaks_processing.sh # Paired sample analysis
│ │ └── common_peaks.sh # Common peaks detection
│ ├── analysis/ # Analysis scripts
│ │ ├── kmer/ # K-mer analysis scripts
│ │ │ ├── kmer_analysis_paired.R
│ │ │ ├── kmer_analysis_single.R
│ │ │ ├── kmer_goi_hits_analysis.R
│ │ │ ├── longest_kmer_processing.R
│ │ │ ├── longest_kmer_strand.R
│ │ │ ├── longest_kmer_positions_merge.R
│ │ │ ├── longest_kmer_summary_merge.R
│ │ │ ├── split_kmer_count.sh
│ │ │ ├── split_longest_kmer.sh
│ │ │ └── split_longest_kmer_processing.R
│ │ ├── reads/ # Read processing scripts
│ │ │ ├── extract_kmer_intersecting_reads.sh
│ │ │ ├── extract_kmer_intersecting_reads_by_chr.sh
│ │ │ ├── extract_kmer_flanking_reads.sh
│ │ │ ├── process_kmer_read_intersection.R
│ │ │ ├── process_kmer_read_intersection_by_chr.R
│ │ │ ├── process_flanking_lk_reads.sh
│ │ │ ├── process_flanking_reads.sh
│ │ │ ├── shuffle_read_names.sh
│ │ │ └── count_bam_reads.sh
│ │ └── other/ # Other analysis scripts
│ │ ├── antisense_sense_overlap.sh
│ │ ├── antisense_sense_overlap.R
│ │ └── common_peak_calling.R
│ ├── preparation/ # Data preparation scripts
│ │ ├── data_preparation.R
│ │ ├── transcript_preparation.sh
│ │ ├── shuffle_probes_positional.sh
│ │ └── setup_directories.sh
│ ├── utils/ # Utility functions
│ │ ├── functions.R
│ │ └── overlap_summary.sh
│ ├── config/ # Configuration files
│ │ ├── paths.R
│ │ └── paths.sh
│ ├── plots/ # Visualization scripts
│ │ ├── plot_antisense_sense_overlap.R
│ │ ├── plot_kmer_goi_hits.R
│ │ ├── plot_longest_kmer_heatmap.R
│ │ ├── plot_method_comparison.R
│ │ ├── plot_motif_across_transcript.R
│ │ ├── plot_peak_counts.R
│ │ ├── plot_peak_overlap.R
│ │ ├── plot_read_flanking.R
│ │ ├── plot_read_intersection.R
│ │ ├── plot_read_intersection_by_chr.R
│ │ ├── plot_read_intersection_published.R
│ │ ├── plot_read_peak_correlation.R
│ │ ├── plot_signal_over_background.R
│ │ ├── plot_study_characteristics.R
│ │ └── plot_subsample_peak_analysis.R
│ ├── docs/ # Documentation
│ ├── data/ # Data files and templates
│ └── diagnostic/ # Diagnostic tools
├── input/ # Configuration files
│ ├── config.xlsx # Species-specific configuration
│ ├── sample_table.xlsx # Sample metadata
│ ├── probes.xlsx # Probe sequences
│ └── published_peaks_input.xlsx # Published peaks info
├── commands/ # Generated job tables
├── figures/ # Cross-species figures
├── human/ # Human species data
│ ├── reads/ # FASTQ files
│ ├── alignment/ # BAM files
│ ├── peaks/
│ │ ├── current/ # MACS2 peaks
│ │ ├── published/ # Published peaks
│ │ ├── pairs/ # Even/odd overlap analysis
│ │ └── common/ # Common peaks across studies
│ ├── coverage/ # BigWig files
│ ├── probes/ # Probe FASTA files
│ ├── transcripts/ # Target transcript sequences
│ ├── auxiliary/ # Reference files
│ ├── jobs/ # Job output files
│ └── plots/ # Analysis plots
└── mouse/ # Mouse species data (same structure)
Pipeline Workflow
Phase 1: Data Preparation
preparation/data_preparation.R
├── Download SRA metadata (efetch)
├── Create sample tables
├── Generate probe FASTA files
├── Create dinucleotide-shuffled probe controls (non-RAP)
├── Generate transcript FASTA files
├── Create positionally-shuffled probe controls (RAP)
└── BLAT probes to genome and transcripts
Phase 2: Read Processing
core/pipeline_orchestrator.R (generates job tables)
├── bowtie2 → Read alignment
├── blFilter → Blacklist region removal
├── bamReadCount → Read counting
├── bamIndex → BAM indexing
├── bamCoverage → Coverage track generation
└── chrCount → Chromosome read counts
Phase 3: Peak Calling
├── macs2 → Call peaks
└── modules/peak_processing_unified.sh → For each sample:
├── Sort peaks by score
├── Generate FASTA sequences
├── Calculate nucleotide content
├── Run HOMER motif discovery
├── Run STREME motif discovery
├── Run FIMO on transcripts
└── Count k-mers
Phase 4: Comparative Analysis
├── modules/common_peaks.sh → Find peaks common across studies
├── modules/paired_peaks_processing.sh → Analyze even/odd probe pairs:
│ ├── Find intersecting peaks
│ ├── Create union peaks
│ ├── Run motif analysis
│ └── Count k-mers
└── analysis/other/antisense_sense_overlap.sh → Antisense/sense peak overlap
Phase 5: K-mer and Read Analysis
├── analysis/kmer/kmer_analysis_paired.R → K-mer analysis for paired samples
├── analysis/kmer/kmer_analysis_single.R → K-mer analysis for single samples
├── analysis/kmer/kmer_goi_hits_analysis.R → K-mer hits in genes of interest
├── analysis/kmer/longest_kmer_strand.R → Strand-specific k-mer analysis
├── analysis/reads/extract_kmer_intersecting_reads.sh → Extract reads overlapping k-mers
├── analysis/reads/extract_kmer_flanking_reads.sh → Extract reads flanking k-mers
├── analysis/reads/process_kmer_read_intersection.R → Process read intersections
└── plots/plot_motif_across_transcript.R → Motif distribution on transcripts
Phase 6: Visualization
plots/
├── plot_antisense_sense_overlap.R → Antisense/sense probe overlap
├── plot_kmer_goi_hits.R → K-mer hits in genes of interest
├── plot_longest_kmer_heatmap.R → Longest k-mer heatmap across peak sets
├── plot_method_comparison.R → Deeper dive into (CHART/RAP with a focus on Xist)
├── plot_motif_across_transcript.R → Motif distribution on transcripts
├── plot_peak_counts.R → Peak count visualizations
├── plot_peak_overlap.R → Peak overlap diagrams
├── plot_read_flanking.R → Longest K-mer flanking read distributions
├── plot_read_intersection.R → Longest K-mer read intersection with k-mers
├── plot_read_intersection_by_chr.R → Longest K-mer read intersection by chromosome
├── plot_read_intersection_published.R → Longest K-mer read intersection for published peaks
├── plot_read_peak_correlation.R → Read count vs peak count correlation
├── plot_signal_over_background.R → K-mer signal vs background analysis
├── plot_study_characteristics.R → Study metadata characteristics
└── plot_subsample_peak_analysis.R → Subsample peak retention/trends
Running the Pipeline
Step 1: Prepare Configuration Files
-
Create
input/config.xlsxwith species-specific paths:species: human/mousegenome_assembly: hg19/mm10genome_fasta: Path to genome FASTAbowtie_build: Path to Bowtie2 indexblacklist: Path to blacklist BED filegenome_code: MACS2 genome code (hs/mm)
-
Create
input/sample_table.xlsxwith sample metadata
Step 2: Run Data Preparation
# In R:
source("preparation/data_preparation.R")
This will:
- Download SRA metadata
- Create probe FASTA files
- Generate dinucleotide-shuffled controls (non-RAP)
- Prepare transcript files
- Generate positionally-shuffled controls (RAP)
- BLAT probes to genome and transcripts
Generated job tables in commands/:
download_srr_job_table.tsv- SRA read downloadshuffle_probes_job_table.tsv- Dinucleotide probe shufflingrap_shuffle_probes_job_table.tsv- Positional probe shuffling (RAP)probe_blat_genome_job_table.tsv- BLAT probes to genomeprobe_blat_transcript_job_table.tsv- BLAT probes to transcripts
Step 3: Generate and Submit Job Tables
# In R:
source("core/pipeline_orchestrator.R")
This generates TSV files in commands/ directory:
current_job_table.tsv- Main alignment and peak calling jobscommon_peaks_job_table.tsv- Common peaks analysisjob_table_published_peak_module.tsv- Published peaks processingjob_table_pair_peak_module.tsv- Paired sample analysis (even\odd matched samples)analyze_kmer_job_table.tsv- K-mer analysis jobskmer_goi_hits_job_table.tsv- K-mer GOI hits analysislongest_kmer_reads_job_table.tsv- Read-longest kmer intersection extractionlongest_kmer_reads_published_job_table.tsv- Read-longest kmer intersection (published)motif_transcript_plot_job_table.tsv- Motif across transcript plots
The antisense/sense overlap analysis runs directly via system() at the end of the orchestrator.
Step 4: Submit Jobs to LSF
# Submit main pipeline jobs
./code/core/job_submission.sh /path/to/chirp/commands/current_job_table.tsv
# After macs2 jobs complete, submit downstream analyses
./code/core/job_submission.sh /path/to/chirp/commands/common_peaks_job_table.tsv
./code/core/job_submission.sh /path/to/chirp/commands/job_table_pair_peak_module.tsv
./code/core/job_submission.sh /path/to/chirp/commands/analyze_kmer_job_table.tsv
./code/core/job_submission.sh /path/to/chirp/commands/kmer_goi_hits_job_table.tsv
./code/core/job_submission.sh /path/to/chirp/commands/longest_kmer_reads_job_table.tsv
Job Submission Script Usage
./code/core/job_submission.sh <job_table.tsv> [ERROR_FILE] [FORCE_SUBMIT]
# Arguments:
# job_table.tsv - TSV file with job specifications
# ERROR_FILE - TRUE/FALSE to create error files (default: FALSE)
# FORCE_SUBMIT - TRUE/FALSE to resubmit completed jobs (default: FALSE)
Input Files
Required Configuration Files
| File | Description |
|---|---|
input/config.xlsx |
Species-specific paths and parameters |
input/sample_table.xlsx |
Sample metadata (GSE, GSM, conditions) |
input/published_peaks_input.xlsx |
Published peak file information |
input/probes.xlsx |
Probe sequences for each study |
Required Reference Files
| File | Location | Description |
|---|---|---|
| Genome FASTA | {species}/auxiliary/{assembly}.fa |
Reference genome |
| Blacklist BED | {species}/auxiliary/blacklist.bed |
Regions to exclude |
| Bowtie2 index | As specified in config | Alignment index |
| Transcript BED | {species}/auxiliary/txCoords12.bed |
Target transcript coordinates (BED12) |
Output Files
Per-Sample Outputs
| Directory | Contents |
|---|---|
alignment/ |
{GSM}.bam, {GSM}_blFilter.bam |
coverage/ |
{GSM}.bw (BigWig) |
peaks/current/peaks/ |
{GSM}_peaks.narrowPeak |
peaks/current/peaks_fasta/ |
Peak sequences (FASTA) |
peaks/current/peaks_nuc/ |
Nucleotide content |
peaks/current/homer/{GSM}/ |
HOMER motif results |
peaks/current/streme/{GSM}/ |
STREME motif results |
peaks/current/kmer/ |
K-mer count files |
peaks/current/longest_kmer/ |
Longest k-mer analysis |
Paired Analysis Outputs
| Directory | Contents |
|---|---|
peaks/pairs/peaks/ |
Intersect and union BED files |
peaks/pairs/homer/ |
Motif analysis on overlaps |
peaks/pairs/analyze_overlaps/ |
Overlap statistics |
Common Peaks Outputs
| Directory | Contents |
|---|---|
peaks/common/ |
Common peaks across studies |
Cross-Species Figures
| Directory | Contents |
|---|---|
figures/ |
antisense_sense_overlap.pdf, subsample_peak_analysis.pdf, reads_peaks_correlation_plot.pdf |
Script Descriptions
Main Orchestration (core/)
| Script | Purpose |
|---|---|
pipeline_orchestrator.R |
Generates job tables for entire pipeline |
pipeline_orchestrator_subsample.R |
Generates job tables for subsample analysis |
job_submission.sh |
Submits jobs from TSV tables to LSF |
Data Preparation (preparation/)
| Script | Purpose |
|---|---|
data_preparation.R |
SRA metadata download, probe generation, shuffled controls, transcript prep |
transcript_preparation.sh |
Generate transcript FASTA/BED files from txCoords12.bed |
shuffle_probes_positional.sh |
Positional probe shuffling for RAP (BLAT-based) |
setup_directories.sh |
Create required directory structure |
Utility Functions (utils/)
| Script | Purpose |
|---|---|
functions.R |
Core utility functions (validation, k-mer helpers, sample name repair, theming) |
overlap_summary.sh |
Overlap summary helper |
Peak Processing Modules (modules/)
| Script | Purpose |
|---|---|
peak_processing_unified.sh |
Unified peak processing: handles both MACS2-called and published peaks, motifs, k-mers |
peak_processing.sh |
Standard peak processing |
peak_processing_published.sh |
Published peak processing |
paired_peaks_processing.sh |
Analyze even/odd paired samples |
common_peaks.sh |
Find common peaks across studies |
K-mer Analysis (analysis/kmer/)
| Script | Purpose |
|---|---|
kmer_analysis_paired.R |
K-mer analysis for paired (even/odd) samples |
kmer_analysis_single.R |
K-mer analysis for single samples |
kmer_goi_hits_analysis.R |
K-mer hits within genes of interest (GOI) |
longest_kmer_processing.R |
Find longest matching k-mers per peak |
longest_kmer_strand.R |
Determine k-mer strand via probe alignment |
longest_kmer_positions_merge.R |
Merge k-mer position tables |
longest_kmer_summary_merge.R |
Merge k-mer summary tables |
split_kmer_count.sh |
Split large peak sets for k-mer counting |
split_longest_kmer.sh |
Split largest k-mer FASTA files |
split_longest_kmer_processing.R |
Process split longest k-mer files |
Read Processing (analysis/reads/)
| Script | Purpose |
|---|---|
extract_kmer_intersecting_reads.sh |
Extract reads overlapping k-mers |
extract_kmer_intersecting_reads_by_chr.sh |
Extract reads overlapping k-mers by chromosome |
extract_kmer_flanking_reads.sh |
Extract reads flanking k-mers |
process_kmer_read_intersection.R |
Summarise read-kmer intersection distances |
process_kmer_read_intersection_by_chr.R |
Summarise read-kmer intersections by chromosome |
process_flanking_lk_reads.sh |
Process flanking longest k-mer reads |
process_flanking_reads.sh |
Process flanking reads |
shuffle_read_names.sh |
Shuffle read names for control analysis |
count_bam_reads.sh |
Count reads in BAM files |
Other Analysis (analysis/other/)
| Script | Purpose |
|---|---|
antisense_sense_overlap.sh |
Compute antisense/sense probe peak overlaps |
common_peak_calling.R |
Common peak calling across studies |
Visualization (plots/)
| Script | Purpose |
|---|---|
plot_antisense_sense_overlap.R |
Antisense/sense probe peak overlap |
plot_kmer_goi_hits.R |
K-mer hits in genes of interest |
plot_longest_kmer_heatmap.R |
Longest k-mer heatmap across peak sets |
plot_method_comparison.R |
Cross-method comparison (ChIRP/CHART/RAP) |
plot_motif_across_transcript.R |
Motif distribution on target transcripts |
plot_peak_counts.R |
Peak count visualizations |
plot_peak_overlap.R |
Peak overlap diagrams |
plot_read_flanking.R |
Flanking read distributions |
plot_read_intersection.R |
Read intersection with k-mers |
plot_read_intersection_by_chr.R |
Read intersection by chromosome |
plot_read_intersection_published.R |
Read intersection for published peaks |
plot_read_peak_correlation.R |
Read count vs peak count correlation |
plot_signal_over_background.R |
Signal vs background analysis |
plot_study_characteristics.R |
Study metadata characteristics |
plot_subsample_peak_analysis.R |
Subsample peak retention and trends |
Notes
Memory Requirements
- Bowtie2 alignment: 8 GB
- MACS2 peak calling: 6 GB
- HOMER motif finding: 12 GB
- STREME: 25 GB
- K-mer counting (large peak sets): 55 GB
- K-mer analysis scripts: 25-100 GB
Typical Runtime
- Alignment: Several hours per sample
- Peak calling: 30 min - 2 hours
- Motif analysis: 1-4 hours
- K-mer analysis: 1-8 hours depending on peak count
Contact
For questions about this pipeline, please contact the authors.
Citation
If you use this pipeline, please cite the associated publication: