Sovereign 8s.is mirror of standardgalactic/RNAChromatin
  • R 75.1%
  • Shell 24.9%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
2026-02-11 13:33:52 +02:00
analysis Initial commit: RNAChromatin analysis pipeline 2026-02-11 13:33:52 +02:00
config Initial commit: RNAChromatin analysis pipeline 2026-02-11 13:33:52 +02:00
core Initial commit: RNAChromatin analysis pipeline 2026-02-11 13:33:52 +02:00
data/templates Initial commit: RNAChromatin analysis pipeline 2026-02-11 13:33:52 +02:00
diagnostic Initial commit: RNAChromatin analysis pipeline 2026-02-11 13:33:52 +02:00
docs Initial commit: RNAChromatin analysis pipeline 2026-02-11 13:33:52 +02:00
modules Initial commit: RNAChromatin analysis pipeline 2026-02-11 13:33:52 +02:00
plots Initial commit: RNAChromatin analysis pipeline 2026-02-11 13:33:52 +02:00
preparation Initial commit: RNAChromatin analysis pipeline 2026-02-11 13:33:52 +02:00
utils Initial commit: RNAChromatin analysis pipeline 2026-02-11 13:33:52 +02:00
.gitignore Initial commit: RNAChromatin analysis pipeline 2026-02-11 13:33:52 +02:00

RNAChromatin Analysis Pipeline

A comprehensive pipeline for analyzing RNA-chromatin sequencing datasets, including ChIRP-seq, CHART-seq, and RAP-seq data. This pipeline processes raw sequencing data through peak calling, motif discovery, and k-mer analysis to characterize RNA-chromatin interactions.

Table of Contents

  1. Overview
  2. Requirements
  3. Directory Structure
  4. Pipeline Workflow
  5. Running the Pipeline
  6. Input Files
  7. Output Files
  8. Script Descriptions

Overview

This pipeline performs the following main analyses:

  • Read alignment to reference genomes using Bowtie2
  • Peak calling using MACS2
  • Motif discovery using HOMER and STREME
  • K-mer analysis to identify RNA-binding sequences within peaks
  • Comparative analysis of even/odd probe pairs
  • Antisense/sense probe overlap analysis for control validation
  • Visualization of results

The pipeline is designed to run on an LSF (Load Sharing Facility) HPC cluster.


Requirements

HPC Environment

  • LSF job scheduler (bsub, bjobs commands)
  • Module system for loading software

Software Dependencies

Software Version Purpose
Bowtie2 2.5.1 Read alignment
SAMtools 1.21 BAM file processing
BEDTools 2.30.0 Genomic interval operations
MACS2 2.2.9.1 Peak calling
HOMER 4.11 Motif discovery
MEME Suite 5.5.0 STREME motif discovery, FIMO
deepTools 3.5.2 Coverage track generation
BLAT 3.7 Probe alignment
bedops 2.4.41 BED/PSL format conversion
Picard 2.25.1 BAM processing utilities
SRA-Toolkit 3.0.10 SRA data download
R 4.3.1 Statistical analysis
Java (JRE) 21.0.2 Custom k-mer counting and shuffling tools

R Packages

Data Manipulation:

  • stringr, tidyr, dplyr, magrittr, readr, tibble, purrr

Bioinformatics:

  • Biostrings, GenomicRanges, IRanges, rtracklayer

Visualization:

  • ggplot2, hrbrthemes, patchwork, ggh4x, tidyplot, viridis, geomtextpath, ggseqlogo, ggpattern, ggrepel, cowplot, grid, gridExtra

Other:

  • GetoptLong (string interpolation and command-line parsing)
  • readxl (Excel file reading)
  • XML, reutils (NCBI E-utilities)

External Custom Scripts

The pipeline depends on custom Java tools. Paths are configured in config/paths.sh and config/paths.R:

$KMER_COUNT_TOOL        - scripts.lincs.CountKmers (k-mer counting and positions recording)
$ANALYZE_OVERLAPS_TOOL  - scripts.lincs.AnalyzePeakOverlaps (peak overlap analysis)
$SHUFFLE_SEQUENCES_TOOL - scripts.lincs.util.ShuffleSequences (sequence shuffling)

Directory Structure

The pipeline expects the following directory structure:

chirp/
├── code/                          # This repository
│   ├── core/                      # Main pipeline orchestration
│   │   ├── pipeline_orchestrator.R          # Main orchestration script
│   │   ├── pipeline_orchestrator_subsample.R # Subsample analysis orchestrator
│   │   └── job_submission.sh                # LSF job submission script
│   ├── modules/                   # Shell processing modules
│   │   ├── peak_processing_unified.sh  # Unified peak processing (MACS2 + published)
│   │   ├── paired_peaks_processing.sh  # Paired sample analysis
│   │   └── common_peaks.sh             # Common peaks detection
│   ├── analysis/                  # Analysis scripts
│   │   ├── kmer/                  # K-mer analysis scripts
│   │   │   ├── kmer_analysis_paired.R
│   │   │   ├── kmer_analysis_single.R
│   │   │   ├── kmer_goi_hits_analysis.R
│   │   │   ├── longest_kmer_processing.R
│   │   │   ├── longest_kmer_strand.R
│   │   │   ├── longest_kmer_positions_merge.R
│   │   │   ├── longest_kmer_summary_merge.R
│   │   │   ├── split_kmer_count.sh
│   │   │   ├── split_longest_kmer.sh
│   │   │   └── split_longest_kmer_processing.R
│   │   ├── reads/                 # Read processing scripts
│   │   │   ├── extract_kmer_intersecting_reads.sh
│   │   │   ├── extract_kmer_intersecting_reads_by_chr.sh
│   │   │   ├── extract_kmer_flanking_reads.sh
│   │   │   ├── process_kmer_read_intersection.R
│   │   │   ├── process_kmer_read_intersection_by_chr.R
│   │   │   ├── process_flanking_lk_reads.sh
│   │   │   ├── process_flanking_reads.sh
│   │   │   ├── shuffle_read_names.sh
│   │   │   └── count_bam_reads.sh
│   │   └── other/                 # Other analysis scripts
│   │       ├── antisense_sense_overlap.sh
│   │       ├── antisense_sense_overlap.R
│   │       └── common_peak_calling.R
│   ├── preparation/               # Data preparation scripts
│   │   ├── data_preparation.R
│   │   ├── transcript_preparation.sh
│   │   ├── shuffle_probes_positional.sh
│   │   └── setup_directories.sh
│   ├── utils/                     # Utility functions
│   │   ├── functions.R
│   │   └── overlap_summary.sh
│   ├── config/                    # Configuration files
│   │   ├── paths.R
│   │   └── paths.sh
│   ├── plots/                     # Visualization scripts
│   │   ├── plot_antisense_sense_overlap.R
│   │   ├── plot_kmer_goi_hits.R
│   │   ├── plot_longest_kmer_heatmap.R
│   │   ├── plot_method_comparison.R
│   │   ├── plot_motif_across_transcript.R
│   │   ├── plot_peak_counts.R
│   │   ├── plot_peak_overlap.R
│   │   ├── plot_read_flanking.R
│   │   ├── plot_read_intersection.R
│   │   ├── plot_read_intersection_by_chr.R
│   │   ├── plot_read_intersection_published.R
│   │   ├── plot_read_peak_correlation.R
│   │   ├── plot_signal_over_background.R
│   │   ├── plot_study_characteristics.R
│   │   └── plot_subsample_peak_analysis.R
│   ├── docs/                      # Documentation
│   ├── data/                      # Data files and templates
│   └── diagnostic/                # Diagnostic tools
├── input/                         # Configuration files
│   ├── config.xlsx                # Species-specific configuration
│   ├── sample_table.xlsx          # Sample metadata
│   ├── probes.xlsx                # Probe sequences
│   └── published_peaks_input.xlsx # Published peaks info
├── commands/                      # Generated job tables
├── figures/                       # Cross-species figures
├── human/                         # Human species data
│   ├── reads/                     # FASTQ files
│   ├── alignment/                 # BAM files
│   ├── peaks/
│   │   ├── current/               # MACS2 peaks
│   │   ├── published/             # Published peaks
│   │   ├── pairs/                 # Even/odd overlap analysis
│   │   └── common/                # Common peaks across studies
│   ├── coverage/                  # BigWig files
│   ├── probes/                    # Probe FASTA files
│   ├── transcripts/               # Target transcript sequences
│   ├── auxiliary/                 # Reference files
│   ├── jobs/                      # Job output files
│   └── plots/                     # Analysis plots
└── mouse/                         # Mouse species data (same structure)

Pipeline Workflow

Phase 1: Data Preparation

preparation/data_preparation.R
├── Download SRA metadata (efetch)
├── Create sample tables
├── Generate probe FASTA files
├── Create dinucleotide-shuffled probe controls (non-RAP)
├── Generate transcript FASTA files
├── Create positionally-shuffled probe controls (RAP)
└── BLAT probes to genome and transcripts

Phase 2: Read Processing

core/pipeline_orchestrator.R (generates job tables)
├── bowtie2         → Read alignment
├── blFilter        → Blacklist region removal
├── bamReadCount    → Read counting
├── bamIndex        → BAM indexing
├── bamCoverage     → Coverage track generation
└── chrCount        → Chromosome read counts

Phase 3: Peak Calling

├── macs2                              → Call peaks
└── modules/peak_processing_unified.sh → For each sample:
    ├── Sort peaks by score
    ├── Generate FASTA sequences
    ├── Calculate nucleotide content
    ├── Run HOMER motif discovery
    ├── Run STREME motif discovery
    ├── Run FIMO on transcripts
    └── Count k-mers

Phase 4: Comparative Analysis

├── modules/common_peaks.sh              → Find peaks common across studies
├── modules/paired_peaks_processing.sh   → Analyze even/odd probe pairs:
│   ├── Find intersecting peaks
│   ├── Create union peaks
│   ├── Run motif analysis
│   └── Count k-mers
└── analysis/other/antisense_sense_overlap.sh → Antisense/sense peak overlap

Phase 5: K-mer and Read Analysis

├── analysis/kmer/kmer_analysis_paired.R     → K-mer analysis for paired samples
├── analysis/kmer/kmer_analysis_single.R     → K-mer analysis for single samples
├── analysis/kmer/kmer_goi_hits_analysis.R   → K-mer hits in genes of interest
├── analysis/kmer/longest_kmer_strand.R      → Strand-specific k-mer analysis
├── analysis/reads/extract_kmer_intersecting_reads.sh  → Extract reads overlapping k-mers
├── analysis/reads/extract_kmer_flanking_reads.sh      → Extract reads flanking k-mers
├── analysis/reads/process_kmer_read_intersection.R    → Process read intersections
└── plots/plot_motif_across_transcript.R     → Motif distribution on transcripts

Phase 6: Visualization

plots/
├── plot_antisense_sense_overlap.R    → Antisense/sense probe overlap
├── plot_kmer_goi_hits.R              → K-mer hits in genes of interest
├── plot_longest_kmer_heatmap.R     → Longest k-mer heatmap across peak sets
├── plot_method_comparison.R          → Deeper dive into (CHART/RAP with a focus on Xist)
├── plot_motif_across_transcript.R    → Motif distribution on transcripts
├── plot_peak_counts.R                → Peak count visualizations
├── plot_peak_overlap.R               → Peak overlap diagrams
├── plot_read_flanking.R              → Longest K-mer flanking read distributions
├── plot_read_intersection.R          → Longest K-mer read intersection with k-mers
├── plot_read_intersection_by_chr.R   → Longest K-mer read intersection by chromosome
├── plot_read_intersection_published.R → Longest K-mer read intersection for published peaks
├── plot_read_peak_correlation.R      → Read count vs peak count correlation
├── plot_signal_over_background.R     → K-mer signal vs background analysis
├── plot_study_characteristics.R      → Study metadata characteristics
└── plot_subsample_peak_analysis.R    → Subsample peak retention/trends

Running the Pipeline

Step 1: Prepare Configuration Files

  1. Create input/config.xlsx with species-specific paths:

    • species: human/mouse
    • genome_assembly: hg19/mm10
    • genome_fasta: Path to genome FASTA
    • bowtie_build: Path to Bowtie2 index
    • blacklist: Path to blacklist BED file
    • genome_code: MACS2 genome code (hs/mm)
  2. Create input/sample_table.xlsx with sample metadata

Step 2: Run Data Preparation

# In R:
source("preparation/data_preparation.R")

This will:

  • Download SRA metadata
  • Create probe FASTA files
  • Generate dinucleotide-shuffled controls (non-RAP)
  • Prepare transcript files
  • Generate positionally-shuffled controls (RAP)
  • BLAT probes to genome and transcripts

Generated job tables in commands/:

  • download_srr_job_table.tsv - SRA read download
  • shuffle_probes_job_table.tsv - Dinucleotide probe shuffling
  • rap_shuffle_probes_job_table.tsv - Positional probe shuffling (RAP)
  • probe_blat_genome_job_table.tsv - BLAT probes to genome
  • probe_blat_transcript_job_table.tsv - BLAT probes to transcripts

Step 3: Generate and Submit Job Tables

# In R:
source("core/pipeline_orchestrator.R")

This generates TSV files in commands/ directory:

  • current_job_table.tsv - Main alignment and peak calling jobs
  • common_peaks_job_table.tsv - Common peaks analysis
  • job_table_published_peak_module.tsv - Published peaks processing
  • job_table_pair_peak_module.tsv - Paired sample analysis (even\odd matched samples)
  • analyze_kmer_job_table.tsv - K-mer analysis jobs
  • kmer_goi_hits_job_table.tsv - K-mer GOI hits analysis
  • longest_kmer_reads_job_table.tsv - Read-longest kmer intersection extraction
  • longest_kmer_reads_published_job_table.tsv - Read-longest kmer intersection (published)
  • motif_transcript_plot_job_table.tsv - Motif across transcript plots

The antisense/sense overlap analysis runs directly via system() at the end of the orchestrator.

Step 4: Submit Jobs to LSF

# Submit main pipeline jobs
./code/core/job_submission.sh /path/to/chirp/commands/current_job_table.tsv

# After macs2 jobs complete, submit downstream analyses
./code/core/job_submission.sh /path/to/chirp/commands/common_peaks_job_table.tsv
./code/core/job_submission.sh /path/to/chirp/commands/job_table_pair_peak_module.tsv
./code/core/job_submission.sh /path/to/chirp/commands/analyze_kmer_job_table.tsv
./code/core/job_submission.sh /path/to/chirp/commands/kmer_goi_hits_job_table.tsv
./code/core/job_submission.sh /path/to/chirp/commands/longest_kmer_reads_job_table.tsv

Job Submission Script Usage

./code/core/job_submission.sh <job_table.tsv> [ERROR_FILE] [FORCE_SUBMIT]

# Arguments:
#   job_table.tsv - TSV file with job specifications
#   ERROR_FILE    - TRUE/FALSE to create error files (default: FALSE)
#   FORCE_SUBMIT  - TRUE/FALSE to resubmit completed jobs (default: FALSE)

Input Files

Required Configuration Files

File Description
input/config.xlsx Species-specific paths and parameters
input/sample_table.xlsx Sample metadata (GSE, GSM, conditions)
input/published_peaks_input.xlsx Published peak file information
input/probes.xlsx Probe sequences for each study

Required Reference Files

File Location Description
Genome FASTA {species}/auxiliary/{assembly}.fa Reference genome
Blacklist BED {species}/auxiliary/blacklist.bed Regions to exclude
Bowtie2 index As specified in config Alignment index
Transcript BED {species}/auxiliary/txCoords12.bed Target transcript coordinates (BED12)

Output Files

Per-Sample Outputs

Directory Contents
alignment/ {GSM}.bam, {GSM}_blFilter.bam
coverage/ {GSM}.bw (BigWig)
peaks/current/peaks/ {GSM}_peaks.narrowPeak
peaks/current/peaks_fasta/ Peak sequences (FASTA)
peaks/current/peaks_nuc/ Nucleotide content
peaks/current/homer/{GSM}/ HOMER motif results
peaks/current/streme/{GSM}/ STREME motif results
peaks/current/kmer/ K-mer count files
peaks/current/longest_kmer/ Longest k-mer analysis

Paired Analysis Outputs

Directory Contents
peaks/pairs/peaks/ Intersect and union BED files
peaks/pairs/homer/ Motif analysis on overlaps
peaks/pairs/analyze_overlaps/ Overlap statistics

Common Peaks Outputs

Directory Contents
peaks/common/ Common peaks across studies

Cross-Species Figures

Directory Contents
figures/ antisense_sense_overlap.pdf, subsample_peak_analysis.pdf, reads_peaks_correlation_plot.pdf

Script Descriptions

Main Orchestration (core/)

Script Purpose
pipeline_orchestrator.R Generates job tables for entire pipeline
pipeline_orchestrator_subsample.R Generates job tables for subsample analysis
job_submission.sh Submits jobs from TSV tables to LSF

Data Preparation (preparation/)

Script Purpose
data_preparation.R SRA metadata download, probe generation, shuffled controls, transcript prep
transcript_preparation.sh Generate transcript FASTA/BED files from txCoords12.bed
shuffle_probes_positional.sh Positional probe shuffling for RAP (BLAT-based)
setup_directories.sh Create required directory structure

Utility Functions (utils/)

Script Purpose
functions.R Core utility functions (validation, k-mer helpers, sample name repair, theming)
overlap_summary.sh Overlap summary helper

Peak Processing Modules (modules/)

Script Purpose
peak_processing_unified.sh Unified peak processing: handles both MACS2-called and published peaks, motifs, k-mers
peak_processing.sh Standard peak processing
peak_processing_published.sh Published peak processing
paired_peaks_processing.sh Analyze even/odd paired samples
common_peaks.sh Find common peaks across studies

K-mer Analysis (analysis/kmer/)

Script Purpose
kmer_analysis_paired.R K-mer analysis for paired (even/odd) samples
kmer_analysis_single.R K-mer analysis for single samples
kmer_goi_hits_analysis.R K-mer hits within genes of interest (GOI)
longest_kmer_processing.R Find longest matching k-mers per peak
longest_kmer_strand.R Determine k-mer strand via probe alignment
longest_kmer_positions_merge.R Merge k-mer position tables
longest_kmer_summary_merge.R Merge k-mer summary tables
split_kmer_count.sh Split large peak sets for k-mer counting
split_longest_kmer.sh Split largest k-mer FASTA files
split_longest_kmer_processing.R Process split longest k-mer files

Read Processing (analysis/reads/)

Script Purpose
extract_kmer_intersecting_reads.sh Extract reads overlapping k-mers
extract_kmer_intersecting_reads_by_chr.sh Extract reads overlapping k-mers by chromosome
extract_kmer_flanking_reads.sh Extract reads flanking k-mers
process_kmer_read_intersection.R Summarise read-kmer intersection distances
process_kmer_read_intersection_by_chr.R Summarise read-kmer intersections by chromosome
process_flanking_lk_reads.sh Process flanking longest k-mer reads
process_flanking_reads.sh Process flanking reads
shuffle_read_names.sh Shuffle read names for control analysis
count_bam_reads.sh Count reads in BAM files

Other Analysis (analysis/other/)

Script Purpose
antisense_sense_overlap.sh Compute antisense/sense probe peak overlaps
common_peak_calling.R Common peak calling across studies

Visualization (plots/)

Script Purpose
plot_antisense_sense_overlap.R Antisense/sense probe peak overlap
plot_kmer_goi_hits.R K-mer hits in genes of interest
plot_longest_kmer_heatmap.R Longest k-mer heatmap across peak sets
plot_method_comparison.R Cross-method comparison (ChIRP/CHART/RAP)
plot_motif_across_transcript.R Motif distribution on target transcripts
plot_peak_counts.R Peak count visualizations
plot_peak_overlap.R Peak overlap diagrams
plot_read_flanking.R Flanking read distributions
plot_read_intersection.R Read intersection with k-mers
plot_read_intersection_by_chr.R Read intersection by chromosome
plot_read_intersection_published.R Read intersection for published peaks
plot_read_peak_correlation.R Read count vs peak count correlation
plot_signal_over_background.R Signal vs background analysis
plot_study_characteristics.R Study metadata characteristics
plot_subsample_peak_analysis.R Subsample peak retention and trends

Notes

Memory Requirements

  • Bowtie2 alignment: 8 GB
  • MACS2 peak calling: 6 GB
  • HOMER motif finding: 12 GB
  • STREME: 25 GB
  • K-mer counting (large peak sets): 55 GB
  • K-mer analysis scripts: 25-100 GB

Typical Runtime

  • Alignment: Several hours per sample
  • Peak calling: 30 min - 2 hours
  • Motif analysis: 1-4 hours
  • K-mer analysis: 1-8 hours depending on peak count

Contact

For questions about this pipeline, please contact the authors.

Citation

If you use this pipeline, please cite the associated publication:

https://www.biorxiv.org/content/10.1101/2025.11.11.687850v1