Simplify Alignment Job and Resolve SAM Flag Issue in CRAM to FASTQ Conversion

[michael-harper]

Context and Motivation

The primary motivation for these changes is to address an underlying bug in bazam, the tool we previously used for converting CRAM files to FASTQ format for re-alignment. The bug causes the SAM flag relating to read pair orientation to be overwritten for correctly oriented read pairs on the reverse strand. This is critical because it impacts the accurate representation of read pair orientations in re-aligned CRAM files. Notably, this issue does not affect the individual read strand flag, which remains correctly preserved.

Changes Introduced

• Removal of bazam for CRAM to fastq conversion in favour of samtools fastq:
  • Replaced the use of bazam with direct extraction of FASTQ files using samtools fastq, ensuring the integrity of SAM flags.
  • The validation of samtools fastq is discussed in this Slack post.
• Removal of unused aligners (BWA and BWA-MEM) in favour of just Dragmap.
• Removal of unused duplicate marking tools, only use picard to mark duplicates now.
• Simplified Alignment Job Workflow:
  • Refactored _get_alignment_input. This was capable of realigning from a specified 'cram version' but was seemingly unable to produce new cram versions when we wanted to realign. I have added the functionality to realign from an input cram and subsequently save the cram to the newly created 'new_version' directory.
  • This was done to be in line with previous code in the align job that already referred to ['workflow']['realign_from_cram_version']
  • Adjusted the _align_one function to handle direct FASTQ extraction and alignment using the selected aligner.
• Enhanced Logging and Error Handling:
  • Added informative logging regarding cram realignment process as well as to track the progress and actions taken during the alignment process.
  • Improved error handling to provide clearer messages when alignment inputs are missing or incorrect.
• The ability to subset a cram was valuable during testing to improve the turnaround time. I have left the subset_cram function in for future use/testing purposes (it is not currently used within the stage).