Incorrect MD5 checksums being used for some files
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 38/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Stale
- Domain
- bioinformatics
Research direction
Start by tracing how the metadata fields fastq_ftp, fastq_md5, fastq_1, and md5_1 are parsed and paired, using the SRR17001000 example in the issue to reproduce the mismatch. Done means the downloaded file is assigned the correct metadata checksum, including records with extra FASTQ entries, and checksum validation succeeds.
Written by the indexing model from the issue text.
Description
Description of the bug
I've been having an issue with some files failing at checksum in some studies. Upon investigation, for at least some of these failing samples, it appears to be due to the pipeline not picking the correct MD5 value from the metadata.
For example, manually downloading the this file finishes and yields a 7b730 checksum:
(aspera) jonsan@nf-head:~/fetchngs/EA_pharma/fetchngs_exec/test$ ascp -QT -l 300m -P33001 -i $CONDA_PREFIX/etc/aspera/aspera_bypass_dsa.pem [email protected]:/vol1/fastq/SRR170/000/SRR17001000/SRR17001000.fastq.gz SRX13191258_SRR17001000_1.fastq.gz
SRX13191258_SRR17001000_1.fastq.gz 100% 26MB 15.3Mb/s 00:10
Completed: 26745K bytes transferred in 11 seconds
(19634K bits/sec), in 1 file.
(aspera) jonsan@nf-head:~/fetchngs/EA_pharma/fetchngs_exec/test$ md5sum SRX13191258_SRR17001000_1.fastq.gz 7b7e0af5429bcb54b2c232489ea8212b SRX13191258_SRR17001000_1.fastq.gz
However, looking at the command.sh file for this operation, the pipeline is comparing with a 3fcee checksum:
#!/bin/bash -euo pipefail
ascp \
-QT -l 300m -P33001 \
-i $CONDA_PREFIX/etc/aspera/aspera_bypass_dsa.pem \
[email protected]:/vol1/fastq/SRR170/000/SRR17001000/SRR17001000.fastq.gz \
SRX13191258_SRR17001000_1.fastq.gz
echo "3fcee2e72a2ec6221cac142538aff092 SRX13191258_SRR17001000_1.fastq.gz" > SRX13191258_SRR17001000_1.fastq.gz.md5
md5sum -c SRX13191258_SRR17001000_1.fastq.gz.md5
If we look at the metadata downloaded for this run, we see both checksums being represented, but in different columns:
fastq_md5 **7b7e0**af5429bcb54b2c232489ea8212b**;3fcee**2e72a2ec6221cac142538aff092;383df08e03e1cd1ee071fd67c16b085b
fastq_bytes 27387589;1445187226;1481254395
fastq_ftp ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/000/SRR17001000/SRR17001000.fastq.gz;ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/000/SRR17001000/SRR17001000_1.fastq.gz;ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/000/SRR17001000/SRR17001000_2.fastq.gz
fastq_galaxy ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/000/SRR17001000/SRR17001000.fastq.gz;ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/000/SRR17001000/SRR17001000_1.fastq.gz;ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/000/SRR17001000/SRR17001000_2.fastq.gz
fastq_aspera fasp.sra.ebi.ac.uk:/vol1/fastq/SRR170/000/SRR17001000/SRR17001000.fastq.gz;fasp.sra.ebi.ac.uk:/vol1/fastq/SRR170/000/SRR17001000/SRR17001000_1.fastq.gz;fasp.sra.ebi.ac.uk:/vol1/fastq/SRR170/000/SRR17001000/SRR17001000_2.fastq.gz
fastq_1 ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/000/SRR17001000/SRR17001000_1.fastq.gz
fastq_2 ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/000/SRR17001000/SRR17001000_2.fastq.gz
md5_1 **3fcee**2e72a2ec6221cac142538aff092
md5_2 383df08e03e1cd1ee071fd67c16b085b
It appears as if there are three fastq files, and the workflow is grabbing the first one (maybe an index read? it's much smaller than the other two) and renaming it _1.fastq.gz, then comparing against the latter's MD5. I haven't looked in the code yet to determine where the logic is that's splitting reads 1 and 2, but it appears that it might be making too liberal an assumption about the structure of the fastq_ftp field?
Maybe related to issue #260 ?
Either way, this is leading to failed downloads, it seems like it might properly be considered a bug.
Command used and terminal output
Relevant files
No response
System information
No response
- Dominant language
- Nextflow
- Stars
- 202
- Forks
- 96
- Avg merge
- 6h 36m
- Merged PRs (30d)
- 4
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from nf-core/fetchngs
-
bug
Difficulty 1/5 Under an hour Newbie friendliness 78/100
nf-core/fetchngs#416 · 1 comment ·
Maintainers usually reply within 1 day
-
SRA_FASTQ_FTP requests 12 GB / 2 CPUs (up to 36 GB / 6 CPUs on retry) for a download that uses ~15 MBPossibly taken A pull request linked to this issue is open or already merged. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 74/100
nf-core/fetchngs#401 · 1 reaction ·
Maintainers usually reply within 1 day
-
bug
Difficulty 3/5 1-2 days Newbie friendliness 74/100
Maintainers usually reply within 1 day
-
restore mkfg file from CUSTOM_SRATOOLSNCBISETTINGS / FASTQ_DOWNLOAD_PREFETCH_FASTERQDUMP_SRATOOLSOpenbug
Difficulty 4/5 3-5 days Newbie friendliness 25/100
Maintainers usually reply within 1 day
All issues in nf-core/fetchngs
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
openvax/vaxrank#764 · 2 comments ·
Maintainers usually reply within 1 day