Logging inconsistently switches between resolved and unresolved file paths
#436 opened on Oct 23, 2025
Repository metrics
- Stars
- (3 stars)
- PR merge metrics
- (PR metrics pending)
Description
Hi,
as submitting a case requires a certain file and folder structure, we use symlinks to avoid duplicating files and save some storage. We also pseudonymize our patients an additional time before submitting.
An example of the required file and folder structure (original ID of symlink source changed):
$ tree
.
`-- MKR4ZHC8LR84GG03M236NU0WFU94UGZP
|-- encrypted_files
|-- files
| `-- MKR4ZHC8LR84GG03M236NU0WFU94UGZP
| |-- MKR4ZHC8LR84GG03M236NU0WFU94UGZP_S7_R1_001.fastq.gz -> ../../../../Fastq/VP_CST/Routine/H13753-25_S7_R1_001.fastq.gz
| |-- MKR4ZHC8LR84GG03M236NU0WFU94UGZP_S7_R2_001.fastq.gz -> ../../../../Fastq/VP_CST/Routine/H13753-25_S7_R2_001.fastq.gz
| `-- VariantPlex_Complete_Solid_Tumor_22534_TargetROI-v1.0.bed
|-- logs
`-- metadata
`-- metadata.json
We use grz-cli version 1.3.1:
$ grz-cli --version
grz-cli v1.3.1
Currently accepted metadata schema versions: 1.3.0, 1.2.1, 1.3, 1.2.2
grz-common v1.3.1
grz-pydantic-models v2.3.1
When running the validation the progress of the checksum validation reveals the source file names:
2025-10-23 10:01 AM [INFO] grz_cli.cli: Running command: /staging/anaconda3/envs/grz-tools/bin/grz-cli validate --submission-dir /media/dfsM/_Sequences_NGS_Patho/_NovaSeq/250214_A02023_0196_BHV2LKDRX5/grz-upload/MKR4ZHC8LR84GG03M236NU0WFU94UGZP/
2025-10-23 10:01 AM [INFO] grz_cli.commands.validate: Starting validation...
2025-10-23 10:01 AM [INFO] grz_common.workers.worker.Worker: Encrypted files directory: /media/dfsM/_Sequences_NGS_Patho/_NovaSeq/250214_A02023_0196_BHV2LKDRX5/grz-upload/MKR4ZHC8LR84GG03M236NU0WFU94UGZP/encrypted_files
2025-10-23 10:01 AM [INFO] grz_common.workers.worker.Worker: Log directory: /media/dfsM/_Sequences_NGS_Patho/_NovaSeq/250214_A02023_0196_BHV2LKDRX5/grz-upload/MKR4ZHC8LR84GG03M236NU0WFU94UGZP/logs
2025-10-23 10:01 AM [WARNING] grz_pydantic_models.submission.metadata.v1: VCF file missing for lab datum 'FFPE DNA Tumor' in donor 'MKR4ZHC8LR84GG03M236NU0WFU94UGZP'.VCF files are recommended, but not required.
2025-10-23 10:01 AM [INFO] grz_common.workers.worker.Worker: Starting metadata validation...
2025-10-23 10:01 AM [INFO] grz_common.workers.worker.Worker: Metadata validation successful!
2025-10-23 10:01 AM [INFO] grz_common.workers.worker.Worker: Starting checksum validation (fallback)...
SHA256 ▕██████████████████████████████████████████████████▏ 6.12G/6.12G ( 65.5MiB/s, ETA: 00:00) , H13753-25_S7_R1_001.fastq.gz
SHA256 ▕██████████████████████████████████████████████████▏ 6.45G/6.45G ( 65.8MiB/s, ETA: 00:00) , H13753-25_S7_R2_001.fastq.gz
SHA256 ▕██████████████████████████████████████████████████▏ 556k/556k ( 62.8MiB/s, ETA: 00:00) , VariantPlex_Complete_Solid_Tumor_22534_TargetROI-v1.0.bed
2025-10-23 10:05 AM [INFO] grz_common.workers.worker.Worker: Checksum validation successful!
2025-10-23 10:05 AM [INFO] grz_common.workers.worker.Worker: Starting sequencing data validation (fallback)...
FASTQ ▕██████████████████████████████████████████████████▏ 6.12G/6.12G ( 28.2MiB/s, ETA: 00:00) , MKR4ZHC8LR84GG03M236NU0WFU94UGZP_S7_R1_001.fastq.gz
FASTQ ▕██████████████████████████████████████████████████▏ 6.45G/6.45G ( 28.2MiB/s, ETA: 00:00) , MKR4ZHC8LR84GG03M236NU0WFU94UGZP_S7_R2_001.fastq.gz
2025-10-23 10:12 AM [INFO] grz_common.workers.worker.Worker: Sequencing data validation successful!
2025-10-23 10:12 AM [INFO] grz_cli.commands.validate: Validation finished!
Interestingly, the sequencing data validation does not reveal the source file names but shows the symlink file names. However, the log files for both contain the original source file names:
MKR4ZHC8LR84GG03M236NU0WFU94UGZP/logs/
|-- progress_checksum_validation.cjson
`-- progress_sequencing_data_validation.cjson
Also, the encryption progress does not show the source file names. But the encryption log file again reveals the original source file names.
May this be a data protection issue? Are the log file also submitted?