BfArM-MVH/grz-tools

Logging inconsistently switches between resolved and unresolved file paths

Open

#436 opened on Oct 23, 2025

View on GitHub
 (3 comments) (0 reactions) (0 assignees)Python (12 forks)auto 404
good first issuepackage: grz-clipackage: grzctlpriority: lowtype: bug

Repository metrics

Stars
 (3 stars)
PR merge metrics
 (PR metrics pending)

Description

Hi,

as submitting a case requires a certain file and folder structure, we use symlinks to avoid duplicating files and save some storage. We also pseudonymize our patients an additional time before submitting.

An example of the required file and folder structure (original ID of symlink source changed):

$ tree
.
`-- MKR4ZHC8LR84GG03M236NU0WFU94UGZP
    |-- encrypted_files
    |-- files
    |   `-- MKR4ZHC8LR84GG03M236NU0WFU94UGZP
    |       |-- MKR4ZHC8LR84GG03M236NU0WFU94UGZP_S7_R1_001.fastq.gz -> ../../../../Fastq/VP_CST/Routine/H13753-25_S7_R1_001.fastq.gz
    |       |-- MKR4ZHC8LR84GG03M236NU0WFU94UGZP_S7_R2_001.fastq.gz -> ../../../../Fastq/VP_CST/Routine/H13753-25_S7_R2_001.fastq.gz
    |       `-- VariantPlex_Complete_Solid_Tumor_22534_TargetROI-v1.0.bed
    |-- logs
    `-- metadata
        `-- metadata.json

We use grz-cli version 1.3.1:

$ grz-cli --version
grz-cli v1.3.1
Currently accepted metadata schema versions: 1.3.0, 1.2.1, 1.3, 1.2.2
grz-common v1.3.1
grz-pydantic-models v2.3.1

When running the validation the progress of the checksum validation reveals the source file names:

2025-10-23 10:01 AM [INFO] grz_cli.cli: Running command: /staging/anaconda3/envs/grz-tools/bin/grz-cli validate --submission-dir /media/dfsM/_Sequences_NGS_Patho/_NovaSeq/250214_A02023_0196_BHV2LKDRX5/grz-upload/MKR4ZHC8LR84GG03M236NU0WFU94UGZP/
2025-10-23 10:01 AM [INFO] grz_cli.commands.validate: Starting validation...
2025-10-23 10:01 AM [INFO] grz_common.workers.worker.Worker: Encrypted files directory: /media/dfsM/_Sequences_NGS_Patho/_NovaSeq/250214_A02023_0196_BHV2LKDRX5/grz-upload/MKR4ZHC8LR84GG03M236NU0WFU94UGZP/encrypted_files
2025-10-23 10:01 AM [INFO] grz_common.workers.worker.Worker: Log directory: /media/dfsM/_Sequences_NGS_Patho/_NovaSeq/250214_A02023_0196_BHV2LKDRX5/grz-upload/MKR4ZHC8LR84GG03M236NU0WFU94UGZP/logs
2025-10-23 10:01 AM [WARNING] grz_pydantic_models.submission.metadata.v1: VCF file missing for lab datum 'FFPE DNA Tumor' in donor 'MKR4ZHC8LR84GG03M236NU0WFU94UGZP'.VCF files are recommended, but not required.
2025-10-23 10:01 AM [INFO] grz_common.workers.worker.Worker: Starting metadata validation...
2025-10-23 10:01 AM [INFO] grz_common.workers.worker.Worker: Metadata validation successful!
2025-10-23 10:01 AM [INFO] grz_common.workers.worker.Worker: Starting checksum validation (fallback)...
SHA256   ▕██████████████████████████████████████████████████▏      6.12G/6.12G      (   65.5MiB/s, ETA:  00:00) , H13753-25_S7_R1_001.fastq.gz 
SHA256   ▕██████████████████████████████████████████████████▏      6.45G/6.45G      (   65.8MiB/s, ETA:  00:00) , H13753-25_S7_R2_001.fastq.gz
SHA256   ▕██████████████████████████████████████████████████▏       556k/556k       (   62.8MiB/s, ETA:  00:00) , VariantPlex_Complete_Solid_Tumor_22534_TargetROI-v1.0.bed
2025-10-23 10:05 AM [INFO] grz_common.workers.worker.Worker: Checksum validation successful!
2025-10-23 10:05 AM [INFO] grz_common.workers.worker.Worker: Starting sequencing data validation (fallback)...
FASTQ    ▕██████████████████████████████████████████████████▏      6.12G/6.12G      (   28.2MiB/s, ETA:  00:00) , MKR4ZHC8LR84GG03M236NU0WFU94UGZP_S7_R1_001.fastq.gz
FASTQ    ▕██████████████████████████████████████████████████▏      6.45G/6.45G      (   28.2MiB/s, ETA:  00:00) , MKR4ZHC8LR84GG03M236NU0WFU94UGZP_S7_R2_001.fastq.gz
2025-10-23 10:12 AM [INFO] grz_common.workers.worker.Worker: Sequencing data validation successful!
2025-10-23 10:12 AM [INFO] grz_cli.commands.validate: Validation finished!

Interestingly, the sequencing data validation does not reveal the source file names but shows the symlink file names. However, the log files for both contain the original source file names:

MKR4ZHC8LR84GG03M236NU0WFU94UGZP/logs/
|-- progress_checksum_validation.cjson
`-- progress_sequencing_data_validation.cjson

Also, the encryption progress does not show the source file names. But the encryption log file again reveals the original source file names.

May this be a data protection issue? Are the log file also submitted?

Contributor guide