BfArM-MVH/grz-tools

Logging inconsistently switches between resolved and unresolved file paths

Aperta

#436 aperta il 23 ott 2025

 (3 commenti) (0 reazioni) (0 assegnatari)Python (12 fork)auto 404
good first issuepackage: grz-clipackage: grzctlpriority: lowtype: bug

Metriche repository

Star
 (3 stelle)
Metriche merge PR
 (Metriche PR in attesa)

Descrizione

Hi,

as submitting a case requires a certain file and folder structure, we use symlinks to avoid duplicating files and save some storage. We also pseudonymize our patients an additional time before submitting.

An example of the required file and folder structure (original ID of symlink source changed):

$ tree
.
`-- MKR4ZHC8LR84GG03M236NU0WFU94UGZP
    |-- encrypted_files
    |-- files
    |   `-- MKR4ZHC8LR84GG03M236NU0WFU94UGZP
    |       |-- MKR4ZHC8LR84GG03M236NU0WFU94UGZP_S7_R1_001.fastq.gz -> ../../../../Fastq/VP_CST/Routine/H13753-25_S7_R1_001.fastq.gz
    |       |-- MKR4ZHC8LR84GG03M236NU0WFU94UGZP_S7_R2_001.fastq.gz -> ../../../../Fastq/VP_CST/Routine/H13753-25_S7_R2_001.fastq.gz
    |       `-- VariantPlex_Complete_Solid_Tumor_22534_TargetROI-v1.0.bed
    |-- logs
    `-- metadata
        `-- metadata.json

We use grz-cli version 1.3.1:

$ grz-cli --version
grz-cli v1.3.1
Currently accepted metadata schema versions: 1.3.0, 1.2.1, 1.3, 1.2.2
grz-common v1.3.1
grz-pydantic-models v2.3.1

When running the validation the progress of the checksum validation reveals the source file names:

2025-10-23 10:01 AM [INFO] grz_cli.cli: Running command: /staging/anaconda3/envs/grz-tools/bin/grz-cli validate --submission-dir /media/dfsM/_Sequences_NGS_Patho/_NovaSeq/250214_A02023_0196_BHV2LKDRX5/grz-upload/MKR4ZHC8LR84GG03M236NU0WFU94UGZP/
2025-10-23 10:01 AM [INFO] grz_cli.commands.validate: Starting validation...
2025-10-23 10:01 AM [INFO] grz_common.workers.worker.Worker: Encrypted files directory: /media/dfsM/_Sequences_NGS_Patho/_NovaSeq/250214_A02023_0196_BHV2LKDRX5/grz-upload/MKR4ZHC8LR84GG03M236NU0WFU94UGZP/encrypted_files
2025-10-23 10:01 AM [INFO] grz_common.workers.worker.Worker: Log directory: /media/dfsM/_Sequences_NGS_Patho/_NovaSeq/250214_A02023_0196_BHV2LKDRX5/grz-upload/MKR4ZHC8LR84GG03M236NU0WFU94UGZP/logs
2025-10-23 10:01 AM [WARNING] grz_pydantic_models.submission.metadata.v1: VCF file missing for lab datum 'FFPE DNA Tumor' in donor 'MKR4ZHC8LR84GG03M236NU0WFU94UGZP'.VCF files are recommended, but not required.
2025-10-23 10:01 AM [INFO] grz_common.workers.worker.Worker: Starting metadata validation...
2025-10-23 10:01 AM [INFO] grz_common.workers.worker.Worker: Metadata validation successful!
2025-10-23 10:01 AM [INFO] grz_common.workers.worker.Worker: Starting checksum validation (fallback)...
SHA256   ▕██████████████████████████████████████████████████▏      6.12G/6.12G      (   65.5MiB/s, ETA:  00:00) , H13753-25_S7_R1_001.fastq.gz 
SHA256   ▕██████████████████████████████████████████████████▏      6.45G/6.45G      (   65.8MiB/s, ETA:  00:00) , H13753-25_S7_R2_001.fastq.gz
SHA256   ▕██████████████████████████████████████████████████▏       556k/556k       (   62.8MiB/s, ETA:  00:00) , VariantPlex_Complete_Solid_Tumor_22534_TargetROI-v1.0.bed
2025-10-23 10:05 AM [INFO] grz_common.workers.worker.Worker: Checksum validation successful!
2025-10-23 10:05 AM [INFO] grz_common.workers.worker.Worker: Starting sequencing data validation (fallback)...
FASTQ    ▕██████████████████████████████████████████████████▏      6.12G/6.12G      (   28.2MiB/s, ETA:  00:00) , MKR4ZHC8LR84GG03M236NU0WFU94UGZP_S7_R1_001.fastq.gz
FASTQ    ▕██████████████████████████████████████████████████▏      6.45G/6.45G      (   28.2MiB/s, ETA:  00:00) , MKR4ZHC8LR84GG03M236NU0WFU94UGZP_S7_R2_001.fastq.gz
2025-10-23 10:12 AM [INFO] grz_common.workers.worker.Worker: Sequencing data validation successful!
2025-10-23 10:12 AM [INFO] grz_cli.commands.validate: Validation finished!

Interestingly, the sequencing data validation does not reveal the source file names but shows the symlink file names. However, the log files for both contain the original source file names:

MKR4ZHC8LR84GG03M236NU0WFU94UGZP/logs/
|-- progress_checksum_validation.cjson
`-- progress_sequencing_data_validation.cjson

Also, the encryption progress does not show the source file names. But the encryption log file again reveals the original source file names.

May this be a data protection issue? Are the log file also submitted?

Guida contributor