Size of DNAStringSet object not compatible with `nchar()`

Aperta
#129 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
25/100
Tipo di issue
Bug
Chiarezza
Da chiarire
Stato di attività
Ferma
Stack tecnologico
r
Ambito
performance

Direzione di ricerca

Inizia eseguendo il reprex R fornito e confrontando obj_size() con nchar() per gli oggetti DNAStringSet genome, promoters e promoters_subset. Traccia il modo in cui il subsetting di DNAStringSet conserva l'archiviazione delle sequenze, quindi determina se le dimensioni osservate sono previste o indicano un bug; il lavoro deve includere una spiegazione documentata oppure una correzione verificata con copertura di regressione.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Hi all,

For a long time, I've noticed that {Biostrings} is very weird in terms of memory usage. For instance, when one subsets a DNAStringSet object to create a much smaller DNAStringSet object, object sizes do not change much (if at all). Here is a demonstration, comparing a DNAStringSet object with an entire plant genome to another (much smaller) DNAStringSet object with promoter sequences only:

suppressPackageStartupMessages({
    library(Biostrings)
    library(GenomicRanges)
    library(lobstr)
})
options(timeout = 1e4)

# Load genome and annotation 
genome <- readDNAStringSet(
    "https://ftp.ebi.ac.uk/ensemblgenomes/pub/release-61/plants/fasta/hordeum_vulgare/dna/Hordeum_vulgare.MorexV3_pseudomolecules_assembly.dna_rm.toplevel.fa.gz"
)
names(genome) <- gsub(" .*", "", names(genome))

annot <- rtracklayer::import(
    "https://ftp.ebi.ac.uk/ensemblgenomes/pub/release-61/plants/gff3/hordeum_vulgare/Hordeum_vulgare.MorexV3_pseudomolecules_assembly.61.gff3.gz"
)

# Keep genes only and add seqlengths
annot <- annot[annot$type == "gene"]
sl <- setNames(width(genome), names(genome))
sl <- sl[seqlevels(annot)]
seqlengths(annot) <- sl

# Extract promoter sequences
prom_ranges <- trim(promoters(annot, 1000, 200))
prom_seqs <- BSgenome::getSeq(genome, prom_ranges)
names(prom_seqs) <- prom_ranges$gene_id

# Select promoters of random genes
sel_genes <- sample(names(prom_seqs), 1e4, replace = FALSE)
prom_seqs_subset <- prom_seqs[sel_genes]

# Inspect objects
size_df <- data.frame(
    name = c("genome", "promoters", "promoters_subset"),
    size = c(
        obj_size(genome), 
        obj_size(prom_seqs), 
        obj_size(prom_seqs_subset)
    ),
    Mbp = c(
        sum(nchar(genome)) / 1e6, 
        sum(nchar(prom_seqs)) / 1e6, 
        sum(nchar(prom_seqs_subset)) / 1e6
    )
)

size_df
#>               name    size        Mbp
#> 1           genome 4.23 GB 4225.57752
#> 2        promoters 4.25 GB   42.98556
#> 3 promoters_subset 4.25 GB   11.99864

# Does removing `genome` do anything?
rm(genome)

size_df2 <- data.frame(
    name = c("promoters", "promoters_subset"),
    size = c(
        obj_size(prom_seqs), 
        obj_size(prom_seqs_subset)
    ),
    Mbp = c(
        sum(nchar(prom_seqs)) / 1e6, 
        sum(nchar(prom_seqs_subset)) / 1e6
    )
)

size_df2
#>               name    size      Mbp
#> 1        promoters 4.25 GB 42.98556
#> 2 promoters_subset 4.25 GB 11.99864


sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.4.1 (2024-06-14)
#>  os       Ubuntu 22.04.4 LTS
#>  system   x86_64, linux-gnu
#>  ui       X11
#>  language (EN)
#>  collate  en_US.UTF-8
#>  ctype    en_US.UTF-8
#>  tz       Europe/Brussels
#>  date     2025-08-07
#>  pandoc   3.2 @ /usr/lib/rstudio/resources/app/bin/quarto/bin/tools/x86_64/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package              * version   date (UTC) lib source
#>  abind                  1.4-5     2016-07-21 [1] CRAN (R 4.4.1)
#>  Biobase                2.64.0    2024-04-30 [1] Bioconductor 3.19 (R 4.4.1)
#>  BiocGenerics         * 0.50.0    2024-04-30 [1] Bioconductor 3.19 (R 4.4.1)
#>  BiocIO                 1.14.0    2024-04-30 [1] Bioconductor 3.19 (R 4.4.1)
#>  BiocParallel           1.38.0    2024-04-30 [1] Bioconductor 3.19 (R 4.4.1)
#>  Biostrings           * 2.72.1    2024-06-02 [1] Bioconductor 3.19 (R 4.4.1)
#>  bitops                 1.0-7     2021-04-24 [1] CRAN (R 4.4.1)
#>  BSgenome               1.72.0    2024-04-30 [1] Bioconductor 3.19 (R 4.4.1)
#>  cli                    3.6.3     2024-06-21 [1] CRAN (R 4.4.1)
#>  codetools              0.2-20    2024-03-31 [1] CRAN (R 4.4.1)
#>  crayon                 1.5.3     2024-06-20 [1] CRAN (R 4.4.1)
#>  curl                   5.2.1     2024-03-01 [1] CRAN (R 4.4.1)
#>  DelayedArray           0.30.1    2024-05-07 [1] Bioconductor 3.19 (R 4.4.1)
#>  digest                 0.6.36    2024-06-23 [1] CRAN (R 4.4.1)
#>  evaluate               0.24.0    2024-06-10 [1] CRAN (R 4.4.1)
#>  fastmap                1.2.0     2024-05-15 [1] CRAN (R 4.4.1)
#>  fs                     1.6.4     2024-04-25 [1] CRAN (R 4.4.1)
#>  GenomeInfoDb         * 1.40.1    2024-05-24 [1] Bioconductor 3.19 (R 4.4.1)
#>  GenomeInfoDbData       1.2.12    2024-07-24 [1] Bioconductor
#>  GenomicAlignments      1.40.0    2024-04-30 [1] Bioconductor 3.19 (R 4.4.1)
#>  GenomicRanges        * 1.56.1    2024-06-12 [1] Bioconductor 3.19 (R 4.4.1)
#>  glue                   1.8.0     2024-09-30 [1] https://cran.r-universe.dev (R 4.4.1)
#>  htmltools              0.5.8.1   2024-04-04 [1] CRAN (R 4.4.1)
#>  httr                   1.4.7     2023-08-15 [1] CRAN (R 4.4.1)
#>  IRanges              * 2.38.1    2024-07-03 [1] Bioconductor 3.19 (R 4.4.1)
#>  jsonlite               1.8.8     2023-12-04 [1] CRAN (R 4.4.1)
#>  knitr                  1.48      2024-07-07 [1] CRAN (R 4.4.1)
#>  lattice                0.22-6    2024-03-20 [1] CRAN (R 4.4.1)
#>  lifecycle              1.0.4     2023-11-07 [1] CRAN (R 4.4.1)
#>  lobstr               * 1.1.2     2022-06-22 [1] CRAN (R 4.4.1)
#>  Matrix                 1.7-0     2024-04-26 [1] CRAN (R 4.4.1)
#>  MatrixGenerics         1.16.0    2024-04-30 [1] Bioconductor 3.19 (R 4.4.1)
#>  matrixStats            1.3.0     2024-04-11 [1] CRAN (R 4.4.1)
#>  prettyunits            1.2.0     2023-09-24 [1] CRAN (R 4.4.1)
#>  R6                     2.5.1     2021-08-19 [1] CRAN (R 4.4.1)
#>  RCurl                  1.98-1.16 2024-07-11 [1] CRAN (R 4.4.1)
#>  reprex                 2.1.1     2024-07-06 [1] CRAN (R 4.4.1)
#>  restfulr               0.0.15    2022-06-16 [1] CRAN (R 4.4.1)
#>  rjson                  0.2.21    2022-01-09 [1] CRAN (R 4.4.1)
#>  rlang                  1.1.4     2024-06-04 [1] CRAN (R 4.4.1)
#>  rmarkdown              2.27      2024-05-17 [1] CRAN (R 4.4.1)
#>  Rsamtools              2.20.0    2024-04-30 [1] Bioconductor 3.19 (R 4.4.1)
#>  rstudioapi             0.16.0    2024-03-24 [1] CRAN (R 4.4.1)
#>  rtracklayer            1.64.0    2024-04-30 [1] Bioconductor 3.19 (R 4.4.1)
#>  S4Arrays               1.4.1     2024-05-20 [1] Bioconductor 3.19 (R 4.4.1)
#>  S4Vectors            * 0.42.1    2024-07-03 [1] Bioconductor 3.19 (R 4.4.1)
#>  sessioninfo            1.2.2     2021-12-06 [1] CRAN (R 4.4.1)
#>  SparseArray            1.4.8     2024-05-24 [1] Bioconductor 3.19 (R 4.4.1)
#>  SummarizedExperiment   1.34.0    2024-05-01 [1] Bioconductor 3.19 (R 4.4.1)
#>  UCSC.utils             1.0.0     2024-04-30 [1] Bioconductor 3.19 (R 4.4.1)
#>  withr                  3.0.0     2024-01-16 [1] CRAN (R 4.4.1)
#>  xfun                   0.51      2025-02-19 [1] CRAN (R 4.4.1)
#>  XML                    3.99-0.17 2024-06-25 [1] CRAN (R 4.4.1)
#>  XVector              * 0.44.0    2024-04-30 [1] Bioconductor 3.19 (R 4.4.1)
#>  yaml                   2.3.9     2024-07-05 [1] CRAN (R 4.4.1)
#>  zlibbioc               1.50.0    2024-04-30 [1] Bioconductor 3.19 (R 4.4.1)
#> 
#>  [1] /home/faalm/R/x86_64-pc-linux-gnu-library/4.4
#>  [2] /usr/local/lib/R/site-library
#>  [3] /usr/lib/R/site-library
#>  [4] /usr/lib/R/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

Created on 2025-08-07 with reprex v2.1.1

Note that the number of sequences is dramatically different, while object sizes remain pretty much the same. Is that something you are aware of? Why does this happen?

As a side comment, I've also noticed that, for small sets of sequences, it's more efficient to store gzipped FASTA files compared to binary .rds files with XStringSet objects. This is not true for large XStringSet objects, though. It seems like subsetting a large XStringSet object preserves something of the original object in the subset. Does that make sense?

All the best,
Fabricio

Lingua principale
R
Stelle
72
Fork
17
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Bioconductor/Biostrings

Tutte le issue di Bioconductor/Biostrings

Issue simili

Altre issue su R

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.