[RFC]: Bringing `glob` Dependency In-House

Aperta
#11,234 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Tranquilla
Stack tecnologico
javascript, nodejs
Ambito
tooling

Direzione di ricerca

Inizia con i layout dei pacchetti proposti per @stdlib/utils/regexp-from-glob e @stdlib/fs/glob, quindi fai un inventario dei 50+ consumer di require('glob') in @stdlib/_tools/pkgs, _tools/static-analysis e _tools/lint. Esegui i test filtrati elencati e i comandi CLI interessati, confrontando al contempo il comportamento con la dipendenza esistente. Il lavoro è completato quando entrambi i pacchetti sono testati e documentati, i consumer sono migrati, glob è rimosso e gli output e le prestazioni rimangono equivalenti.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Bringing the glob Dependency In-House

Replacing the external glob npm package with stdlib-native equivalents to eliminate a supply-chain risk and align all production and tooling code with stdlib's quality standards (docs, tests, examples, benchmarks, backward compatibility).

Background

What glob Does

A robust glob implementation in JavaScript that matches files using patterns the shell uses (like stars and question marks). It works by walking the filesystem and applying regex-based matching to paths.
More here: https://github.com/isaacs/node-glob

Its core dependencies
  • minimatch: Converts the glob strings into regular expressions and performs the actual string matching.
  • fs.realpath / inflight / once: Various utilities used under the hood to manage concurrent filesystem calls and path resolutions.
How stdlib uses it
  • 50+ files across: @stdlib/_tools/pkgs/*, @stdlib/_tools/lint/*, @stdlib/_tools/static-analysis/*, and various internal bundling or testing scripts.
  • Only the core discovery functions are used: glob( pattern, opts, clbk ) and glob.sync( pattern, opts ).
  • Primary use cases: Finding package.json files, source .js files, or .c files, while explicitly passing an ignore option array (e.g., ['node_modules/**', '.git/**']) to prevent scanning massive dependency directories.
  • No advanced API usage: None of the files instantiate the Glob class directly as an event emitter or use advanced bash features like brace expansion ({a,b}).

Proposed Changes

The plan creates 2 new packages mirroring stdlib's decomposable architecture, with each package being independently consumable.


Component 1: @stdlib/utils/regexp-from-glob

Replaces the minimatch npm dependency (transitive via glob).

A utility to convert standard shell wildcard patterns into native JavaScript RegExp objects.

Scope: Strictly scoped to the wildcards stdlib actually uses (*, **, ?). It will not support brace expansions or extglobs to remain intentionally minimal and maintain high execution speed.

@stdlib/utils/regexp-from-glob/
├── lib/
│   ├── index.js          # re-export main
│   └── main.js           # regexpFromGlob(str) → RegExp
├── test/
│   └── test.js
├── benchmark/
│   └── benchmark.js
├── docs/
│   ├── repl.txt
│   └── types/
│       ├── index.d.ts
│       └── test.ts
├── examples/
│   └── index.js
├── README.md
└── package.json

API:

var regexpFromGlob = require( '@stdlib/utils/regexp-from-glob' );

var re = regexpFromGlob( '**/*.js' );
// returns RegExp

re.test( 'lib/index.js' ); // => true
re.test( 'lib/index.json' ); // => false

Component 2: @stdlib/fs/glob

The core package — the direct replacement for require('glob').

A filesystem traversal utility that applies the generated glob regexes to discover files. This is the main package all _tools files will switch to.

@stdlib/fs/glob/
├── lib/
│   ├── index.js          # re-export main
│   ├── main.js           # async implementation
│   ├── sync.js           # sync implementation
│   ├── walk.js           # internal DFS/BFS directory walker
│   └── validate.js       # options validation
├── test/
│   ├── test.js           # async tests
│   ├── test.sync.js      # sync tests
│   └── test.walk.js      # directory walker tree-pruning tests
├── benchmark/
│   ├── benchmark.js      # benchmark: async traversal
│   └── benchmark.sync.js # benchmark: sync traversal
├── docs/
│   ├── repl.txt
│   └── types/
│       ├── index.d.ts
│       └── test.ts
├── examples/
│   └── index.js
├── README.md
└── package.json

API (drop-in compatible with current usage):

var glob = require( '@stdlib/fs/glob' );

var opts = {
    'cwd': __dirname,
    'ignore': [ 'node_modules/**' ],
    'realpath': true
};

// Async usage
glob( '**/*.js', opts, function onGlob( error, matches ) {
    if ( error ) {
        console.error( error );
        return;
    }
    console.dir( matches );
});

// Sync usage
var matches = glob.sync( '**/*.js', opts );

Key implementation details:

  • Uses @stdlib/fs/read-dir and standard fs.stat under the hood.
  • Tree-Pruning (Crucial for perf): The internal walk.js algorithm must evaluate the ignore option before descending into a directory.
  • Options support: cwd (defaults to process.cwd()), ignore (array of globs), and realpath (boolean).
  • Returns arrays strictly normalized, preventing duplicates.
  • No external dependencies (uses existing stdlib utilities).

Component 3: Migration — Updating All Consumers

This is the mechanical bulk change, done after the new packages are created and verified.

[MODIFY] All 50+ files using require( 'glob' )

The change is a single-line replacement per file:

-var glob = require( 'glob' );
+var glob = require( '@stdlib/fs/glob' );

Migration strategy (phased to reduce risk):

Phase Scope Files
Phase 1 @stdlib/_tools/pkgs/* (find, deps, clis, etc.) ~15 files
Phase 2 @stdlib/_tools/static-analysis/* (sloc-glob, etc.) ~10 files
Phase 3 @stdlib/_tools/lint/* (filenames, pkg-json, etc.) ~20 files
Phase 4 Remaining bundles and scripts ~10 files

Each phase follows the same process:

  1. Run find + sed to replace require( 'glob' ) → require( '@stdlib/fs/glob' ).
  2. Run existing internal tool tests for the affected script area.
  3. Verify output matches the original glob dependency.
Uninstall the dependancy

After all phases are complete:

npm uninstall glob

Verification Plan

Automated Tests
1. Unit tests for @stdlib/utils/regexp-from-glob
make TESTS_FILTER=".*/utils/regexp-from-glob/.*" test

Tests should cover:

  • Core wildcards (*, **, ?).
  • Escaping behavior for dot ., plus +, parentheses (), and brackets [].
  • Prefix, suffix, and exact match conditions.
2. Unit tests for @stdlib/fs/glob
make TESTS_FILTER=".*/fs/glob/.*" test

Tests should cover:

  • Standard sync and async pattern matching against a mock filesystem.
  • ignore arrays effectively pruning traversal.
  • cwd option shifts the search base.
  • realpath correctly resolves to absolute paths.
  • Error handling (e.g., trying to read restricted directories gracefully).
3. Regression tests for migrated modules

Run the full test suite for each phase to ensure the tools operate as expected:

# Phase 1:
make TESTS_FILTER=".*/_tools/pkgs/.*" test

# Phase 2:
make TESTS_FILTER=".*/_tools/static-analysis/.*" test

# Phase 3:
make TESTS_FILTER=".*/_tools/lint/.*" test

# Phase 4: Full internal tools suite
make test
Manual Verification

After migration, verify the tools perform correctly on the monorepo:

  1. Verify Package Discovery:

    node ./lib/node_modules/@stdlib/_tools/pkgs/find/bin/cli
    

    → Verify: Outputs all valid stdlib sub-packages identically to previous runs, without diving into node_modules.

  2. Verify SLOC execution:

    node ./lib/node_modules/@stdlib/_tools/static-analysis/js/sloc-glob/bin/cli
    

    → Verify: Accurately computes lines of code for the workspace.

  3. Performance Check:
    Time the execution before and after the swap.

    time node ./lib/node_modules/@stdlib/_tools/pkgs/find/bin/cli
    

    → Verify: Our native, tree-pruned walker executes in equivalent or faster time than the external dependency.

Lingua principale
JavaScript
Stelle
6k
Fork
1.3k
Merge medio
1g 21h
PR unite (30g)
224

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di stdlib-js/stdlib

Tutte le issue di stdlib-js/stdlib

Issue simili

Altre issue su JavaScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.