[RFC]: Bringing `glob` Dependency In-House

Aberta
#11,234 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
5/5
Tempo estimado
Mais de uma semana
Facilidade para iniciantes
35/100
Tipo de issue
Funcionalidade
Clareza
Razoavelmente clara
Status de atividade
Pouca atividade
Stack de tecnologia
javascript, nodejs
Domínio
tooling

Direção de pesquisa

Comece com os layouts de pacote propostos para @stdlib/utils/regexp-from-glob e @stdlib/fs/glob e, em seguida, faça um inventário dos mais de 50 consumidores de require('glob') em @stdlib/_tools/pkgs, _tools/static-analysis e _tools/lint. Execute os testes filtrados listados e os comandos CLI afetados, comparando o comportamento com a dependência existente. O trabalho estará concluído quando ambos os pacotes estiverem testados e documentados, os consumidores tiverem sido migrados, glob tiver sido removido e as saídas e o desempenho permanecerem equivalentes.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

Bringing the glob Dependency In-House

Replacing the external glob npm package with stdlib-native equivalents to eliminate a supply-chain risk and align all production and tooling code with stdlib's quality standards (docs, tests, examples, benchmarks, backward compatibility).

Background

What glob Does

A robust glob implementation in JavaScript that matches files using patterns the shell uses (like stars and question marks). It works by walking the filesystem and applying regex-based matching to paths.
More here: https://github.com/isaacs/node-glob

Its core dependencies
  • minimatch: Converts the glob strings into regular expressions and performs the actual string matching.
  • fs.realpath / inflight / once: Various utilities used under the hood to manage concurrent filesystem calls and path resolutions.
How stdlib uses it
  • 50+ files across: @stdlib/_tools/pkgs/*, @stdlib/_tools/lint/*, @stdlib/_tools/static-analysis/*, and various internal bundling or testing scripts.
  • Only the core discovery functions are used: glob( pattern, opts, clbk ) and glob.sync( pattern, opts ).
  • Primary use cases: Finding package.json files, source .js files, or .c files, while explicitly passing an ignore option array (e.g., ['node_modules/**', '.git/**']) to prevent scanning massive dependency directories.
  • No advanced API usage: None of the files instantiate the Glob class directly as an event emitter or use advanced bash features like brace expansion ({a,b}).

Proposed Changes

The plan creates 2 new packages mirroring stdlib's decomposable architecture, with each package being independently consumable.


Component 1: @stdlib/utils/regexp-from-glob

Replaces the minimatch npm dependency (transitive via glob).

A utility to convert standard shell wildcard patterns into native JavaScript RegExp objects.

Scope: Strictly scoped to the wildcards stdlib actually uses (*, **, ?). It will not support brace expansions or extglobs to remain intentionally minimal and maintain high execution speed.

@stdlib/utils/regexp-from-glob/
├── lib/
│   ├── index.js          # re-export main
│   └── main.js           # regexpFromGlob(str) → RegExp
├── test/
│   └── test.js
├── benchmark/
│   └── benchmark.js
├── docs/
│   ├── repl.txt
│   └── types/
│       ├── index.d.ts
│       └── test.ts
├── examples/
│   └── index.js
├── README.md
└── package.json

API:

var regexpFromGlob = require( '@stdlib/utils/regexp-from-glob' );

var re = regexpFromGlob( '**/*.js' );
// returns RegExp

re.test( 'lib/index.js' ); // => true
re.test( 'lib/index.json' ); // => false

Component 2: @stdlib/fs/glob

The core package — the direct replacement for require('glob').

A filesystem traversal utility that applies the generated glob regexes to discover files. This is the main package all _tools files will switch to.

@stdlib/fs/glob/
├── lib/
│   ├── index.js          # re-export main
│   ├── main.js           # async implementation
│   ├── sync.js           # sync implementation
│   ├── walk.js           # internal DFS/BFS directory walker
│   └── validate.js       # options validation
├── test/
│   ├── test.js           # async tests
│   ├── test.sync.js      # sync tests
│   └── test.walk.js      # directory walker tree-pruning tests
├── benchmark/
│   ├── benchmark.js      # benchmark: async traversal
│   └── benchmark.sync.js # benchmark: sync traversal
├── docs/
│   ├── repl.txt
│   └── types/
│       ├── index.d.ts
│       └── test.ts
├── examples/
│   └── index.js
├── README.md
└── package.json

API (drop-in compatible with current usage):

var glob = require( '@stdlib/fs/glob' );

var opts = {
    'cwd': __dirname,
    'ignore': [ 'node_modules/**' ],
    'realpath': true
};

// Async usage
glob( '**/*.js', opts, function onGlob( error, matches ) {
    if ( error ) {
        console.error( error );
        return;
    }
    console.dir( matches );
});

// Sync usage
var matches = glob.sync( '**/*.js', opts );

Key implementation details:

  • Uses @stdlib/fs/read-dir and standard fs.stat under the hood.
  • Tree-Pruning (Crucial for perf): The internal walk.js algorithm must evaluate the ignore option before descending into a directory.
  • Options support: cwd (defaults to process.cwd()), ignore (array of globs), and realpath (boolean).
  • Returns arrays strictly normalized, preventing duplicates.
  • No external dependencies (uses existing stdlib utilities).

Component 3: Migration — Updating All Consumers

This is the mechanical bulk change, done after the new packages are created and verified.

[MODIFY] All 50+ files using require( 'glob' )

The change is a single-line replacement per file:

-var glob = require( 'glob' );
+var glob = require( '@stdlib/fs/glob' );

Migration strategy (phased to reduce risk):

Phase Scope Files
Phase 1 @stdlib/_tools/pkgs/* (find, deps, clis, etc.) ~15 files
Phase 2 @stdlib/_tools/static-analysis/* (sloc-glob, etc.) ~10 files
Phase 3 @stdlib/_tools/lint/* (filenames, pkg-json, etc.) ~20 files
Phase 4 Remaining bundles and scripts ~10 files

Each phase follows the same process:

  1. Run find + sed to replace require( 'glob' ) → require( '@stdlib/fs/glob' ).
  2. Run existing internal tool tests for the affected script area.
  3. Verify output matches the original glob dependency.
Uninstall the dependancy

After all phases are complete:

npm uninstall glob

Verification Plan

Automated Tests
1. Unit tests for @stdlib/utils/regexp-from-glob
make TESTS_FILTER=".*/utils/regexp-from-glob/.*" test

Tests should cover:

  • Core wildcards (*, **, ?).
  • Escaping behavior for dot ., plus +, parentheses (), and brackets [].
  • Prefix, suffix, and exact match conditions.
2. Unit tests for @stdlib/fs/glob
make TESTS_FILTER=".*/fs/glob/.*" test

Tests should cover:

  • Standard sync and async pattern matching against a mock filesystem.
  • ignore arrays effectively pruning traversal.
  • cwd option shifts the search base.
  • realpath correctly resolves to absolute paths.
  • Error handling (e.g., trying to read restricted directories gracefully).
3. Regression tests for migrated modules

Run the full test suite for each phase to ensure the tools operate as expected:

# Phase 1:
make TESTS_FILTER=".*/_tools/pkgs/.*" test

# Phase 2:
make TESTS_FILTER=".*/_tools/static-analysis/.*" test

# Phase 3:
make TESTS_FILTER=".*/_tools/lint/.*" test

# Phase 4: Full internal tools suite
make test
Manual Verification

After migration, verify the tools perform correctly on the monorepo:

  1. Verify Package Discovery:

    node ./lib/node_modules/@stdlib/_tools/pkgs/find/bin/cli
    

    → Verify: Outputs all valid stdlib sub-packages identically to previous runs, without diving into node_modules.

  2. Verify SLOC execution:

    node ./lib/node_modules/@stdlib/_tools/static-analysis/js/sloc-glob/bin/cli
    

    → Verify: Accurately computes lines of code for the workspace.

  3. Performance Check:
    Time the execution before and after the swap.

    time node ./lib/node_modules/@stdlib/_tools/pkgs/find/bin/cli
    

    → Verify: Our native, tree-pruned walker executes in equivalent or faster time than the external dependency.

Linguagem predominante
JavaScript
Estrelas
6k
Forks
1.3k
Merge médio
1d 21h
PRs com merge (30d)
224

Preparar o ambiente

Abrir no Codespaces

Inicia o contêiner de desenvolvimento do projeto no navegador, com a sua própria conta do GitHub.

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de stdlib-js/stdlib

Todas as issues de stdlib-js/stdlib

Issues semelhantes

Mais issues de JavaScript

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.