Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

feat(dev): add managed local model serving tool

Open
  • Is something wrong?
#252 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@binaryaaron is already working on this.

Since Jul 6, 2026.

  • #212 by @binaryaaron — open

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
python
Domain
ai-infra-agents, cli, devops

Research direction

Start by reviewing PR #212 and the repository's local-model dependency group, since the implementation is already tracked there. Check how the source-tree tool should expose compile, launch, probe, status, and stop, then verify the acceptance criteria for deterministic plans, capability probes, bounded cleanup, and both GPU-host and supplied-container workflows.

Written by the indexing model from the issue text.

Description

task
Priority Level

Medium

Task Summary

Add a source-tree developer tool that deploys and manages local models through one vLLM process lifecycle. The tool should let contributors run Anonymizer against pinned generation and detection models on a GPU host or inside the supplied container without introducing a second deployment domain.

Technical Details & Implementation Plan

The implementation should:

  • Compile pinned TOML profiles into normalized plans with task, model, vllm, and local sections.
  • Provide compile, launch, probe, status, and stop commands.
  • Support stock Hugging Face generation models and NVIDIA GLiNER or GLiNER2 through vLLM Factory.
  • Preserve process identity with launch receipts, PID start markers, and process-group ownership checks.
  • Probe task-specific capabilities rather than treating a listening port as readiness.
  • Bound termination and cleanup so failed or stale processes cannot hang the tool indefinitely.
  • Run the same compiler and local-process lifecycle on a GPU host and inside the supplied container.
  • Document the deployment workflow and provide representative pinned profiles.

Acceptance criteria:

  • Compilation is pure and produces deterministic, digest-protected plans.
  • Plan digests are documented as accidental corruption detection, not authentication or semantic proof.
  • Generation and GLiNER profiles share one lifecycle implementation.
  • Capability probes validate chat completions or GLiNER dynamic labels, offsets, and scores as appropriate.
  • Status and stop operations reject stale or reused process identities.
  • Cleanup is bounded and reports whether it completed.
  • Tests cover lifecycle invariants and behavior rather than profile inventory tables.
  • Documentation covers both direct GPU-host deployment and the supplied container.
  • A full Anonymizer dataset run succeeds with locally hosted GLiNER and a generation model, and both services clean up successfully.

Out of scope:

  • Public package APIs for deployment.
  • Remote compute orchestration or attachment to existing endpoints.
  • Docker-specific compiler branches.
  • Native Transformers GLiNER serving.
  • Model-cache discovery.
  • Cryptographic authentication of plans.

Implementation is tracked in PR #212.

Dependencies

vLLM Factory integration is pinned by the repository's local-model dependency group.

Dominant language
Python
Stars
126
Forks
20
Avg merge
3d 19h
Merged PRs (30d)
16

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA-NeMo/Anonymizer

All issues in NVIDIA-NeMo/Anonymizer

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.