osquery-go bump, then Terraform (providers) and JavaScript (project/package mgmt) lockfile tables

Open
  • Is something wrong?
#134 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
go, terraform
Domain
database

Research direction

Start with the issue’s commit breakdown and the existing go_modules table; the proposed pkg/projectscan move and .terraform.lock.hcl parsing define the Terraform work. Review the listed dependency changes, then compare the new terraform_providers rows with the described provider pins, versions, constraints and hashes. The issue also proposes a JavaScript lockfile table and several osquery-go changes, so confirm the maintainers’ preferred scope and order before starting.

Written by the indexing model from the issue text.

Description

As foreshadowed/threatened/promised, moving on to bumps/cleanups and more new tables!
The pinned osquery-go (v0.0.0-20250131154556-629f995b6947) opens a Thrift transport in NewClient that it never closes, so every query to a table that opens a client (sofa, wifi_network, alt_system_info, crowdstrike_falcon, mcp_servers, go_modules) leaks a socket, and in turn a plain JOIN with users calling a table once per user leaks once per row. The current revision, eb39ad3443df, fixes that and adds column options and an optional buffered transport. While looking at improved dependencies to bump, micromdm/plist v0.3.0 is also overdue: it now bounds its binary parser (nesting depth, object count, decoded size) and rejects self-referencing plists, which is especially important for us because exhausting the stack or memory in any table ends the process that serves EVERY table, which obvs is not something panic recovery can catch..

I'd like to also discuss sending two other pull requests after this, broken down along lines of commits that can be reviewed and taken one at a time, the latter two new-table ones chosen to be the next-largest ones so we'll soon be through at least some of the 'heavier' (changed-files and LoC-wise) tables that are pending to contribute. If something jumps out about the brief game plan I'm mentioning in those sections that I should open separate issues that's all good, just let me know - the tf one feels straightforward and we've been in a good rhythm discussing design that I can potentially get my point across in less words/answer the question as posed 😅

1. The osquery-go bump and updates for some of what it lets every table do (broken up into ~5 commits, currently estimated to land around 1k changed lines total)

Commit Files Lines Tests
osquery-go eb39ad3443df and plist v0.3.0 4 +11 -16
utils.RecoverPanics on every table through one registration helper, and a buffered Thrift transport on the extension server 3 +47 -35
Every table opens its osquery client through utils.OsqueryClienter, so sofa, crowdstrike_falcon and wifi_network can be tested with a mock 14 +155 -47 +83
QueryRowsContext: a query already cancelled opens no client and sends no roster request, and a roster error is reported as a cancellation only when it is one 15 +290 -98 +176 -78
Column options (RequiredColumn, IndexColumn, OptimizedColumn, AdditionalColumn) on constraint columns, following core's definitions in column.h, each asserted by a test 24 +108 -29 +89 -12

The (sequentially third in the table above) utils.OsqueryClienter commit is the only one that is purely for testability and could be postponed/dropped modularly without affecting the others.

2. 🎉New Table! terraform_providers, with a refactor/library-ish move to leverage the scan that the go_modules PR refined (spread over ~2 commits, about 5.3k changed lines, roughly half of which be moves)

The project discovery that is not about Go (user roster, home checks, recorded projects, caps, time budget, completeness warnings) would move out of go_modules into pkg/projectscan, and go_modules keeps its rows and warnings unchanged. terraform_providers lists each pin/provider in a project's .terraform.lock.hcl , with its version, constraints and hashes. Dependency-wise it adds hashicorp/hcl/v2, zclconf/go-cty, x/tools, three small indirect modules, and bumps golang.org/x/sys (v0.36.0 → v0.44.0) and x/sync (v0.17.0 → v0.20.0).

3. 🎉New Table! js_lockfile_packages (spread over ~3 commits, about 4.5k changed lines)

One row per package an npm, pnpm, Yarn or Bun lockfile pins at a recorded project's root: name, version, direct/indirect, scope (although that's apparently not applicable for a version of pnpm), resolved host and URL with credentials redactricated, and integrity record. Core's npm_packages reads installed node_modules; this reads what projects pin, including projects with nothing installed. No new dependency.
Two small consolidations that didn't feel right to stick alongside the osquery-go PR but still are enhancements that we'd benefit from: setting up a shared powermetrics 'runner' for energyimpact, socpower and thermalthrottling, and equality-constraint helpers in pkg/utils for five tables that help reduce duplicated lines.

Does this order and grouping work for you, and are the bumps and added HCL etc. dependencies acceptable for the Terraform table?

Dominant language
Go
Stars
134
Forks
42
Avg merge
3d 23h
Merged PRs (30d)
3

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from macadmins/osquery-extension

All issues in macadmins/osquery-extension

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.