trufflesecurity/trufflehog

Fix broken & inaccurate detector unit tests

Open

#3,817 创建于 2024年12月26日

在 GitHub 查看
 (7 评论) (0 反应) (1 负责人)Go (26,285 star) (2,397 fork)batch import
bugcontributions welcomedgood first issuepkg/detectors

描述

The detector unit tests were created by "reverse engineering" the patterns, thus they do not provide any real value or confirm how accurate the detectors are.

Additionally, the structure of the tests are (in my opinion) not maintainable as they are difficult to understand and do not reflect any realistic scenarios.

https://github.com/trufflesecurity/trufflehog/blob/def734a783b2d4542f2b353d326130d618f3c421/pkg/detectors/netsuite/netsuite_test.go#L14-L44

Examples

There are dozens, if not hundreds, of problematic test files. These are illustrative.

BombBomb

The "valid" tests for BombBomb do not match the detector's pattern.

https://github.com/trufflesecurity/trufflehog/blob/def734a783b2d4542f2b353d326130d618f3c421/pkg/detectors/bombbomb/bombbomb.go#L24

https://github.com/trufflesecurity/trufflehog/blob/def734a783b2d4542f2b353d326130d618f3c421/pkg/detectors/bombbomb/bombbomb_test.go#L15

Kraken

The "valid" pattern is nonsensical and not correct base64 encoding. The detector should not match this, that is a defect.

https://github.com/trufflesecurity/trufflehog/blob/def734a783b2d4542f2b353d326130d618f3c421/pkg/detectors/kraken/kraken_test.go#L16

image https://support.kraken.com/hc/en-us/articles/360000919966-How-to-create-an-API-key

viewneo

A few hundred detectors contain tests tightly coupled to the current implementation of PrefixRegex. Any changes to the prefix pattern will break the detector tests, which seems inadvisable.

https://github.com/trufflesecurity/trufflehog/blob/def734a783b2d4542f2b353d326130d618f3c421/pkg/detectors/viewneo/viewneo_test.go#L38-L42

贡献者指南

技术栈
go
领域
testing
议题类型
bug
难度面向新贡献者的预计实现难度，1 表示很小改动，5 表示专家级工作。
4
预计时间有经验贡献者完成调查、实现、测试并准备 pull request 的粗略时间范围。
3-5 days
活动状态议题当前的可参与程度：新鲜、活跃、陈旧、阻塞或等待维护者输入。
active
清晰度议题是否清楚说明期望改动、验收标准和下一步。
mostly clear
前置要求
Go programmingunderstanding of detector patternstest writing in Go
新手友好度1-100 的估计分数，表示该议题对首次贡献者的友好程度。
25
研究方向
The issue identifies that detector unit tests across many files (e.g., pkg/detectors/netsuite/netsuite test.go, bombbomb test.go, kraken test.go, viewneo test.go) are inaccurate and tightly coupled to implementation details. First, review the examples provided to understand the specific problems: invalid test data, incorrect base64 encoding, and reliance on PrefixRegex. Then, investigate a subset of detectors to define a standard for valid test patterns that match real world credentials. After establishing a pattern, refactor the tests to be accurate and maintainable, and consider how to decouple them from internal regex changes. Do not fix all tests at once; propose a plan for incremental improvements and get maintainer feedback.
Fix broken & inaccurate detector unit tests · trufflesecurity/trufflehog#3817 | Good First Issue