Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[Bug]: Findings from first pass at mutation testing

Open
  • Is something wrong?
#245 0 comments 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 7 days

@liunan-ms is already working on this.

Since Sep 1, 2026.

Assessment

This issue has not been assessed yet.

Description

area: testing bug

Mutation Testing Triage — azldev

Generated from two full-repo gremlins passes (merged):

  • Default mutators (5 types: CONDITIONALS_BOUNDARY, CONDITIONALS_NEGATION, ARITHMETIC_BASE, INVERT_NEGATIVES, INCREMENT_DECREMENT) — 87.27% efficacy, 238 LIVED survivors.
  • Inverted config (.gremlins.yaml, 6 less-common types: INVERT_LOGICAL, INVERT_LOOPCTRL, INVERT_ASSIGNMENTS, INVERT_BITWISE, REMOVE_SELF_ASSIGNMENTS) — 85.01% efficacy, 97 LIVED survivors.

The two mutant sets are disjoint, so this is a clean union of 335 survivors.

How to read this

Each LIVED survivor (a mutation no test caught) is classified on three axes:

Axis Values Meaning
Verdict fix / nofix Is this a real assertion gap worth closing? nofix = equivalent mutant, log/error-wording only, cosmetic, or unreachable/defensive.
Pri hi / med / lo / — Severity if the underlying behavior actually broke (only for fix).
Prag y / n Pragmatic to test? Is the test worth the effort? n even for real gaps when catching it needs disproportionate scaffolding (containers, real binaries, network, root) or is inherently brittle (timing/jitter, map-iteration order). All nofix are n.

The actionable set is fix + Prag=y: real gaps that a cheap, focused test would close.

Mutator abbreviations: CB=CONDITIONALS_BOUNDARY, CN=CONDITIONALS_NEGATION, AB=ARITHMETIC_BASE, IN=INVERT_NEGATIVES, ID=INCREMENT_DECREMENT, IL=INVERT_LOGICAL, ILC=INVERT_LOOPCTRL, IA=INVERT_ASSIGNMENTS, IB=INVERT_BITWISE, RSA=REMOVE_SELF_ASSIGNMENTS.

Summary

Default New Combined
LIVED survivors 238 97 335
fix 109 72 181
nofix 129 25 154
fix · hi 16 16 32
fix · med 59 36 95
fix · lo 34 20 54
fix · Prag=y (actionable) — — ~170
fix · Prag=n (real but not worth it) — — ~11

Headline: the tests are genuinely solid (154/335 survivors are equivalent/cosmetic), and of the 181 real gaps, the large majority are cheap, focused unit tests — only a handful (timing/jitter, map-order, integration-only) aren't worth the cost.

High-priority shortlist (the 32 hi gaps, all Prag=y unless noted)

Security / data integrity
Location Mut Why it matters
cmds/component/changed.go:517 IL path-traversal escape guard never fires (silently disabled)
utils/fileutils/copy.go:194 ILC recursive copy stops after first subdir, dropping siblings
utils/archive/archive.go:286 IB extracted file perms forced to 0777
projectconfig/resources.go:108 CN+IL merge wipes earlier-layer RpmRepos
projectconfig/resources.go:116 CN+IL merge wipes earlier-layer RpmRepoSetTemplates
projectconfig/resources.go:124 CN+IL merge wipes earlier-layer RpmRepoSets
projectconfig/resources.go:929 CN GPG-enabled set without key passes validation
Correctness
Location Mut Why it matters
core/components/resolver.go:470 CN spec-path conflict detection inverted (wrong spec accepted)
core/components/resolver.go:145 ILC break drops remaining loose components after dedup hit
core/components/resolver.go:318 ILC break drops later matches after excluded spec
core/components/resolver.go:406 ILC break drops remaining group members after duplicate
core/sources/synthistory.go:682 IL upstream-commit boundary defeated, collects from HEAD
cmds/image/boot.go:347 IL valid image/iso-only boot wrongly rejected
cmds/component/update.go:179 IL false "no components matched" on a matched filter
rpm/spec/edit.go:346 IL InsertTag lands tag in wrong section/package
rpm/spec/edit.go:395 RSA InsertTag can place tag inside %if (conditional, changes build)
rpm/spec/edit.go:777 AB+IN GetHighestPatchTagNumber off-by-one → patch number reuse
utils/dirdiff/dirdiff.go:247 IL every changed file emits "special" message, not a real diff
utils/dirdiff/dirdiff.go:274 CB phantom blank line in added-file diff
utils/dirdiff/dirdiff.go:278 CB phantom blank line in removed-file diff
utils/retry/retry.go:106 CB MaxAttempts==0 → operation never runs
lockfile/store.go:107 CN uncached Get makes every fresh-process lock load fail
rpm/repoquery.go:175 CN wrong/missing --releasever queries wrong distro
rpm/extractor.go:220 CN symlinks silently dropped during RPM extraction
projectconfig/loader.go:108 CN Project-merge early-return skips all later merges
projectconfig/project.go:90 CN valid rpm-repos skip downstream validators
cmds/advanced/mock.go:205 CN explicit --mock-config path silently ignored

Convergence: resources.go:108/116/124 and rpm/extractor.go:220-225 were each flagged by both runs — strong signals. One MergeUpdatesFrom test (non-empty receiver, distinct keys) kills ~6 mutants.

Full triage by package

A. internal/rpm/spec — 66 (43 default + 23 new)
Location Mut Verdict Pri Prag Notes
edit.go:191 CB fix lo y tagFamily all-digit tag returns it, no panic
edit.go:191 IL fix med y tagFamily("Source9999")=="source"; no direct coverage
edit.go:289 CB fix lo y family-tag insert position when tag at line 0
edit.go:293 CB fix lo y insert when only tag is line 0
edit.go:330 CN nofix — n sectionFound still set; equivalent
edit.go:331 IL fix med y InsertTag to missing sub-package → ErrSectionNotFound
edit.go:336 CN nofix — n scan bound only; unobservable
edit.go:337 CN nofix — n sectionEnd bound; unobservable
edit.go:337 IL fix med y sectionEnd from correct package (multi-package spec)
edit.go:346 IL fix hi y InsertTag ignores foreign-section tags
edit.go:385 CB fix lo y inserted tag lands after %endif
edit.go:385 CB nofix — n defensive len() bound; panic-only
edit.go:394 CB nofix — n marginal scan bound / defensive
edit.go:394 ILC/ID fix lo/med y skip past multi-line conditional stops at sectionEnd
edit.go:395 RSA fix hi y InsertTag past nested %if/%endif stays unconditional
edit.go:433 IL fix med y PrependLinesToSection inserts after header
edit.go:604 IL nofix — n Atoi("") fails regardless; equivalent
edit.go:622 CN nofix — n HasSection still true via body; equivalent
edit.go:668 RSA nofix — n totalRemoved 0 before first accumulation
edit.go:681 RSA fix lo y RemovePatchEntry tag-only (not patchlist) match
edit.go:765 CB nofix — n running-max >/>= identical
edit.go:777 AB+IN fix hi y bare Patch:+Patch1 returns 1 (dup-number bug)
edit.go:777 CB nofix — n inner guard makes >=0 equivalent
edit.go:896 CB/IL nofix — n removable section never at line 0; equivalent
edit.go:909 CB nofix — n documented-unreachable fallback
edit.go:999 CB nofix — n %if never equals section bounds; equivalent
edit.go:1000 CB nofix — n directive line never equals section index
edit.go:1004 ILC fix med y RemoveSection balances multiple conditional pairs
edit.go:1015 CB nofix — n running-min identical
edit.go:1027/1050/1063/1093 AB nofix — n error-message line number only
edit.go:1046 AB+IN fix med y removal errors when trim-boundary line has content
edit.go:1085 CB nofix — n %else never coincides; equivalent
edit.go:1085 IL fix lo y %else/%elif spanning-section removal errors
edit.go:1086 CB nofix — n %if/%endif never equals section index
edit.go:1086 CN fix med y RemoveSection with internal %if/%else/%endif
edit.go:1086 IL fix lo y straddling pair not treated as fully-inside
spec.go:258 AB nofix — n internal parse counter; output unaffected
spec.go:259 IA fix med y no duplicate insertion to non-final section
spec.go:259 RSA fix med y visitor cursor correct after InsertLinesAfter
spec.go:440 CN fix med y Visit propagates section-end error
spec.go:449 CN fix lo y Visit propagates SpecEnd callback error
spec.go:559 CB+CN fix lo y trailing "-n" empty; -n overrides positional
spec.go:571 ID nofix — n loop self-corrects via default case
specquery.go:63/144 IL nofix — n logging/diagnostic only
specquery.go:139 ILC fix lo y parse robust to blank line mid-output (continue)
spectool.go:24 IL fix/nofix lo y/n scheme-only "mailto:" non-URL (one of two is equivalent)
spectool.go:50 ILC fix lo y blank line skipped, not truncating list
spectool.go:56 ILC fix lo y malformed line doesn't drop later sources
B. internal/projectconfig — 44 (35 default + 9 new)
Location Mut Verdict Pri Prag Notes
component.go:419 CB nofix — n leading-hyphen pkg name unrealistic
component.go:423 RSA nofix — n accumulation masked by delimiter scan
component.go:472 ILC nofix — n at-most-one-group invariant; break==continue
distro.go:210 CN fix med y amd64 host selects X86_64 mock override
distro.go:210 IL fix med y amd64 + empty X86_64 leaves path unchanged
distro.go:212 CN fix/nofix lo y/n Aarch64-only def must not select arm path
distro.go:212 IL fix med y amd64 + set Aarch64 leaves path unchanged
loader.go:108 CN fix hi y sections after Project still merge (no early return)
loader.go:149 CN nofix — n mergeResources never errors; equivalent
loader.go:161 CN fix med y Resources actually merged when present
overlay.go:258 CN fix med y invalid regex in search-replace-in-spec must error
overlay.go:278 CN fix med y invalid regex in search-replace-in-file must error
package.go:121 ILC nofix — n at-most-one-group invariant
project.go:90 CN fix hi y valid rpm-repos must not skip validators
project.go:94/98/105/109 CN fix med y invalid sets/distro/collision after valid still rejected
resources.go:108 CN+IL fix hi y merge preserves earlier-only RpmRepos (no wipe)
resources.go:108 CN fix med y merge into nil RpmRepos allocates
resources.go:108/116/124 CB nofix — n empty→non-nil map; loop no-ops
resources.go:116 CN+IL fix hi y merge preserves earlier-only templates
resources.go:116 CN fix med y merge into nil templates allocates
resources.go:124 CN+IL fix hi y merge preserves earlier-only sets
resources.go:124 CN fix med y merge into nil sets allocates
resources.go:546 CB×4+CN fix lo y URI scheme first-char letter detection (table test)
resources.go:596 CN fix lo y Default(): ""→binary, others pass through
resources.go:816 CN nofix — n diagnostic description text only
resources.go:921 CN fix med y invalid base-uri / skipped GPG must error
resources.go:929 CN fix hi y GPG-enabled set without key must error (inverted)
resources.go:937 CN fix med y malformed gpg-key must be rejected
resources.go:1049 AB nofix — n 1-based index in error message
resources.go:558 ILC nofix — n break exits switch only (last stmt); equivalent
C. internal/app/azldev/cmds — 71 (47 default + 24 new)
Location Mut Verdict Pri Prag Notes
advanced/mock.go:179 CB/CN nofix — n empty-list guard / interactive RunShell
advanced/mock.go:205 CN fix hi y explicit MockConfigPath returns NewRunner(path)
component/build.go:412 ILC fix med y RPMs after none-channel RPM still moved
component/changed.go:467 IL fix med y no fingerprint-match when in only one lock map
component/changed.go:517 IL fix hi y repoRelPath rejects ".."/escaping paths
component/diffsources.go:70 CN fix med y error on >1 components, success on one
component/preparesources.go:90 CN fix med y no spurious error when resolve succeeds
component/render.go:1002/1119 CN nofix — n guards slog.Debug on error only
component/render.go:1094 ILC fix med y failure marker written for errored after non-errored
component/render.go:1163 IL fix hi y --clean-stale -a without --output-dir/--force accepted
component/render.go:1254 AB nofix — n map capacity hint
component/render.go:1258 CN fix med y aliased component dir not reported orphan
component/render.go:1281/1296 ILC fix lo n orphan detection after sibling — map-iteration-order brittle
component/update.go:179 CN+IL fix hi/med y "no components matched" only when zero matched
component/update.go:263/290/322/327/352 CB/CN nofix — n log/error-wording guards
component/update.go:359 ILC fix med y savable lock written after skipped/upToDate
component/update.go:394 CN+IL fix med y local clears / upstream keeps lock.ImportCommit
component/update.go:401 CN+IL fix lo/med y seed ImportCommit only for upstream first update
component/update.go:402 CN fix med y seed only for upstream source type
component/update.go:651/653/711 ID/AB nofix — n log-only summary/progress counters
component/update.go:700 CB/CN nofix — n progress-display guard
component/update.go:771/784 ID nofix — n progress sync-count only
component/update.go:773/786 ILC fix med y components after up-to-date/reuse-locked still processed
downloadsources.go:138 CN fix med y error when all lookaside URIs fail
downloadsources.go:141 ILC fix med y download stops after first successful URI
downloadsources.go:156/161 CN nofix — n logged display path only
image/boot.go:288 CN fix med y needEmptyDisk/validation for ISO-only boot
image/boot.go:347 IL fix hi y --image-path-only boot accepted
image/pytestrunner.go:296/335 CN/IL fix med y pytest args use absolute image/glob path
pkg/list.go:222/223/226/227 CN fix lo y sort order by PackageName then Type
pkg/list.go:223/227/319 CB nofix — n <→<= equivalent inside distinct-key guard
pkg/list.go:589 CN nofix — n guards conflicting-mapping slog.Warn
pkg/list.go:599 ILC fix med y srpm entries after duplicate packageName still mapped
pkg/list.go:647 ILC fix med y -debuginfo for packages after debug-named entry
pkg/list.go:652/679 ILC fix lo n -debuginfo/-debugsource after collision — map-order brittle
repo/query.go:368/525 AB nofix — n slice capacity hints
repo/query.go:389/399 ILC fix med y --no-srpms skips only source; later arch repo emitted
D. internal/app/azldev (+ core/components, core/sources) — 64 (44 default + 20 new)
Location Mut Verdict Pri Prag Notes
app.go:88/123/195/294/332/378 CN/RSA/IL nofix — n version string / root guard / usage tmpl / log gates
app.go:239 CN fix lo y empty GroupID gets default, explicit preserved
app.go:637 CN fix med y failing post-init callback propagates (injectable)
app.go:648 CN fix med y runnable leaf command survives removeEmptyCommands
command.go:107 IL fix med y error when projectDir set but config nil
command.go:183 IL fix lo y nil / bool result yields no formatted output
command.go:255 CB nofix — n MarshalIndent never empty; equivalent
core/components/resolver.go:145 ILC fix hi y break drops remaining loose components
core/components/resolver.go:318 ILC fix hi y break drops later matches after excluded spec
core/components/resolver.go:406 ILC fix hi y break drops remaining group members
core/components/resolver.go:470 CN fix hi y conflicting spec path errors; matching doesn't
core/components/resolver.go:764-770 CN/ILC nofix — n lock-drift slog.Warn scan
core/components/resolver.go:837/839 CB/CN nofix/fix lo n/y fix-suggestion text (mostly cosmetic)
core/sources/mockprocessor.go:272/278 ILC fix med y later components processed after missing/errored result
core/sources/sourceprep.go:233 IL fix med y .git preserved only when overlays AND withGitRepo
core/sources/sourceprep.go:541/564 CN/AB/IN nofix — n removal-failure warn / count in slog.Info
core/sources/sourceprep.go:1001 CN fix lo y macro filename with '_'/'+' accepted
core/sources/synthistory.go:315/335/371 CN/CB nofix — n updateHead error path / debug-gate / log value
core/sources/synthistory.go:434 CN fix med y branch ref (not detached HEAD) updated after rewrite
core/sources/synthistory.go:682 IL fix hi y commits newer than upstreamCommit excluded (boundary)
env.go:135 CN fix lo y classicToolkitDir derived only when ProjectDir set
env.go:226 CB nofix — n retries==1 clamps either way; equivalent
env.go:341/345/352/357 CN/CB/AB/IA/RSA nofix — n console suggestion-box sizing; cosmetic
env.go:395 CN×4 / IL×2 fix med y newLockStore nil/non-nil per input guards
env.go:461/465 CN fix med y Distro() resolves valid config; errors on empty name
E. internal/utils — 56 (41 default + 15 new)
Location Mut Verdict Pri Prag Notes
archive/archive.go:286 CB fix med y entry sized exactly maxEntryBytes accepted
archive/archive.go:286 IB fix hi y extracted file mode preserved, not forced 0777
dirdiff/dirdiff.go:247 IL fix hi y modified text file yields unified diff, not special-msg
dirdiff/dirdiff.go:274/278 CB fix hi y no phantom blank line in added/removed-file diff
dirdiff/dirdiff.go:406 CB fix med y leading NUL byte detected as binary
dirdiff/json.go:88/96/99 ID fix med y line numbers across sequential add/remove/context
dirdiff/json.go:115 CB+CN fix med y 4-field hunk header parses real start lines
dirdiff/render.go:82 ILC nofix — n empty line trailing only; equivalent
downloader/downloader.go:180 CB/CN nofix — n progress telemetry only
externalcmd/externalcmd.go:268 CB/CN nofix/fix med y configured file listeners tail and receive lines
externalcmd/externalcmd.go:407 IL fix med y RunAndGetOutput errors w/ only file/stdout listener
externalcmd/externalcmd.go:437 CN nofix — n gates stderr warning log
fileutils/aferocustom/.../allowedroots.go:189 ILC nofix — n parents exist; pathsToCreate unchanged
fileutils/copy.go:194 ILC fix hi y siblings after a subdirectory entry still copied
fileutils/embedfs.go:142 CB fix lo y Readdir(0) reads all entries
fileutils/file.go:102 CB nofix — n DEL(0x7f) boundary unreachable in RPM names
iso/iso.go:78 CN nofix — n default description label; cosmetic
kiwi/kiwi.go:359/363 CN fix med y --target-arch/--profile appended iff set
kiwi/kiwi.go:424 CN nofix — n SELinux error-message wording
parmap/parmap.go:76 CB nofix — n clamping limit 1→1; no-op
parmap/parmap.go:122 ILC nofix — n identical Cancelled results; redundant notifications
prereqs/prereqs.go:48/79/88 CN fix med/lo y prompt + auto-install + proceed on success
prereqs/prereqs.go:128 CN/IL fix med/lo y fallback to /usr/lib/os-release; surface other errors
reflectable/prettywriter.go:56/70 CN/IL nofix — n table styling/header-color; cosmetic
reflectable/tablewriter.go:223/224 IL/ILC fix med y OmitEmpty rendering; fields after omitted still rendered
retry/retry.go:106 CB fix hi y MaxAttempts==0 defaults so operation still runs
retry/retry.go:110/114/118/129/132/138/142 CN/AB/IN/CB nofix — n backoff/jitter timing magnitude; non-deterministic
retry/retry.go:82 ILC nofix — n loop exits next iteration anyway; equivalent
retry/retry.go:139 IA/RSA fix/nofix lo n delay within jitter band — brittle/statistical
F. internal/providers, rpm (non-spec), lockfile, repolayout, projectgen, fingerprint — 34 (28 default + 6 new)
Location Mut Verdict Pri Prag Notes
fingerprint/fingerprint.go:154 CB nofix — n len()>0 before sorted-key loop; equivalent
lockfile/lockfile.go:223 ILC fix lo n remaining-component validation — map-order flaky
lockfile/lockfile.go:287 IL fix med y non-lock/hidden files not reported as orphans
lockfile/store.go:107 CN fix hi y fresh (uncached) Get of on-disk lock succeeds
lockfile/store.go:162 CN fix lo y Save propagates write failure
lockfile/store.go:194 CN fix med y Exists false after Remove (cache eviction)
projectgen/projectgen.go:66 CN fix lo y error returned when .gitignore write fails
providers/rpmprovider/rpmprovider.go:66 CN nofix — n telemetry/event args only
providers/sourceproviders/fedorasource.go:265 AB nofix — n sourceIndex+1 in progress log
providers/sourceproviders/fedorasourceprovider.go:298 CN nofix — n deferred cleanup logs Debug only
providers/sourceproviders/sourcemanager.go:209 CB/CN nofix — n retry config; no accessor, brittle
providers/sourceproviders/sourcemanager.go:228/236/237 CN/IL nofix — n Warn/Debug log gates
providers/sourceproviders/sourcemanager.go:322 CN×2/IL fix med/lo y lookaside tried when hash/hashType present
providers/sourceproviders/sourcemanager.go:356/364/425/451 CN fix med y lookaside attempted/built; origin success; UpstreamName
repo/repolayout/layout.go:80 AB nofix — n slice capacity hint
rpm/extractor.go:220 CN fix hi y symlink extracted when fsLinker present
rpm/extractor.go:221 CN/IB fix med y symlink header detected as link
rpm/extractor.go:225 IL fix med y symlink unsupported when fsLinker nil
rpm/mock/mock.go:568/662/666/693 CB/CN fix lo y config-opts only emitted when corresponding value set
rpm/mock/mock.go:671 CB nofix — n len()>0 before sorted-key loop; equivalent
rpm/repoquery.go:175 CN fix hi y --releasever appended with value only when set

Suggested order of attack (Prag=y, hi first)

  1. One test, many mutants: projectconfig/resources.go MergeUpdatesFrom (non-empty receiver, distinct keys) — kills the 108/116/124 wipe mutants from both runs.
  2. Security: changed.go:517 path-traversal guard; archive.go:286 perms; copy.go:194 recursive-copy break.
  3. Resolver break-drops: resolver.go:145/318/406 — multi-component fixtures asserting all survive.
  4. Validation gates: resources.go:929 GPG; loader.go:108 / project.go:90 early-return/skip.
  5. The remaining hi, then med. Skip all Prag=n (real but brittle/expensive: retry jitter, map-order-dependent orphan/debuginfo ordering, no-accessor retry config).
Dominant language
Go
Stars
18
Forks
28
Avg merge
6d 19h
Merged PRs (30d)
19

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/azure-linux-dev-tools

All issues in microsoft/azure-linux-dev-tools

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.