awslabs/stickler

View on GitHub

A library for evaluating structured data and AI outputs with weighted field comparison and custom comparators

Stars: 45
Forks: 34
Open beginner issues: 5
Indexed issues: 54
Avg merge: 3d 10h
Merged PRs (30d): 28

Dominant language: Python
License: Apache-2.0
Last GitHub push: Sep 14, 2026
Latest indexed: Sep 14, 2026
Contributing guide: Contributing guide
Code of conduct: Code of conduct
Beginner labels: good first issue help wanted

Beginner friendly Everything

Open All

54 open issues indexed

Converge `compare()` and `compare_with()` on absent-on-both (1.1.0) Open

Difficulty 5/5 Over a week Newbie friendliness 30/100

awslabs/stickler#357 · Opened Sep 14, 2026
Docs: versioned docsite with `mike` (dev preview + per-release pinning) Open

Difficulty 4/5 3-5 days Newbie friendliness 64/100

awslabs/stickler#356 · Opened Sep 14, 2026
CI: build the docs on pull requests Open

Difficulty 3/5 1-2 days Newbie friendliness 74/100

awslabs/stickler#355 · Opened Sep 14, 2026
`matched` changed semantics in 1.0.0 without a documented break Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

awslabs/stickler#354 · 1 comment · Opened Sep 14, 2026
auto.md defines `matched` as the removed `all_fields_matched`, and README's Quick Reference publishes clip-under-threshold as `false` when it resolves to `True` Open

bug documentation

Difficulty 4/5 3-5 days Newbie friendliness 48/100

awslabs/stickler#352 · Opened Sep 11, 2026
A mixed list of plain models and StructuredModels raises or scores depending on element order Open

bug documentation

Difficulty 4/5 3-5 days Newbie friendliness 68/100

awslabs/stickler#351 · Opened Sep 11, 2026
[BUG]: model_from_json silently ignores every unrecognized config key, so "threshhold": 0.99 builds at 0.5 Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 58/100

awslabs/stickler#350 · Opened Sep 11, 2026
Schema round trip silently drops model-scope keys on `["object","null"]`, mapping fields, and union-typed nested models Open

bug documentation

Difficulty 5/5 Over a week Newbie friendliness 45/100

awslabs/stickler#349 · Opened Sep 11, 2026
Comparator, threshold and clip resolve differently on the JSON Schema and Stickler-config paths than on the Python path, so #300 and #316 landed on one entry point each Open

bug documentation

Difficulty 5/5 Over a week Newbie friendliness 45/100

awslabs/stickler#348 · Opened Sep 11, 2026
Stickler threshold & extension typo-validation not applied to objects referenced via $ref/$defs Open

Difficulty 3/5 1-2 days Newbie friendliness 82/100

awslabs/stickler#340 · Opened Sep 10, 2026
[BUG]: MAP scoring needs to be page aware Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 48/100

awslabs/stickler#339 · Opened Sep 10, 2026
Behaviour table overstates the [Plain, NoteSM] row for a bare Optional[List[Any]] annotation (measured 0.5, documented 1.0) Open

documentation

Difficulty 2/5 1-3 hours Newbie friendliness 84/100

awslabs/stickler#338 · Opened Sep 10, 2026
Plain model holding a non-JSON-able value (np.ndarray) scores 0.5 against an identical copy Open

bug

Difficulty 5/5 Over a week Newbie friendliness 45/100

awslabs/stickler#337 · Opened Sep 10, 2026
Class-mismatch warning fires from discarded cost-matrix cells, then warn_once silences the real mismatch Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 75/100

awslabs/stickler#336 · Opened Sep 10, 2026
docs: llm-as-a-judge examples now print 0.0 where the prose promises partial credit Open

documentation

Difficulty 2/5 1-2 days Newbie friendliness 84/100

awslabs/stickler#335 · Opened Sep 10, 2026
[BUG]: a comparator-named threshold is discarded on the JSON-Schema and stickler-config paths Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 78/100

awslabs/stickler#334 · Opened Sep 10, 2026
[BUG]: List[Dict[...]] exports clip_under_threshold=false, and its items are mistyped as string Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 68/100

awslabs/stickler#333 · Opened Sep 10, 2026
[BUG]: a refused list element is counted as a false discovery but produces no non-match record Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 68/100

awslabs/stickler#332 · Opened Sep 10, 2026
List field scoring: docs contradict each other, raw_similarity_score is not raw, and List[StructuredModel] ignores clip_under_threshold Open

bug documentation

Difficulty 4/5 3-5 days Newbie friendliness 45/100

awslabs/stickler#330 · Opened Sep 9, 2026
docs: stale defaults in model_from_json docstring, plus three inaccurate doc claims split out of #328 Open

Difficulty 3/5 1-2 days Newbie friendliness 68/100

awslabs/stickler#329 · Opened Sep 9, 2026
A different class, including a subclass, should always be a false discovery Open

bug

Difficulty 5/5 Over a week Newbie friendliness 35/100

awslabs/stickler#327 · Opened Sep 9, 2026
[BUG]: one unknown key in comparator_config silently discards all the valid ones Open

Difficulty 3/5 1-2 days Newbie friendliness 76/100

awslabs/stickler#326 · 1 comment · Opened Sep 8, 2026
Make infer_unspecified_fields default to true Open

Difficulty 4/5 3-5 days Newbie friendliness 42/100

awslabs/stickler#325 · Opened Sep 8, 2026
Hungarian gate for List[Optional[Model]] depends on ground-truth element order Open

Difficulty 3/5 1-2 days Newbie friendliness 78/100

awslabs/stickler#322 · Opened Sep 8, 2026
A wrong-class element in a list is not counted as a false discovery Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

awslabs/stickler#321 · Opened Sep 4, 2026
[BUG]: a non-numeric string in a numeric field aborts the run instead of scoring 0.0 Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 52/100

awslabs/stickler#310 · 1 comment · Opened Sep 2, 2026
[BUG]: ComparableField rejects default_factory, so collection fields cannot use the idiomatic pydantic default Open

bug

Difficulty 2/5 1-3 hours Newbie friendliness 88/100

awslabs/stickler#306 · Opened Sep 2, 2026
[FEATURE]: Compare paired evaluation runs by document ID Open

Difficulty 5/5 Over a week Newbie friendliness 48/100

awslabs/stickler#303 · Opened Aug 30, 2026
[BUG]: pretty_print does not accept EvalResult, prints a false success message, and is untested Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 68/100

awslabs/stickler#298 · Opened Aug 28, 2026
Delete the superseded ConfusionMatrixCalculator nested and list methods (329 dead lines) and their StructuredModel wrappers Open

enhancement help wanted

Difficulty 3/5 1-2 days Newbie friendliness 74/100

awslabs/stickler#296 · 1 comment · Opened Aug 28, 2026
[BUG]: Bulk_Confidence_AUROC notebook raises ValueError on execution, and always has Open

bug

Difficulty 1/5 Under an hour Newbie friendliness 92/100

awslabs/stickler#292 · Opened Aug 27, 2026
[REFACTOR]: Unify the three optional-unwrapping helpers into one shared implementation Open

Difficulty 4/5 3-5 days Newbie friendliness 52/100

awslabs/stickler#282 · Opened Aug 25, 2026
[FEATURE]: restore per-field overrides on auto (dotted paths, loud errors, model-list guard) Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 32/100

awslabs/stickler#263 · 2 comments · Opened Aug 18, 2026
No per-field score exists for leaves under List[StructuredModel], and the reason is not recorded Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 82/100

awslabs/stickler#249 · 4 comments · Opened Aug 17, 2026
[BUG]: unrecognized x-aws-stickler-* keys are silently dropped, and explain() reports the fallback as "explicit" Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 58/100

awslabs/stickler#210 · 5 comments · Opened Aug 6, 2026
[FEATURE]: batch comparator hook so network-backed comparators do not make n*m sequential calls Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 48/100

awslabs/stickler#204 · 2 comments · Opened Jul 31, 2026
[CHORE]: declare a stable result contract and consolidate compare_with()'s nine output flags Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

awslabs/stickler#202 · 2 comments · Opened Jul 31, 2026
[BUG]: ComparableField makes every field optional, so Optional[T] and required T are indistinguishable Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 55/100

awslabs/stickler#189 · 1 comment · Opened Jul 30, 2026
CI does not gate merges: no branch protection on dev/main, and first-contributor PRs merge with zero CI runs Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

awslabs/stickler#182 · Opened Jul 27, 2026
Follow-ups from PR #127 (list-form nullable types): order-dependent list routing, PEP 604 export, round-trip gaps, docs Open

Difficulty 4/5 3-5 days Newbie friendliness 52/100

awslabs/stickler#179 · 3 comments · Opened Jul 27, 2026
proposal: automate the release process (prepare-release + tag-and-publish) Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 42/100

awslabs/stickler#177 · 1 comment · Opened Jul 24, 2026
[BUG]: need to update the bounding box examples in the rich value pattern to be page aware. Open

bug

Difficulty 2/5 1-3 hours Newbie friendliness 74/100

awslabs/stickler#174 · Opened Jul 17, 2026
[BUG]: Extra/hallucinated fields on matched list elements are not surfaced in per-field details Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 65/100

awslabs/stickler#165 · 1 comment · Opened Jul 2, 2026
[FEATURE]: Cohesive error handling — typed exception hierarchy and consistent user-facing messages Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

awslabs/stickler#140 · Opened May 28, 2026
[FEATURE]: Collapse the non-match triplet (non_match_field / non_matches_helper / non_match_collector) Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 45/100

awslabs/stickler#137 · Opened May 22, 2026
[FEATURE]: Inline ConfusionMatrixBuilder into ComparisonEngine Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 48/100

awslabs/stickler#136 · Opened May 22, 2026
[RFC]: Resolve the parallel evaluation paths — trees/ (ANLSTree) vs ComparisonEngine Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 28/100

awslabs/stickler#135 · Opened May 22, 2026
[FEATURE]: Consolidate the *_helper.py modules in structured_object_evaluator/models/ Open

enhancement good first issue

Difficulty 5/5 Over a week Newbie friendliness 38/100

awslabs/stickler#134 · 1 comment · Opened May 22, 2026
[FEATURE]: Finish the StructuredModel delegation refactor Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 32/100

awslabs/stickler#133 · Opened May 22, 2026
[FEATURE]: Automated execution checks for example notebooks and scripts Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 52/100

awslabs/stickler#118 · 1 comment · Opened May 1, 2026

awslabs/stickler

Get new issues in your inbox