Conformance tests: Align automated and manual scoring
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- python
- Domain
- testing-qa
Research direction
Start with conformance/src/unexpected_fails.py and inspect the remaining unchecked discrepancies for mypy/namedtuples_define_functional.toml, pyre/directives_reveal_type.toml, and pyre/namedtuples_define_functional.toml. Compare each automated result with its manual judgment and reconcile the remaining cases. Done means the three listed discrepancies are resolved and the conformance scoring agrees with the manual results.
Written by the indexing model from the issue text.
Description
In #1678 I added an automated tool for scoring the conformance test suite, and #1682 and #1683 added some features to support edge cases.
Now I am working on going through the remaining files where the automated conformance judgment on a file differs from the manual one (open PRs: #1684, #1685, #1687, #1688, #1689, #1690, #1691).
I'll use this issue to track the remaining discrepancies (from conformance/src/unexpected_fails.py):
- mypy/dataclasses_usage.toml: Pass vs. Fail (#1684)
- mypy/directives_version_platform.toml: Pass vs. Fail (#1685)
- mypy/namedtuples_define_functional.toml: Pass vs. Fail
- mypy/overloads_basic.toml: Pass vs. Fail (#1687)
- mypy/specialtypes_never.toml: Pass vs. Fail (#1688)
- mypy/typeddicts_alt_syntax.toml: Pass vs. Fail (#1689)
- mypy/typeddicts_type_consistency.toml: Pass vs. Fail (#1690)
- pyre/directives_reveal_type.toml: Unsupported vs. Pass
- pyre/namedtuples_define_functional.toml: Pass vs. Fail
- pyright/classes_override.toml: Pass vs. Fail (#1691)
- pyright/dataclasses_frozen.toml: Pass vs. Fail (#1699)
- pyright/dataclasses_slots.toml: Pass vs. Fail (#1699)
- pyright/dataclasses_transform_field.toml: Pass vs. Fail (#1699)
- pyright/dataclasses_transform_func.toml: Pass vs. Fail (#1699)
- pyright/generics_defaults.toml: Pass vs. Fail (#1700)
- pyright/literals_literalstring.toml: Pass vs. Fail (#1701)
- pyright/overloads_basic.toml: Pass vs. Fail (#1687)
- pyright/protocols_variance.toml: Pass vs. Fail (#1702)
- pyright/specialtypes_never.toml: Pass vs. Fail (#1688)
- pyright/tuples_unpacked.toml: Pass vs. Fail (#1703)
- pytype/dataclasses_usage.toml: Pass vs. Fail (#1684)
- pytype/directives_assert_type.toml: Pass vs. Fail (#1706)
- pytype/directives_version_platform.toml: Pass vs. Fail (#1685)
- pytype/generics_scoping.toml: Pass vs. Fail (#1704)
- pytype/typeddicts_inheritance.toml: Pass vs. Fail (#1705)
Any help is appreciated.
- Dominant language
- Python
- Stars
- 1.8k
- Forks
- 302
- Avg merge
- 23h
- Merged PRs (30d)
- 8
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from python/typing
-
topic: typing spec
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
topic: typing spec
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
topic: documentation
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
topic: documentation
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
-
topic: conformance tests topic: typing spec
Difficulty 3/5 1-2 days Newbie friendliness 72/100
Similar issues
-
Add: hunch Open
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
AbdelStark/awesome-typesafe#104 ·
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
DiamondLightSource/dodal#2211 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
openml/openml-python#1749 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
sipyourdrink-ltd/bernstein#6191 ·