Risky testing of versions of provisional extensions
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 30/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- cpp
- Domain
- testing-qa
Research direction
Start with the version checks in test_conformance/extensions/cl_khr_command_buffer/basic_command_buffer.h, mutable_command_basic.h, and mutable_command_info.cpp. Review how each handles extension versions older or newer than the tested version, then resolve the expected behavior before changing the checks. Done means all three checks handle version mismatches consistently without running incompatible tests.
Written by the indexing model from the issue text.
Description
OpenCL CTS includes testing for the provisional extensions cl_khr_command_buffer and cl_khr_command_buffer_mutable_dispatch. Both of these contain a warning:
This is a provisional extension and must be used with caution. See the description of provisional header files for enablement and stability details.
This warning is because as long as the extension has provisional status, future versions must be anticipated to potentially contain both ABI-incompatible and API-incompatible changes. Any code that wishes to use this extension must pay attention to the extension's version. This is not hypothetical: ABI-incompatible and API-incompatible changes have been made earlier this year.
For cl_khr_command_buffer, the extension version is checked by https://github.com/KhronosGroup/OpenCL-CTS/blob/918d561c6c42fb7f3e2ffe1d1574c0c37bb2b15b/test_conformance/extensions/cl_khr_command_buffer/basic_command_buffer.h#L102-L112
For cl_khr_command_buffer_mutable_dispatch, the extension version is checked by https://github.com/KhronosGroup/OpenCL-CTS/blob/918d561c6c42fb7f3e2ffe1d1574c0c37bb2b15b/test_conformance/extensions/cl_khr_command_buffer/cl_khr_command_buffer_mutable_dispatch/mutable_command_basic.h#L90-L95 and https://github.com/KhronosGroup/OpenCL-CTS/blob/918d561c6c42fb7f3e2ffe1d1574c0c37bb2b15b/test_conformance/extensions/cl_khr_command_buffer/cl_khr_command_buffer_mutable_dispatch/mutable_command_info.cpp#L127-L133
All three checks have the following in common: if the reported extension version is older than what OpenCL CTS tests, skip the test.
Please consider what happens when for whatever reason, a not fully up to date version of OpenCL CTS is used. Suppose today's version of OpenCL CTS is used to test a future OpenCL implementation that reports availability of version 0.9.7 of cl_khr_command_buffer. The result is that the current logic to skip the test fails, the current version of the test is run, and because 0.9.7 is permitted to break compatibility, we get implicitly undefined behavior that may manifest itself in any number of ways. If we are lucky, functions that are looked for will not be found. If we are unlucky, function that are looked for will be found, but will not have the expected behavior.
In addition, although it only rarely makes sense to run older versions of OpenCL CTS1, if people look for guidance on how to use these extensions by checking how they get tested in OpenCL CTS and the version checks that are used here are copied elsewhere, it becomes more likely that they are used in software that is not updated as frequently for newer versions of the extension.
I would like to prevent this from becoming an issue in the future, and would like to work to submit a pull request to make improvements here, but would like to request feedback on what behavior would be desired.
At time of writing, the most recent version of the cl_khr_command_buffer extension is 0.9.5.
What behavior is expected when running against an implementation that supports version 0.9.3 (an older version)?
At the moment, the result is that tests are skipped. It is possible to leave it that way, but a possible alternate behavior is to mark the test as failed.
What behavior is expected when running against an implementation that supports version 0.9.7 (a newer version)?
At the moment, the result is that tests are run and we risk undefined behavior. It is possible to leave it that way, but possible alternate behaviors are to mark the test as skipped, or to mark the test as failed.
My personal preference here is, in both cases, to mark the test as failed: the current OpenCL spec says that the version of the extension is 0.9.5, so an implementation that claims to implement this extension, but reports its version as something other than 0.9.5, does not implement the current OpenCL spec. That said, all options are justifiable.
Background: uncertainty over the API and ABI implications is why in oneAPI Construction Kit, we have not yet updated to the current version of the cl_khr_command_buffer extension. We would like to update to the current version. Doing so would break our compatibility with older versions of OpenCL CTS. We would like to say that we consider this an issue in those older versions of OpenCL CTS, and work to ensure that this issue does not arise again in the future, but we do not yet know whether this point of view will be shared.
1. A legitimate reason for running older versions of OpenCL CTS might be that a change that breaks compilation for some users is accidentally merged, and this goes unnoticed for a while.
- Dominant language
- C++
- Stars
- 234
- Forks
- 240
- Avg merge
- 6d 12h
- Merged PRs (30d)
- 15
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from KhronosGroup/OpenCL-CTS
-
cognizant-backlog
Difficulty 2/5 1-3 hours Newbie friendliness 61/100
KhronosGroup/OpenCL-CTS#2882 · 1 comment ·
Maintainers usually reply within 1 day
-
cl_khr_command_buffer: negative_enqueue_inconsistent_device expects CL_INCOMPATIBLE_COMMAND_QUEUE_KHR instead of CL_INVALID_DEVICEPossibly taken @kpet claimed this 11 days ago. Open
Difficulty 1/5 Under an hour Newbie friendliness 82/100
KhronosGroup/OpenCL-CTS#2872 · 1 comment · 1 assignee ·
Maintainers usually reply within 1 day
-
test_compiler: feature macros for cl_ext_float_atomics are not testedPossibly taken @shajder claimed this 10 days ago. Opencognizant-triage
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
KhronosGroup/OpenCL-CTS#2818 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
KhronosGroup/OpenCL-CTS#2699 · 1 comment ·
Maintainers usually reply within 1 day
-
cleanup/refactoring
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
KhronosGroup/OpenCL-CTS#2057 ·
Maintainers usually reply within 1 day
All issues in KhronosGroup/OpenCL-CTS
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
useless statementOpen
Difficulty 1/5 Under an hour Newbie friendliness 86/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
grizzlypeak3d/DJV#962 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day