HTTP Storage Plugin INDEX pagination mode is field-order sensitive and should not be
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
Research direction
Start with the HTTP Storage Plugin's INDEX pagination implementation and the HTTP pagination readme, then reproduce the behavior using the linked Python3 server, configuration, and query. Trace how the results, has_next, and next_page fields are processed; done means pagination works regardless of JSON field order, with regression coverage if the existing test structure supports it.
Written by the indexing model from the issue text.
Description
Describe the bug
HTTP Storage Plugin INDEX pagination mode is field-order sensitive and should not be (JSON is supposed to be unordered). When the pagination fields 'has_next' and 'next_page' come after the data path field 'results', the presence of a next page is ignored. The next page fields are only honored when they come before the data path field.
To Reproduce
Steps to reproduce the behavior:
- Start Drill.
- Start the Python3 HTTP server that demonstrates the issue:
https://gist.github.com/hyperbolix/b66f07d5d007ca2df321550241d748ac - Enable the http storage plugin and configure it using this configuration: https://gist.github.com/hyperbolix/7df108c8169cf54bcc72fdc499be1dce
- Open the Drill query interface and issue this query:
select * from http.a_bug_test where total_results_val=300 and page_size_val=10 limit 15 - See that it returns 10 results, but it should return 15.
- Modify the Python3 HTTP server, moving lines 52-58 so they come before line 50, so that the 'has_next' and 'next_page' fields come before the 'results' field.
- Restart the Python3 HTTP server and issue the query again from step 4.
- Observe than now 15 results are returned.
Expected behavior
The INDEX pagination mode page token fields should be honored regardless of field ordering in the JSON response. Or, at a minimum, the documentation should clearly indicate that these fields must come before the results field (which is not the order in the example in the HTTP pagination readme today).
Error detail, log output or screenshots
When the 'has_next' and 'next_page' fields come after the data path 'results' field, they are ignored, meaning the result set is incorrectly trimmed short, as though there are not additional pages, yet there are additional pages.
Drill version
Observed in 1.21.1 and also in the latest commit as of 2024-03-26: 749772cb0bd83c1a8fe455410ec80b1e5a9bf239
- Dominant language
- Java
- Stars
- 2k
- Forks
- 989
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 5
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/drill
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
bug
Difficulty 4/5 3-5 days Newbie friendliness 35/100
-
bug
Difficulty 4/5 3-5 days Newbie friendliness 28/100
-
bug
Difficulty 4/5 3-5 days Newbie friendliness 35/100
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
Similar issues
-
bug
Difficulty 1/5 Under an hour Newbie friendliness 90/100
apache/cloudstack#14222 ·
-
[BUG]茶杯方块在取茶时会引发崩溃 Open
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
1.0.0-alpha2 Type/Improvement
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
wso2/dpdp-accelerator#272 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
infinispan/infinispan#18150 ·
-
area/frontend
Difficulty 2/5 1-3 hours Newbie friendliness 65/100