HTTP Storage Plugin INDEX pagination mode is field-order sensitive and should not be

Open
#2,895 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
java
Domain
api, backend

Research direction

Start with the HTTP Storage Plugin's INDEX pagination implementation and the HTTP pagination readme, then reproduce the behavior using the linked Python3 server, configuration, and query. Trace how the results, has_next, and next_page fields are processed; done means pagination works regardless of JSON field order, with regression coverage if the existing test structure supports it.

Written by the indexing model from the issue text.

Description

bug

Describe the bug
HTTP Storage Plugin INDEX pagination mode is field-order sensitive and should not be (JSON is supposed to be unordered). When the pagination fields 'has_next' and 'next_page' come after the data path field 'results', the presence of a next page is ignored. The next page fields are only honored when they come before the data path field.

To Reproduce
Steps to reproduce the behavior:

  1. Start Drill.
  2. Start the Python3 HTTP server that demonstrates the issue:
    https://gist.github.com/hyperbolix/b66f07d5d007ca2df321550241d748ac
  3. Enable the http storage plugin and configure it using this configuration: https://gist.github.com/hyperbolix/7df108c8169cf54bcc72fdc499be1dce
  4. Open the Drill query interface and issue this query:
    select * from http.a_bug_test where total_results_val=300 and page_size_val=10 limit 15
  5. See that it returns 10 results, but it should return 15.
  6. Modify the Python3 HTTP server, moving lines 52-58 so they come before line 50, so that the 'has_next' and 'next_page' fields come before the 'results' field.
  7. Restart the Python3 HTTP server and issue the query again from step 4.
  8. Observe than now 15 results are returned.

Expected behavior
The INDEX pagination mode page token fields should be honored regardless of field ordering in the JSON response. Or, at a minimum, the documentation should clearly indicate that these fields must come before the results field (which is not the order in the example in the HTTP pagination readme today).

Error detail, log output or screenshots
When the 'has_next' and 'next_page' fields come after the data path 'results' field, they are ignored, meaning the result set is incorrectly trimmed short, as though there are not additional pages, yet there are additional pages.

Drill version
Observed in 1.21.1 and also in the latest commit as of 2024-03-26: 749772cb0bd83c1a8fe455410ec80b1e5a9bf239

Dominant language
Java
Stars
2k
Forks
989
Avg merge
1d 8h
Merged PRs (30d)
5

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/drill

All issues in apache/drill

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.