[Improvement][Worker-SqlTask] Make the default query row limit (QUERY_LIMIT = 10000) configurable
Maintainers usually reply within 1 day
Assessment
This issue has not been assessed yet.
Description
Search before asking
- I had searched in the issues and found no similar feature requirement.
Description
The SQL task caps the number of rows fetched by the worker via Statement.setMaxRows. When the
task-level limit parameter is not set (limit <= 0), it falls back to a hard-coded constant:
// dolphinscheduler-task-plugin/dolphinscheduler-task-sql/.../SqlTask.java
private static final int QUERY_LIMIT = 10000;
stmt.setMaxRows(sqlParameters.getLimit() <= 0 ? QUERY_LIMIT : sqlParameters.getLimit());
The setMaxRows protection was introduced specifically to prevent worker OOM when querying large tables, and the default of 10000 was chosen as the fallback when the task-level limit is unset. However, this fallback default itself cannot be adjusted at deployment time: an administrator who wants a smaller (or larger) global default has to modify the source and rebuild, since there is no entry in common.properties for it.
10000 rows is not a one-size-fits-all default:
For wide rows (many columns / large values), 10000 rows can still consume a significant amount of worker heap, so memory-constrained workers may need a much smaller default.
Conversely, deployments with generous worker resources and legitimate large-result use cases have to set limit on every single task definition, which is a per-task burden for what is really a deployment-level concern.
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
- Dominant language
- Java
- Stars
- 14.5k
- Forks
- 5.1k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 22
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/dolphinscheduler
-
[Bug][API] Switching workflow versions leaves schedules onlinePossibly taken @zhuangchong claimed this 1 day ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
apache/dolphinscheduler#18731 ·
Maintainers usually reply within 1 day
-
[Bug] Switching workflow versions can scan all task definition logsPossibly taken @zhuangchong claimed this 1 day ago. Open
Difficulty 3/5 Half a day Newbie friendliness 68/100
apache/dolphinscheduler#18733 ·
Maintainers usually reply within 1 day
-
[Feature][Datasource] Support Kyuubi three-segment JDBC URL syntax (<sessionVars>?<hiveVars>#<kyuubiVars>) in Kyuubi datasource pluginPossibly taken @silvermissile claimed this 2 days ago. Openbackend feature
apache/dolphinscheduler#18725 · 1 assignee ·
Maintainers usually reply within 1 day
-
[Improvement][APIServer][3.4.3] Task Instance page data loading time out.Possibly taken @karthikeya-tummala claimed this 10 days ago. Openbackend good first issue improvement
apache/dolphinscheduler#18700 · 6 comments · 1 assignee ·
Maintainers usually reply within 1 day
-
[Improvement][API] Suspected inverted session expiry check invalidates active sessions on re-loginPossibly taken @karthikeya-tummala claimed this 10 days ago. Openbackend good first issue improvement
Difficulty 4/5 1-2 days Newbie friendliness 48/100
apache/dolphinscheduler#18699 · 5 comments · 1 assignee ·
Maintainers usually reply within 1 day
All issues in apache/dolphinscheduler
Similar issues
-
code-maintenance
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
Maintainers usually reply within 1 day
-
bug priority: medium Review effort [1-5]: 3 status: needs-triage type: bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
carlos-emr/carlos#5258 ·
Maintainers usually reply within 1 day
-
bug: author create/update with an unknown avatarId returns 404 instead of 400 UNKNOWN_REFERENCEPossibly taken @Vinu2111 claimed this today. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
atomityhq/blog-cms#59 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
Maintainers usually reply within 1 day
-
dev/release: verify-release-candidate.sh runs 0 Java tests with Debian's MavenPossibly taken @kou claimed this today. OpenType: bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
apache/arrow-adbc#4947 ·
Maintainers usually reply within 1 day