增大并行度后ALL模式维表缓存带来的数据库连接及内存压力

Open

#300 opened on Apr 15, 2020 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty: 5/5
Estimated time: Over a week
Newbie friendliness: 25/100

Issue type: Feature
Clarity: Needs clarification
Activity status: Stale
Tech stack: java, sql
Domain: databases, performance, stream-processing

Research direction

Start by locating the ALL-mode dimension-table cache implemented with flatMap, then trace how SQL parallelism causes each parallel task to load the dimension data. Investigate the existing scheduled refresh behavior and how task managers and slots are represented. Done means one copy is loaded per task manager while refreshes continue to work without exhausting database connections or causing excessive memory use.

Written by the indexing model from the issue text.

Description

源码中ALL模式维表缓存是个flatMap操作，当任务数据量和计算压力增大后，增大SQL的并行度，每个并行度都会去维表中加载数据，这样一个是容易出现把维表连接打满的情况，另一个就是带来不小的内存开销。

因为存在定时更新数据，广播可能不太好做，能否实现一个taskmanager只加载一份维表数据，而不是每个slot都去加载一遍

Dominant language: Java
Stars: 2k
Forks: 913
PR merge metrics: No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

Read the whole issue, then the project's contributing guide.
Comment on the issue to say you are picking it up — it saves two people doing the same work.
Fork the repository and make your change on a branch.
Open a pull request that references the issue number.

Similar issues

[BUG] Unified agent on Bedrock Converse with no tools is unusable: an empty `toolConfig` is emitted and Bedrock rejects the request Open

bug untriaged

Difficulty 2/5 1-3 hours Newbie friendliness 84/100

opensearch-project/ml-commons#5094 · Opened Sep 22, 2026
[Bug report] iceberg rest server load_table with refs fails precondition check "Cannot set metadata location with changes to table metadata: %s changes" for snapshots with statistics Open

bug

Difficulty 2/5 1-3 hours Newbie friendliness 85/100

apache/gravitino#13452 · Opened Sep 22, 2026
Support Test Project Creation for New Projects Open

emitter:client:csharp feature

Difficulty 2/5 1-3 hours Newbie friendliness 72/100

microsoft/typespec#12027 · 1 comment · 1 reaction · Opened Sep 22, 2026
Job worker stops polling silently when a poll throws an Error Open

affects/8.10 affects/8.9 component/clients kind/bug likelihood/mid severity/mid

Difficulty 2/5 1-3 hours Newbie friendliness 78/100

camunda/camunda#63825 · Opened Sep 22, 2026
Two open-case totals on one screen: the Programs tile says 15,858 and the nav badge says 15,868 Open

bug frontend maui-pilot

Difficulty 2/5 1-3 hours Newbie friendliness 72/100

Taleef7/workwell#651 · 1 comment · Opened Sep 22, 2026

More Java issues