Enable TaskMonitor warnings of slow tasks by default
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 25/100
Research direction
Start with the linked hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/TaskMonitor.java and inspect how hbase.taskmonitor.rpc.warn.time disables the check. Confirm which tasks are monitored and settle the default threshold, such as one hour. Done means slow-task warnings are enabled by default without the older false-warning behavior described in HBASE-22935.
Written by the indexing model from the issue text.
Description
HBase has a TaskMonitor that can warn on any tasks that take too long.
https://github.com/apache/hbase/blob/47996d6c2128815e45bb8bdb6e3a470bfddd6106/hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/TaskMonitor.java#L49
By default it does not warn as the setting hbase.taskmonitor.rpc.warn.time defaults to 0 which disables the check. But it's definitely useful to have a warning like this.
We saw tasks stuck for a week at a customer and this went unnoticed. As I'm not 100% sure on which tasks (i.e. also Procedures?) are monitored I suggest a rather high threshold.
Maybe 1 hour? (Even though most tasks should finish within seconds)
This is a warning that we then - later - should extract from the logs to convert into an alert and/or at least have a dashboard (in Grafana or Opensearch Dashboard) to show these.
The log message will start with: "Task may be stuck"
Also note that older versions have a bug that can wrong warnings: https://issues.apache.org/jira/browse/HBASE-22935
- Dominant language
- Rust
- Stars
- 21
- Forks
- 6
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 9
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from stackabletech/hbase-operator
-
Difficulty 5/5 Over a week Newbie friendliness 15/100
stackabletech/hbase-operator#632 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
stackabletech/hbase-operator#582 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 45/100
stackabletech/hbase-operator#583 ·
-
type/bug
Difficulty 4/5 3-5 days Newbie friendliness 30/100
stackabletech/hbase-operator#551 ·
-
customer-request refinement-needed
Difficulty 5/5 Over a week Newbie friendliness 25/100
stackabletech/hbase-operator#510 ·
All issues in stackabletech/hbase-operator
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Eynzof/Hermes-CN-Desktop#616 ·
-
bug rules
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
app bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
IronCoreLabs/ironcore-alloy#346 ·
-
good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 65/100