confluentinc/ksql

The configuration for number of query errors in the `show queries` does not work anymore

Open

#9,533 opened on Sep 7, 2022

View on GitHub
 (3 comments) (0 reactions) (1 assignee)Java (5,739 stars) (1,048 forks)batch import
P1buggood first issuestreaming-engine

Description

Describe the bug When a query has errors, a list of errors collected is displayed by the SHOW QUERIES EXTENDED or EXPLAIN <query> commands. This list has a config to limit the number of errors displayed. The config is ksql.query.error.max.queue.size and is default to 10. However, the SHOW QUERIES command is displaying more errors than what it is configured and it keeps growing.

Looking at the code, I found the issue was caused by this fix https://github.com/confluentinc/ksql/pull/8875

The queue used to use an EvictionQueue class which evicts queue elements when a size/capacity has been reached. After the previous fix, the capacity is not used anymore, so the queue becomes unbounded on size. It is still time-bounded because the evict() method is called after adding an element. This evict method removes elements that expired in 1 hour window (configured).

Fix: We should keep the queue bounded to capacity and time (as configured). It is important to keep the queue thread-safe, so maybe the fix should be in the evict() method to remove elements if the queue has reach the configured capacity.

Contributor guide

Tech stack
java
Domain
backenddata
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Java programmingUnderstanding of queue data structures
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60
Research direction
Examine the current implementation of the EvictionQueue class and its evict() method. Review the changes introduced in PR #8875 to understand why the capacity limit was removed. The fix should ensure the queue remains bounded both by capacity (ksql.query.error.max.queue.size) and time (1 hour window). Pay attention to thread safety requirements.
The configuration for number of query errors in the `show queries` does not work anymore · confluentinc/ksql#9533 | Good First Issue