Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Operator stops reconciling permanently after its watch connections drop while idle

Open
  • Is something wrong?
#936 3 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
kubernetes, rust
Domain
devops, infrastructure

Research direction

Start with the controller watch and backoff behavior in kube-runtime/src/controller/mod.rs around the referenced line, then inspect how this operator configures Controller and watcher::Config. Reproduce the idle watch disconnect and compare the constant-backoff and reduced-timeout approaches described in the issue. Done means affected operators reconnect after dropped idle watches and continue reconciling without a pod restart.

Written by the indexing model from the issue text.

Description

type/bug
Affected Stackable version

26.7

Affected Trino version

481

Current and expected behavior

The trino, hive and airflow operators flip from recoverable watch-stream errors to a permanent failed to start watching object: client error (Connect) loop and never reconcile again until the pod is restarted. While wedged the operator looks healthy to Kubernetes: the process is running, its conversion webhook still answers.

Expected

A controller whose watch connection is dropped re-establishes it and continues reconciling, as it does for the first several drops.

Actual

ERROR kube_client::client::builder: failed with error client error (Connect)
WARN stackable_operator::logging::controller: Queued reconcile resulted in an error
  controller.name="trinocluster.trino.stackable.tech"
  error=failed to start watching object: ServiceError: client error (Connect)
  error.sources=[ServiceError: client error (Connect), client error (Connect), deadline has elapsed]

The problems seems to be upstream in kube-rs: https://github.com/kube-rs/kube/blob/main/kube-runtime/src/controller/mod.rs#L1704
Possible explanation for this issue: All watch triggers get merged into a single stream that then gets joint backoff - so if an operator watches e.g. 7 resources, the backoff fires instantly 6 times, which quickly leads to high sleep durations that ultimately go beyond the timeout.

While kube-rs has a long timeout configured (>200 secs) on the observed cluster some kind of proxy for the apiserver seems to terminate idle watches after 60 seconds, which is reliably triggered for the mentioned operators.

Possibly related: https://github.com/kube-rs/kube/issues/1915

As mentioned in the beginning, this potentially affects all our operators

Possible solution

Possible Workarounds:

Smaller, constant backoff
    use kube::runtime::utils::Backoff;

    struct ConstantBackoff(Duration);
    impl Iterator for ConstantBackoff {
        type Item = Duration;
        fn next(&mut self) -> Option<Duration> { Some(self.0) }
    }
    impl Backoff for ConstantBackoff {
        fn reset(&mut self) {}
    }

    Controller::new(api, watcher::Config::default())
        .trigger_backoff(ConstantBackoff(Duration::from_secs(2)))
Reduce timeout on watcher side:

Maybe as an optional parameter or something.

const WATCH_TIMEOUT_SECS: u32 = 40;

fn watch_config() -> watcher::Config {
    watcher::Config::default().timeout(WATCH_TIMEOUT_SECS)
}
Additional context

No response

Environment

No response

Would you like to work on fixing this bug?

None

Dominant language
Rust
Stars
63
Forks
14
Avg merge
1d 8h
Merged PRs (30d)
10

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from stackabletech/trino-operator

All issues in stackabletech/trino-operator

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.