Access Delegation

Description

We have several issues in the pipeline that require our operators (and other tools later) to act as authenticated clients of the products they themselve deploy:

• Declarative Resources/Resource CRDs: ZookeeperZnode already exists and we can't make it work in secure clusters because the ZK operator doesn't have access. We want many more of these CRDs.
• Metadata collection: We want to e.g. list tables in Trino, list topics in Kafka etc. for metadata collection
• Scaling: An operator needs to talk to products to change their state (decomissioning, balancing, etc.)
• Cockpit

Our problem is that in real deployments these tools are connected to some customer system we have no control over.

This issue describes our plan on how this question can be solved: How does a Stackable component obtain a credential that the target product accepts, when the customer controls who gets credentials?

The model

An agent per stacklet, deployed per cluster rather than per pod, injected with the credentials needed to talk to that one cluster. It runs a controller that reconciles only the objects relevant to it, e.g. ZookeeperZnode or KafkaTopic.

This came out of the ZK spike in #868 (thanks @labrenbe) and it replaces my earlier idea of a central credential broker that hands out credentials on request at runtime. At least two things that this "agent model" makes easier over my suggestion:

• Credential rotation needs nothing new: The agent consumes a mounted Secret, so our normal restart mechanisms apply.
• There is one credential set per agent, scoped to one cluster, rather than one thing holding a lot of credentials for different clusters

A few questions are still open

• Separate binary or a mode of the product operator
• What lives in the product operator vs. operator-rs
• How the grant is expressed on the product CR

Use cases

They need very different amounts of new design and it's all related (OpenLineage, scaling, Access delegation, metadata collection)

1. Something must happen to a product because the parent CR changed: scale-down, rebalance are the two I can think of right now.. Needs some kind of coordination/communication between agent & operator
2. The agent acts on its own: ZookeeperZnode, KafkaTopic. Needs the agent and a credential, nothing else. Should be relatively easy and it's what was spiked for Kafka already
3. Metadata push: Needs read access to the tool and something to push at

• The agent's identity has to be stable (and secure), because product-side authorization rules target it. I believe our current certs we create (autoTLS) all still look the same even though we had work to change that(?). Either way: This needs to be looked at.
• Someone has to write those authorization rules. Either the customer does it, or we optionally generate them which means we authorize ourselves. That must be a concious decision by the customer/user.

Dimensions

I believe it will not be possible to have one single solution to cover all use-cases and we should design a system that allows for multiple options for the customer. We do not have to implement all of them from the beginning but we should design for them.

Authentication mechanisms

• LDAP / AD
• TLS / mTLS
• OIDC / OAuth2
• Kerberos

Management

This basically defines how much the customer "trusts" us vs. the convenience we can provide

• Static customer managed: The customer provides us with static credentials (e.g. a username/password, a cert, keytab) -> They are responsible for rotation etc.
• Fully-managed: We get the possibility to create credentials in the customer environment (e.g. an intermediate CA, credentials for an OU, ...)
• Platform-managed: Stackable injects something so we can manage authentication ourselves (e.g. a second CA, our own OpenLDAP, chaining to the original, ...)

There's a bit of a fuzzy middle-tier where the customer can give us permission to regenerate passwords (e.g. set a new password on an LDAP account which covers LDAP & Kerberos in most cases, ACME for TLS, not sure for OIDC?). On the one hand it is good because they need to hand over a smaller set of privileges than for the fully-managed version, on the other hand it's extra complexity....

Matrix

As you can see there are still a few things where I'm uncertain. Especially OIDC where I don't have enough knowledge and don't understand everything AI tells me so I kept it out of here for now.

Mechanism	Platform-managed	Customer-managed	Rotation-managed	Fully-managed
Kerberos	Own KDC + cross-realm trust	Static keytab	Delegated password-reset & keytab creation (https://github.com/stackabletech/decisions/issues/96)	Delegated OU, we manage accounts
TLS/mTLS	possible (product specific): Own CA as extra trust anchor	Customer issued cert + key	Not sure... ACME? Not sure if AD has something	1) Customer-signed intermediate, 2) ACME
LDAP / AD	Own LDAP chaining to customer LDAP - not 100% sure...complex for AD? (https://github.com/stackabletech/decisions/issues/95)	Static bind credentials	Delegated password-reset credentials on static accounts (https://github.com/stackabletech/decisions/issues/96)	Delegated OU, we manage accounts
OIDC / OAuth2	https://github.com/stackabletech/decisions/issues/95	Client secret handed over	Not sure how	Not sure how