Per-bucket object storage credentials with key rotation

Start by tracing CloudStack's object-storage account, bucket, credential, migration, and API/UI entry points; the issue names no files or tests. Compare the current account-scoped behavior with the proposed Ceph RGW account model and identify the boundaries for per-bucket credentials and two-slot rotation. Done means the Ceph-specific feature, migration path, APIs, UI, and compatibility behavior are covered.

Problem

CloudStack provisions object storage credentials per account. The first bucket an account creates provisions one backend identity, its access and secret key are stored against the account, and every later bucket reuses that same pair. There is no API to rotate or regenerate it.

An S3 key often exists to be used outside the platform: by a backup tool, a CI pipeline, a third-party application. Those consumers sit outside the operator's span of control. Sharing one pair across every bucket an account owns means:

• one leaked key exposes every bucket that account owns, not just the one it was issued for;
• a bucket cannot be handed to an external consumer without handing over every other bucket with it;
• each bucket usually serves a different tool, with a different supplier and a different supply-chain risk, so each bucket is properly its own boundary;
• there is no safe way to rotate, because there is only one key and revoking it breaks every consumer at once.

Using a separate CloudStack account per bucket does give distinct keys, but it brings its own resource limits, users and networks, and confuses tenancy with a storage-scoping requirement.

Proposed change

A credential per bucket. Each bucket gets its own backend identity instead of sharing the account's, so a leaked key exposes one bucket.

Two key slots per credential. Each credential holds up to two independent key pairs. Rotation is: create a new pair in the free slot while consumers keep using the other, move consumers across, then revoke the old one. The slots are independent rather than a rolling pipeline, which mirrors the two keys of an Azure storage account and the two access keys of an AWS IAM user. Rotating a credential's only active key is safe, because the new key is created before the old one is removed.

Existing buckets are untouched. A bucket with no credential of its own behaves exactly as today. The schema addition is additive and nothing is migrated.

Implementation decisions

I considered two ways to give a bucket its own identity on Ceph RGW.

Plain RGW users per bucket
I found that the writer of an object owns the object, so giving an existing bucket a new identity splits ownership between the old account user and the new one. A public bucket policy then fails on the older objects, the old key keeps reaching them, and the admin REST API cannot repair it: it can relink a bucket to another owner, but only the radosgw-admin bucket chown command rewrites the objects inside, and CloudStack accesses the gateway over REST and S3 with no shell access.

RGW accounts (the chosen option)
Ceph Squid added accounts. An account sits above users, in the same way an AWS account sits above its IAM users. It holds users, each with their own keys and policies, and the account owns the buckets and objects, not the user that created them. This fixes the problem above: ownership no longer depends on which user wrote an object, so a bucket can be given its own identity and a bucket policy applies to everything in it.

Each CloudStack account maps to one RGW account on each object store. The account's existing RGW user becomes the root user of that RGW account. It is the same user, so its keys keep working and its buckets move into the account with it. Each bucket then gets an IAM user inside the account, with a policy limited to that bucket. That user's two access keys are the two key slots.

Objects written before the account was created are still recorded against the original user, and an IAM policy alone cannot read them. To cover this, the bucket policy also grants access to the bucket's IAM user. CloudStack generates that policy together with the bucket's public or private setting.

That introduces some minimum versions. As far as I can see, Cloudian is already using this model, however RGW accounts arrived in Ceph Squid (v19). Older gateways keep today's behaviour and report themselves as unable to support the feature, rather than getting a half-working version of it. Ceph before Squid is already end of life.

Migration

Adopting an existing account into an RGW account is permanent at the backend, so it is never implicit. An administrator triggers it per account per store, existing buckets and their keys keep working throughout, and each bucket is then moved to its own credential individually. The final step rotates the account key, so that the key shared before the migration no longer opens anything.

Scope

In scope: Ceph RGW, the per-bucket credential model, two-slot rotation and revocation, the migration path, and the API and UI for all of it.

Out of scope for the first pass: bulk migration of buckets, and implementations for MinIO, Cloudian and other providers. Those providers are unaffected and keep today's behaviour.

Discussion

Raised on dev@ as "[DISCUSS] Per-bucket credentials and key rotation for object storage".