External resources: replace PVC-usage with dependency sources
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- kubernetes, rust
- Domain
- backend, infrastructure
Research direction
Review the current PVC-based tests and documentation, and check whether issue #177 has been implemented before deciding on the remaining work. Done means dependencySources can expose external sources such as S3Bucket without shared PVC storage, or, if #177 is absent, PVC usage is removed from the tests and documentation.
Written by the indexing model from the issue text.
Description
As a user I want to be able to make external resources and dependencies available to pods.
Previously this was implemented using PVCs, which was not independent of cluster capabilities (Access modes for PVCs etc.). A alternative solution is to dispense with PVCs and instead implement a dependencySources that will use a complex enum to expose different sources. In this example below, this will be an S3Bucket:-
---
apiVersion: s3.stackable.tech/v1alpha1
kind: S3Bucket
metadata:
name: my-bucket-resource1
spec:
bucketName: my-example-bucket1
connection:
reference: my-connection-resource
---
apiVersion: spark.stackable.tech/v1alpha1
kind: SparkApplication
...
spec:
...
driver:
...
dependencySources:
# Use "#[serde(flatten)]" in a complex enum called e.g. "Source"
# allowing s3Bucket (and maybe hdfsFolder or url later on)
- s3Bucket: # Option<S3BucketRef>
inline: [...]
# or
reference: my-bucket-resource1
When a dependencySources is declared for a role, the contents of this source will be copied to an internally defined path (not visible to the user) on the respective pods, from where it can be accessed as required. In other words, where previously a job was started to copy resources to a (usually RWX) PVC - which was then used to back a volume mounted for roles - the same mechanism will be used without using shared storage. This adds some redundancy, but a) is independent of PV storage classes and b) is more transparent for the user.
If #177 has not been implemented, then:
- remove PVC-usage from tests and documentation
- N.B. inform @backstreetkiwi when this is complete so that node labels can be removed from T2 cluster nodes
- Dominant language
- Rust
- Stars
- 32
- Forks
- 6
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 12
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from stackabletech/airflow-operator
-
Difficulty 1/5 Under an hour Newbie friendliness 82/100
stackabletech/airflow-operator#898 ·
Maintainers usually reply within 1 day
-
good first issue type/feature-improvement
Difficulty 4/5 3-5 days Newbie friendliness 45/100
stackabletech/airflow-operator#760 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 42/100
stackabletech/airflow-operator#697 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
stackabletech/airflow-operator#638 ·
Maintainers usually reply within 1 day
-
type/feature-new
Difficulty 5/5 Over a week Newbie friendliness 25/100
stackabletech/airflow-operator#593 ·
Maintainers usually reply within 1 day
All issues in stackabletech/airflow-operator
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
2 advisory ignore entries no longer match anything (`bytes`, `time` already patched in `Cargo.lock`)Possibly taken @zyl71 claimed this today. Open
Difficulty 1/5 Under an hour Newbie friendliness 88/100
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100
stackabletech/druid-operator#905 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
bytecodealliance/wrpc#1785 ·
Maintainers usually reply within 1 day
-
2 advisory ignore entries no longer match anything (`quick-xml` already patched in `Cargo.lock`)Open
Difficulty 1/5 Under an hour Newbie friendliness 85/100
Maintainers usually reply within 1 day