Document how to update spark jobs that are called from Airflow DAGs
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 42/100
- Issue type
- Documentation
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- kubernetes, spark
- Domain
- data-engineering, devops, documentation
Research direction
Start by reproducing the airflow demo upgrade from 23.7 to 23.11, then inspect the existing DAG, its wrapped Spark job, and the config map containing the job definition. Document the procedure needed to update the Spark job when that config map changes, including how to verify that the DAG uses the new definition.
Written by the indexing model from the issue text.
Description
When upgrading the airflow demo from 23.7 to 23.11 the spark job wrapped by an existing DAG is not updated if the config map containing its definition is changed. This should be investigated and documented as to how to do this.
- Dominant language
- Jupyter Notebook
- Stars
- 3
- Forks
- 13
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 2
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from stackabletech/demos
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
stackabletech/demos#402 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 45/100
stackabletech/demos#349 ·
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
stackabletech/demos#293 ·
-
release-retro/25.3.0
Difficulty 4/5 3-5 days Newbie friendliness 35/100
stackabletech/demos#208 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 45/100
stackabletech/demos#204 ·
All issues in stackabletech/demos
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
cisagov/cyhy-reports#149 · 3 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
open-compass/VLMEvalKit#1698 ·