Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[Feature] Integration test case with Data Science Pipeline, CodeFlare and KubeRay

Open
  • Is something wrong?
#425 4 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
kubernetes, python, scikit-learn
Domain
distributed-systems, machine-learning, testing-qa

Research direction

Start with the scikit-learn document-clustering example and map the six listed pipeline stages to the DSP YAML, test image, and CodeFlare image described in the issue. Done means a public-data, non-proprietary run completes in under 20 minutes within the stated Ray resource limits and reports both actor evaluations before shutdown.

Written by the indexing model from the issue text.

Description

Name of Feature or Improvement

Create an integration test case to validate DSP, CodeFlare and KubeRay implementation.

Describe the Solution You Would Like to See
Test environment assumptions:
  1. Data Science Pipeline v1.
  2. Ray cluster shall consist of no more than 2 worker pods, with 2 CPU cores and less than 6 GB available for each pod.
  3. An integration test execution time shall be less than 20 mins in total.
  4. S3 storage may be available, if needed.
  5. Free of proprietary intellectual property.
  6. Public data only.
Proposed test case: Clustering text documents using k-means on scikit-learn education page.

https://scikit-learn.org/stable/auto_examples/text/plot_document_clustering.html

Data Science Pipeline stages:
  1. Downloading test data (https://scikit-learn.org/stable/auto_examples/text/plot_document_clustering.html#loading-text-data)
  2. Launch Ray cluster with two worker pods.
  3. Ray driver launches two Ray actors, deployed to a pod each. The first actor runs TfidfVectorizer, followed by Kmeans clustering and evaluation. The second actor runs HashingVectorizer, followed by Kmeans clustering and evaluation.
  4. Ray driver collects evaluation results from the two actors. Then it reports the summaries.
  5. Ray cluster is stopped and shutdown.
  6. Pipeline run is completed.
Expected test assets:
  1. DSP pipeline yaml to deploy and kick off test runs.
  2. Test image with Ray and document clustering code.
  3. CodeFlare image to deploy the test image.
  4. Preconfigured credentials and configmaps in the test environment.
Dominant language
Python
Stars
35
Forks
66
Avg merge
1d 17h
Merged PRs (30d)
19

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from project-codeflare/codeflare-sdk

All issues in project-codeflare/codeflare-sdk

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.