intelligent-machine-learning/dlrover

Enhance/Replace k8s python client.

Open

#1,291 opened on Oct 12, 2024

View on GitHub
 (2 comments) (0 reactions) (1 assignee)Python (212 forks)github user discovery
Hacktoberfestwip

Repository metrics

Stars
 (1,662 stars)
PR merge metrics
 (PR metrics pending)

Description

Background

Currently, DLRover uses the official Kubernetes Python client to interact with the Kubernetes API Server. This part of implementations are quite important because it involves managing the lifecycle of training workers. However, the Python client has inherent limitations and lags behind the Go (and Java) clients (e.g., lacks an informer implementation), leading to occasional unexpected usage issues in certain scenarios. Therefore, we intend to:

[Option 1] Replace the current Python client with the Go client(need to use CFFI). [Option 2] Enhance k8s client implements in python.

Requirement

The enhancement/replacement should ensure compatibility with all existing Kubernetes-related calls while also adapting the usage in dlrover/python/scheduler/kubernetes.py.

  1. Ensure compatibility with all related features (no regression).
  2. Reimplement the watch mechanism using the 'informer'. (dlrover/python/master/watcher/k8s_watcher.py)

For replacement scheme requires evaluation of:

  1. Limitations on different system platforms.
  2. Issues with cross-language object transfer.
  3. Potential performance overhead.
  4. Additional costs for building and deployment.

For enhancement scheme requires evaluation of:

  1. Feasibility and complexity of implementation.

Contributor guide

Research direction
Evaluate Option 1 (replace with Go client via CFFI) and Option 2 (enhance Python client). For each, ensure compatibility with all existing Kubernetes calls in dlrover/python/scheduler/kubernetes.py. Reimplement the watch mechanism using the informer pattern in dlrover/python/master/watcher/k8s watcher.py. For replacement, assess platform limitations, cross language object transfer, performance overhead, and build/deployment costs. For enhancement, assess feasibility and complexity.
Tech stack
pythongo
Domain
backendinfrastructuredevops
Issue type
Refactor
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Mostly clear
Prerequisites
KubernetesPythonGo
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20