Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Config 2.0 unified storage description

Abierto
  • ¿Algo no está bien?
#30 1 comentario 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
30/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Estancado
Stack tecnológico
yaml
Área
infrastructure

Línea de trabajo

Comienza localizando el esquema de configuración y los puntos de entrada del runner que gestionan las secciones platform y tasks; el issue no menciona archivos ni pruebas. Lee cómo se resuelven actualmente las rutas de E/S de las tareas y la variable $WORKSPACE. La tarea estará terminada cuando las definiciones de platform.storage puedan describir especificaciones específicas del backend y las rutas de las tareas puedan referenciarse de forma coherente.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

This proposes a unified storage description for config 2.0.

Today, MLCube relies on a simple "file path" approach to describe the inputs and outputs of their tasks. However, for many platforms, such like Kubernetes, it is not possible to use a single file path, because they either have complex storage backend, or use their own layer of storage abstractions, which do not use "paths" to refer to the corresponding locations in the data storage. This proposal aims to address this problem by providing a unified way of describing storage that can cover both local file systems and more complex storage solutions.

A storage backend can be described in the platform section of the config, which is supplied by the user at run-time. The storage description consists of 2 main parts: a name that will be used as a reference in the tasks' I/O paths, and a platform-specific spec that provides the details of the storage backend in the target platform, so that the runner can use it to find the right location of data.
We do not change the "path"-like descriptions of task inputs/outputs in order to keep that simple, however, we do introduce a "variable"-like component as a part of the path, so that we can use this "variable" as a reference to the corresponding storage backend and use the rest of the path as a relative path to the given storage.
A most straight-forward example of such "variable" is "$WORKSPACE" which is currently being used to refer to a specific dir in local file system. With the new proposal, the "$WORKSPACE", or any "$CUSTOM_NAME" defined by user, can refer to an arbitrary storage backend as specified in the platform section.

Since the detailed spec of the storage is specified in the platform part, it can be decoupled from the shared MLCube config and only appear in the user's config. This also means that how the spec of a given storage backend is writtern should be agreed between a user and a runner, and not relevant to the MLCube publisher.
While we do not have to provide a standard for that specs, we may provide some "guidelines/examples" for popular platforms so that there can be a convention for runner implementors.

The following is an example of how the storage backend can be defined, notice the specs in the platform section and how they are used in the tasks section. Notice also that if we give the storage a name of "WORKSPACE" then we may redirect our default workspace to the specified storage backend, without change the values in the task I/Os.

name: example-mlcube
platform:
  storage:
  - name: K8S_DATA
    spec:
      kubernetes:
        pvc_name: my-pvc
  - name: NFS_DATA
    spec:
      nfs:
        host: 127.0.0.1
        port: 2049
        path: some/nfs/path
container:
  image: mlcommons/mnist:0.0.1
  build_context: "mnist"
  build_file: "Dockerfile"
tasks:
  download:
    io:
    - {name: data_dir, type: directory, io: output, default: $NFS_DATA/data}
    - {name: log_dir, type: directory, io: output, default: $NFS_DATA/logs}
  train:
    io:
    - {name: data_dir, type: directory, io: input, default: $K8S_DATA/data}
    - {name: parameters_file, type: file, io: input, default: $K8S_DATA/parameters/default.parameters.yaml}
    - {name: log_dir, type: directory, io: output, default: $K8S_DATA/logs}
    - {name: model_dir, type: directory, io: output, default: $K8S_DATA/model}
Lenguaje dominante
Python
Estrellas
33
Forks
32
Sin PR fusionados en 30 d

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de mlcommons/mlcube_examples

Todos los issues de mlcommons/mlcube_examples

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.