Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[Documentation] How does `SingleNodeExecutor` touch the file system?

Abierto
  • ¿Algo no está bien?
#703 5 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
25/100
Tipo de issue
Documentación
Claridad
Necesita aclaración
Estado de actividad
Estancado
Stack tecnológico
python
Área
documentation, hpc

Línea de trabajo

Comienza con SingleNodeExecutor, BaseExecutor, DependencyTaskScheduler, OneProcessTaskScheduler y las herramientas task_scheduler.file mencionadas en el issue. Rastrea cache_directory desde resource_dict a través del envío y la ejecución de tareas, y documenta después el punto en el que se selecciona el almacenamiento en caché respaldado por archivos y en qué se diferencia de create_file_executor. Se considera terminado cuando la ruta de ejecución y la conexión que falta están claramente explicadas.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

documentation

I'm trying to understand where SingleNodeExecutor with a cache_directory gets its instructions to actually write it's result to file.

For instance, consider the following:

from executorlib import SingleNodeExecutor

def foo(x):
    return x + 1

with SingleNodeExecutor() as exe:
    f = exe.submit(foo, 1, resource_dict={"cache_key": "my_key", "cache_directory": "my_dir"})
    print("Result", f.result())

My understanding of the path of events is

Initialization:

  • SingleNodeExecutor.__init__ triggers BaseExecutor.__init__ with a DependencyTaskScheduler as its underlying _task_scheduler
  • That DependencyTaskScheduler is initialized with a OneProcessTaskScheduler as its executor arg, which gets stored in _process_kwargs["executor"]

Submission:

  • SingleNodeExecutor.submit is inherited directly fromBaseExecutor.submit
  • BaseExecutor.submit passes everything (including the resource_dict as a single kwarg) to the _task_scheduler.submit
  • Since DependencyTaskScheduler._generate_dependency_graph has fed through to False with all the default values, this generates the future by a plain super() call to TaskSchedulerBase.submit
  • TaskSchedulerBase.submit sends our information (function, args, kwargs, resource dict, empty future) to self._future_queue.put

Here I get out of my depth, but it seems to me like putting stuff on the future queues is activating the associated Thread, which all the TaskSchedulerBase children initialize in _set_process using a Thread taking some function and the _process_kwargs (which includes the _future_queue!). On that assumption, that means that the self._future_queue.put call we got to from TaskSchedulerBase.submit would route back to the Thread set in the DependencyTaskScheduler._set_process invocation -- i.e. _execute_tasks_with_dependencies

Continuing submission:

  • _execute_tasks_with_dependencies indeed takes an executor, which is DependencyTaskScheduler._process_kwargs["executor"] i.e. our OneProcessTaskScheduler; there's lots going on, but...
    • I don't see any reference to the cache, so I don't think the file system interaction is happening here
    • It looks like we ultimately do a executor_queue.put onto the underlying OneProcessTaskScheduler._future_queue
    • I.e. we move to _execute_task_in_separate_process
  • _execute_task_in_separate_process is in turn re-directing to _wrap_execute_task_in_separate_process and both of these are now taking a spawner: type[BaseSpawner] argument
  • But I'm at the end of the line, I don't see anything other than the default MpiExecSpawner being leveraged, and I never find any references to the cache_directory or any of the task_scheduler.file module tools

With the SlurmClusterExecutor we sometimes route through create_file_executor, in which case the file system connection is obvious, but in the other case we're still going through DependencyTaskScheduler -- this time with an SrunSpawner instead of a MpiExecSpawner. In this later case I also don't see the connection to file system tools, so I feel like I must be missing something at the diverging point: DependencyTaskScheduler.

What am I missing here? When does the SingleNodeExecutor figure out it needs to leverage the "cache_directory" field in the resource_dict?

Lenguaje dominante
Python
Estrellas
77
Forks
9
Merge medio
7 h 9 min
PR fusionados (30 d)
13

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de pyiron/executorlib

Todos los issues de pyiron/executorlib

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.