[Documentation] Case Study: Use executorlib for task handling in a larger application

Open
#916 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
42/100
Issue type
Documentation
Clarity
Mostly clear
Activity status
Stale
Tech stack
python
Domain
distributed-systems, documentation, hpc

Research direction

Start with the issue's referenced APIs, get_future_from_file and terminate_task_in_cache, and review the implementations in pull requests 915 and 914. Document a larger-application case study covering resource_dict={"cache_key": "foo"}, status checks, and task termination, with examples that show these workflows operating together.

Written by the indexing model from the issue text.

Description

documentation

excutorlib can be integrated in existing frameworks and in combination with the flux framework is can provide an efficient task scaling solution, using the available resources while not oversubscribing them. Given the background of executorlib these tasks can be distributed on a high performance computing (HPC) cluster:

  • Explain the use of external cache key in the resource dictionary: resource_dict={"cache_key": "foo"}. This enables external applications to define the key to store a given calculation in.
  • Implement a check status function based on from executorlib import get_future_from_file as implemented in https://github.com/pyiron/executorlib/pull/915 . The external application uses its cache key to check the status of a given job object.
  • Terminate a task which is no longer required using the from executorlib import terminate_task_in_cache function as implemented in https://github.com/pyiron/executorlib/pull/914 . The external application again uses its cache key to remove tasks from the backlog.
Dominant language
Python
Stars
77
Forks
7
Avg merge
10h 32m
Merged PRs (30d)
12

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from pyiron/executorlib

All issues in pyiron/executorlib

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.