Running multiple tasks asynchronously

A issue menciona o scheduler e a execução de cron, mas não informa arquivos do repositório nem testes. Comece localizando o ponto de entrada do cron do scheduler e rastreando como várias tarefas agendadas são despachadas; compare esse fluxo com os exemplos fornecidos de processos PHP e bloqueios de arquivo. A conclusão deve definir e verificar uma execução concorrente segura para tarefas de longa duração sem quebrar a ordem do agendamento nem a proteção de dados.

I take a look at what been going on here and i wonder if one thing is possible with this.

I work on a project with CI4 which relays heavily on cron to fetch the data from APIs. We're fetching data form about 30 endpoints every minute. And each of the request takes a lot of time to complete the processing. So in other to achieve those I need to spawn a lot of processes to work at the same time. So I have a single command which executed the multiple instances of other command by using something like this

/**
 * Spark Exec
 *
 * @param string   $sparkCommand Command
 * @param int|null $timeout      Timeout
 */
function spark_exec(string $sparkCommand, ?int $timeout = null): void
{
    $command = '';

    if ($timeout) {
        $command .= 'timeout ' . $timeout . ' ';
    }

    $command .= '"' . PHP_BINARY . '" -c "' . (get_cfg_var('cfg_file_path') ?: PHP_CONFIG_FILE_PATH) . '" ' . ROOTPATH . 'spark ' . $sparkCommand . ' > /dev/null &';

    exec($command);
}

And the problem is that I cannot have more that one process working working with the same endpoint and writing the same data db so I have mechanism to put some locks in place so I use files to track the locking

/**
     * Crates the Lockfile with the current class name
     *
     * @param string|null $identifier String to append to the filename
     *
     * @return bool|resource Locked file if successful otherwise false
     */
    public static function lock(?string $identifier = null)
    {
        //Add leading dash to identifier if it is set
        if (isset($identifier)) {
            $identifier = '-' . $identifier;
        }

        //Get the locks file directory
        $lockDir = WRITEPATH . 'cron/locks/';

        //Create the locks file directory if it does not exist
        if (! file_exists($lockDir)) {
            mkdir($lockDir, 0755, true);
        }

        //Get the filename
        $filename = $lockDir . $identifier . '.lock';

        //Open the file for writing to lock it
        $lock = fopen($filename, 'wb');

        if ($lock === false) {
            // Unable to open file, check that you have permissions to open/write
            CLI::error('Unable to write the lock file');

            exit;
        }

        if (flock($lock, LOCK_EX | LOCK_NB) === false) {
            // Lock is already in use by another process
            CLI::error('Another instance is already running');

            exit;
        }

        //Return the locked file
        return $lock;
    }

    /**
     * Closes the file removing the lock
     *
     * @param resource $lock Lock
     */
    public static function unlock(&$lock): void
    {
        //Check if lock exist
        if ($lock && get_resource_type($lock) !== 'Unknown') {
            //Get the filename
            $metaData = stream_get_meta_data($lock);
            $filename = $metaData['uri'];

            //Close the file to remove the lock
            fclose($lock);

            //Remove the lock file
            unlink($filename);
        }
    }

    /**
     * Check if lock files with set prefix exist
     *
     * @param string $prefix filename prefix
     *
     * @return bool Ture if there are lock files otherwise false
     */
    public static function isWorking(string $prefix): bool
    {
        $result = false;

        //We read the locks directory if there are  lock files it means we're still working
        if ($openDir = opendir(WRITEPATH . 'cron/locks/')) {
            while (($file = readdir($openDir)) !== false) {
                if (substr($file, 0, strlen($prefix)) === $prefix) {
                    $result = true;
                    break;
                }
            }
        }

        return $result;
    }

I wonder if there are plans to be able to achieve something similar with the scheduler here. From looking at the code I believe all the scheduled actions run on a single thread. So for example if I have 2 tasks scheduled to run first runs every 5 minutes second task runs every minute. And if first task takes more that 1 minute to be completed. On the next cron run second task will be started but once the first process finishes with task 1 it will continue with the second task in the first process and the order of execution is completely broken.

And there are a lot of real case situations where long running tasks are needed. One example is generating sitemap from the database for the huge site