Run LibreSign Behat shards in parallel using isolated Nextcloud environments

Open
#8,975 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
docker, github-actions, mariadb, mysql, php, postgresql, sqlite

Research direction

Start with the existing version-matrix workflow and the isolated nextcloud-docker-development worker lifecycle, then trace how Behat 4 shards and database entries are currently invoked. Benchmark 1, 2, and 4 workers with at least three successful runs per count; done means isolated concurrent workers, complete matrix coverage, diagnosable artifacts, equivalent results, and a documented reliable default.

Written by the indexing model from the issue text.

Description

backend enhancement php

Part of #8969.

Context

The modernization work provides Behat 4, isolated tooling, concurrency-safe scenarios, deterministic sharding and isolated nextcloud-docker-development workers.

The suite can then be parallelized without sharing mutable Nextcloud state.

Goal

Execute multiple Behat shards concurrently and reduce integration-test feedback time without introducing cross-worker state leakage or unacceptable flakiness.

Execution model

Evaluate concurrent nextcloud-docker-development environments on the same CI runner before assuming that a separate GitHub Actions VM is required for every shard.

Each worker must receive a unique:

worker identifier
workspace/environment path
Compose project name
SHARD_INDEX
SHARD_TOTAL

Do not execute multiple shards against one shared Nextcloud installation.

CI security boundary

Parallel workers belonging to one test run may share an ephemeral runner and immutable caches/images.

Do not share mutable application state, credentials or worker directories between unrelated test runs.

For untrusted pull-request execution:

  • do not expose repository write credentials or long-lived secrets to test containers;
  • do not persist mutable Docker/Nextcloud state for reuse by a different PR;
  • cleanup must remove worker-specific containers/networks/state even after failures.

If self-hosted runners are used, the implementation must preserve an isolation boundary appropriate for untrusted PR code rather than assuming that Compose project names alone provide a security boundary.

Benchmark

Record the existing serial baseline before enabling the final configuration.

Compare at least where runner resources permit:

1 worker
2 workers
4 workers

Measure:

  • total wall-clock duration;
  • environment bootstrap time;
  • Behat execution time;
  • total runner usage;
  • CPU/memory pressure where practical;
  • failures/retries across repeated runs.

Select the default worker count from evidence rather than assuming that more workers are always faster.

Diagnostics

Every worker must produce identifiable output.

Where applicable, use worker/shard-specific artifacts such as:

report-shard-1.xml
report-shard-2.xml
nextcloud-shard-1.log

Enable useful Behat 4 failure reporting such as inline failures where it improves CI diagnosis.

A failure must clearly identify:

  • shard;
  • failing feature/scenario;
  • associated environment logs.

Reliability validation

Repeated parallel runs must include isolation-sensitive tests.

The implementation must not hide intermittent failures with unconditional retries.

If parallel execution reveals an existing race or scenario dependency, fix or isolate the cause rather than retrying until green.

Database matrix

Use the isolated nextcloud-docker-development worker lifecycle for the complete LibreSign Behat database matrix:

  • SQLite;
  • MySQL;
  • MariaDB 10.6;
  • MariaDB 10.11;
  • PostgreSQL.

Do not keep SQLite or MariaDB on a separate legacy execution architecture after the parallel migration is complete.

The matrix must continue to combine the database backend with the repository's supported PHP and Nextcloud versions according to the version-matrix workflow already used by LibreSign.

The implementation may choose different shard counts for different matrix entries when benchmark evidence justifies it, but it must not drop a database/PHP/Nextcloud compatibility entry merely to reduce CI time.

Benchmark protocol

Benchmark on the same runner class intended for the resulting CI workflow.

For each candidate worker count (1, 2 and 4 where resources permit):

  • perform at least three successful complete runs;
  • record bootstrap time and Behat execution time separately;
  • record total wall-clock time;
  • record the slowest shard duration;
  • record peak/observed resource pressure when practical;
  • record any failures or retries.

Use the median successful wall-clock result for the primary comparison, while retaining the individual measurements so outliers remain visible.

If four workers cannot complete reliably because of runner CPU/memory limits, record that result rather than hiding it and choose the highest reliable count.

The selected default worker count must be justified in the PR/implementation notes with these measurements.

Done when

  • Multiple isolated workers execute concurrently.
  • Every intended test is executed exactly once across the shards.
  • Serial and parallel results are behaviorally equivalent.
  • Parallel wall-clock performance is measured against baseline.
  • The selected default worker count is documented with benchmark evidence.
  • Worker-specific logs/reports make failures diagnosable.
  • Repeated parallel runs show no cross-worker state leakage.
  • Untrusted PR runs do not reuse mutable state or receive unnecessary long-lived/write credentials.
  • SQLite, MySQL, MariaDB 10.6/10.11 and PostgreSQL execute through the isolated worker lifecycle.
  • Existing PHP/Nextcloud/database compatibility coverage is preserved.
  • The selected concurrency level is supported by at least three measured successful runs on the target runner class.
Dominant language
PHP
Stars
844
Forks
176
Avg merge
14h 3m
Merged PRs (30d)
344

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from LibreSign/libresign

All issues in LibreSign/libresign

Similar issues

More PHP issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.