Run LibreSign Behat shards in parallel using isolated Nextcloud environments
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
Research direction
Start with the existing version-matrix workflow and the isolated nextcloud-docker-development worker lifecycle, then trace how Behat 4 shards and database entries are currently invoked. Benchmark 1, 2, and 4 workers with at least three successful runs per count; done means isolated concurrent workers, complete matrix coverage, diagnosable artifacts, equivalent results, and a documented reliable default.
Written by the indexing model from the issue text.
Description
Part of #8969.
Context
The modernization work provides Behat 4, isolated tooling, concurrency-safe scenarios, deterministic sharding and isolated nextcloud-docker-development workers.
The suite can then be parallelized without sharing mutable Nextcloud state.
Goal
Execute multiple Behat shards concurrently and reduce integration-test feedback time without introducing cross-worker state leakage or unacceptable flakiness.
Execution model
Evaluate concurrent nextcloud-docker-development environments on the same CI runner before assuming that a separate GitHub Actions VM is required for every shard.
Each worker must receive a unique:
worker identifier
workspace/environment path
Compose project name
SHARD_INDEX
SHARD_TOTAL
Do not execute multiple shards against one shared Nextcloud installation.
CI security boundary
Parallel workers belonging to one test run may share an ephemeral runner and immutable caches/images.
Do not share mutable application state, credentials or worker directories between unrelated test runs.
For untrusted pull-request execution:
- do not expose repository write credentials or long-lived secrets to test containers;
- do not persist mutable Docker/Nextcloud state for reuse by a different PR;
- cleanup must remove worker-specific containers/networks/state even after failures.
If self-hosted runners are used, the implementation must preserve an isolation boundary appropriate for untrusted PR code rather than assuming that Compose project names alone provide a security boundary.
Benchmark
Record the existing serial baseline before enabling the final configuration.
Compare at least where runner resources permit:
1 worker
2 workers
4 workers
Measure:
- total wall-clock duration;
- environment bootstrap time;
- Behat execution time;
- total runner usage;
- CPU/memory pressure where practical;
- failures/retries across repeated runs.
Select the default worker count from evidence rather than assuming that more workers are always faster.
Diagnostics
Every worker must produce identifiable output.
Where applicable, use worker/shard-specific artifacts such as:
report-shard-1.xml
report-shard-2.xml
nextcloud-shard-1.log
Enable useful Behat 4 failure reporting such as inline failures where it improves CI diagnosis.
A failure must clearly identify:
- shard;
- failing feature/scenario;
- associated environment logs.
Reliability validation
Repeated parallel runs must include isolation-sensitive tests.
The implementation must not hide intermittent failures with unconditional retries.
If parallel execution reveals an existing race or scenario dependency, fix or isolate the cause rather than retrying until green.
Database matrix
Use the isolated nextcloud-docker-development worker lifecycle for the complete LibreSign Behat database matrix:
- SQLite;
- MySQL;
- MariaDB 10.6;
- MariaDB 10.11;
- PostgreSQL.
Do not keep SQLite or MariaDB on a separate legacy execution architecture after the parallel migration is complete.
The matrix must continue to combine the database backend with the repository's supported PHP and Nextcloud versions according to the version-matrix workflow already used by LibreSign.
The implementation may choose different shard counts for different matrix entries when benchmark evidence justifies it, but it must not drop a database/PHP/Nextcloud compatibility entry merely to reduce CI time.
Benchmark protocol
Benchmark on the same runner class intended for the resulting CI workflow.
For each candidate worker count (1, 2 and 4 where resources permit):
- perform at least three successful complete runs;
- record bootstrap time and Behat execution time separately;
- record total wall-clock time;
- record the slowest shard duration;
- record peak/observed resource pressure when practical;
- record any failures or retries.
Use the median successful wall-clock result for the primary comparison, while retaining the individual measurements so outliers remain visible.
If four workers cannot complete reliably because of runner CPU/memory limits, record that result rather than hiding it and choose the highest reliable count.
The selected default worker count must be justified in the PR/implementation notes with these measurements.
Done when
- Multiple isolated workers execute concurrently.
- Every intended test is executed exactly once across the shards.
- Serial and parallel results are behaviorally equivalent.
- Parallel wall-clock performance is measured against baseline.
- The selected default worker count is documented with benchmark evidence.
- Worker-specific logs/reports make failures diagnosable.
- Repeated parallel runs show no cross-worker state leakage.
- Untrusted PR runs do not reuse mutable state or receive unnecessary long-lived/write credentials.
- SQLite, MySQL, MariaDB 10.6/10.11 and PostgreSQL execute through the isolated worker lifecycle.
- Existing PHP/Nextcloud/database compatibility coverage is preserved.
- The selected concurrency level is supported by at least three measured successful runs on the target runner class.
- Dominant language
- PHP
- Stars
- 844
- Forks
- 176
- Avg merge
- 14h 3m
- Merged PRs (30d)
- 344
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from LibreSign/libresign
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
OpenSSL engine: leaf certificates get O=Internet Widgits Pty Ltd, ST=Some-State from system openssl.cnfPossibly taken @vitormattos claimed this 5 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
LibreSign/libresign#9366 · 1 assignee ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 83/100
LibreSign/libresign#9007 · 2 comments · 1 reaction ·
Maintainers usually reply within 1 day
-
good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
LibreSign/libresign#8284 · 5 comments ·
Maintainers usually reply within 1 day
-
frontend javascript technical debt
Difficulty 5/5 Over a week Newbie friendliness 8/100
Maintainers usually reply within 1 day
All issues in LibreSign/libresign
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 79/100
Automattic/blocks-engine#6318 ·
Maintainers usually reply within 1 day
-
[remove-resource] Delete Open Course Hub, Open Multicourse Hub, and Learn2 with GitSync skeletonsOpen
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
-
Difficulty 1/5 Under an hour Newbie friendliness 82/100
-
35-feedback
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day