Modernize LibreSign Behat infrastructure for Behat 4 and parallel execution
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 20/100
- Issue type
- Refactor
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- docker, php
- Domain
- ci-cd, devops, documentation, testing-qa, tooling
Research direction
Treat this as an epic and begin by splitting the workstreams into focused child issues rather than implementing it as one change. Start with the shared package list, LibreSign's vendor-bin/behat toolchain, and nextcloud-docker-development; use the checklist as acceptance criteria, including Behat 4 execution, isolated deterministic parallel workers, preserved compatibility coverage, measured performance, and canonical documentation.
Written by the indexing model from the issue text.
Description
Context
LibreSign's Behat integration suite is a critical part of the project's regression coverage, but the current infrastructure still reflects assumptions from older Behat versions and executes each environment serially.
The current stack includes:
- Behat 3;
- YAML Behat configuration;
- Behat dependencies installed under
tests/integration; - shared Behat packages tied to Behat 3-era APIs;
- reusable contexts coupled to older Gherkin argument objects;
- PHPUnit used as a runtime assertion dependency in shared Behat code;
- LibreSign feature/context definitions that have accumulated without a single modern Behat 4 convention;
- a suite that is not explicitly designed for concurrent execution;
- complete Behat runs that can take more than ten minutes.
Behat 4 provides the boundary for a coordinated modernization rather than a dependency-only upgrade.
The supported baseline for this work is:
- PHP >= 8.2;
- Behat ^4.0;
- supported Nextcloud versions only.
The LibreSign migration applies to main only. It must not be backported to existing stable branches.
Goal
Modernize the complete LibreSign Behat testing stack around Behat 4 and make the suite safe, maintainable and measurably faster through isolated parallel execution.
Workstreams
Shared Behat packages
Modernize and publish Behat 4-compatible breaking releases of the packages LibreSign depends on:
libresign/behat-builtin-extension;libresign/nextcloud-behat;libresign/mailpit-behat-extension;libresign/behat-tsa-extension.
The shared package work also includes:
- removing PHPUnit as a runtime assertion implementation from
nextcloud-behat; - adopting
DataTableandDocString; - auditing reusable generic steps for Behat 4 signatures and clearer contracts;
- relying on Behat public APIs rather than unsupported internals.
LibreSign Behat toolchain
LibreSign should:
- install Behat through
vendor-bin/behat; - use PHP configuration instead of Behat YAML;
- consume released Behat 4-compatible shared packages;
- keep the Behat Composer environment resolvable for PHP 8.2;
- prefer Behat's built-in JUnit support when it satisfies the existing CI reporting contract.
Test design
Define and apply:
- a small semantic tag taxonomy;
- ownership boundaries between generic Nextcloud steps and LibreSign domain steps;
- modern Behat 4 context/feature conventions;
- independent scenario setup and lifecycle rules.
Tags describe test semantics. They must not be manually maintained shard identifiers.
Parallel execution
Use LibreCodeCoop/nextcloud-docker-development as the isolated Nextcloud environment for the complete parallel Behat database matrix. The worker lifecycle must support SQLite, MySQL, MariaDB 10.6/10.11 and PostgreSQL so the migration does not leave part of the compatibility matrix on a separate legacy execution architecture.
Each worker must have isolated mutable state, including where applicable:
- Nextcloud installation/configuration;
- database;
- data directory;
- users/groups;
- app configuration;
- generated files;
- mail state;
- mock/local services;
- temporary files;
- reports/logs.
Sharding must be deterministic and independent from semantic tagging.
Architecture
Shared package modernization
├── behat-builtin-extension
├── nextcloud-behat
│ ├── standalone assertions
│ ├── DataTable / DocString
│ └── reusable step modernization
├── mailpit-behat-extension
└── behat-tsa-extension
↓
LibreSign vendor-bin/behat + Behat 4
↓
LibreSign context/feature modernization
Test taxonomy ────────────────┐
Scenario concurrency safety ─┤
Deterministic sharding ──────┼─> isolated parallel Behat execution
Docker worker lifecycle ─────┘
↓
final Behat 3 cleanup + canonical documentation
Test-design principles
Scenarios must:
- be executable independently where intended for normal parallel execution;
- not rely on execution order;
- create or declare the state they require;
- not depend on state left by another scenario;
- avoid unnecessary implementation-detail steps;
- reuse generic Nextcloud steps when behavior is generic;
- keep LibreSign business-language steps in the LibreSign layer.
Parallel-execution principles
Parallelism must be introduced only after isolation prerequisites are proven.
Parallel workers belonging to one test run may share immutable caches/images, but must not share mutable application state.
Untrusted pull-request runs must not receive unnecessary repository write credentials or long-lived secrets, and mutable state must not be reused across unrelated test runs.
Performance
Record a serial baseline before selecting the final concurrency level.
Compare at least, where runner resources permit:
- one worker;
- two workers;
- four workers.
Measure:
- wall-clock duration;
- environment bootstrap duration;
- Behat execution duration;
- CPU/memory pressure where practical;
- total runner usage;
- failures/retries;
- evidence of flakiness or state leakage.
Do not select the final worker count without measurement.
Compatibility
- Target PHP >= 8.2.
- Target Behat ^4.0.
- New breaking releases of the shared testing packages do not retain Behat 3 compatibility.
- LibreSign changes target
main. - Do not backport this migration to stable branches.
Done when
- All required shared Behat packages have Composer-installable Behat 4-compatible breaking releases.
- LibreSign
mainexecutes its integration suite using Behat 4. - Behat tooling is isolated under
vendor-bin/behatand remains resolvable for PHP 8.2. - YAML Behat configuration is no longer required.
- Shared contexts use supported/public Behat APIs.
- PHPUnit is not used merely as the runtime assertion library of shared Behat contexts.
- Reusable step APIs use modern Behat argument types where applicable.
- Generic Nextcloud reusable steps and LibreSign domain steps have documented ownership boundaries and have been modernized accordingly.
- The test taxonomy is documented and applied.
- Scenarios intended for normal parallel execution are independent of execution order and shared mutable state.
-
nextcloud-docker-developmentsupports the complete Behat database matrix: SQLite, MySQL, MariaDB 10.6/10.11 and PostgreSQL. - Multiple isolated Nextcloud test environments can run concurrently.
- Deterministic sharding executes every intended test exactly once across workers.
- Parallel CI is measurably faster than the serial baseline.
- Parallel execution does not introduce measurable cross-worker state leakage or systematic flakiness.
- Failure output identifies the failing worker/shard and remains diagnosable.
- Existing database/Nextcloud/PHP compatibility coverage is preserved.
- Legacy Behat 3 infrastructure is removed after the new path is proven.
- The supported developer workflow is documented in LibreSign's canonical documentation.
Target
Branch: main
Milestone: Next Major (36)
Pull requests should implement focused child issues, not this epic.
- Dominant language
- PHP
- Stars
- 844
- Forks
- 176
- Avg merge
- 14h 3m
- Merged PRs (30d)
- 344
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from LibreSign/libresign
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
OpenSSL engine: leaf certificates get O=Internet Widgits Pty Ltd, ST=Some-State from system openssl.cnfPossibly taken @vitormattos claimed this 5 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
LibreSign/libresign#9366 · 1 assignee ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 83/100
LibreSign/libresign#9007 · 2 comments · 1 reaction ·
Maintainers usually reply within 1 day
-
good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
LibreSign/libresign#8284 · 5 comments ·
Maintainers usually reply within 1 day
-
frontend javascript technical debt
Difficulty 5/5 Over a week Newbie friendliness 8/100
Maintainers usually reply within 1 day
All issues in LibreSign/libresign
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 79/100
Automattic/blocks-engine#6318 ·
Maintainers usually reply within 1 day
-
[remove-resource] Delete Open Course Hub, Open Multicourse Hub, and Learn2 with GitSync skeletonsOpen
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
-
Difficulty 1/5 Under an hour Newbie friendliness 82/100
-
35-feedback
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day