Modernize LibreSign Behat infrastructure for Behat 4 and parallel execution

Open
#8,969 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
20/100
Issue type
Refactor
Clarity
Mostly clear
Activity status
Active
Tech stack
docker, php

Research direction

Treat this as an epic and begin by splitting the workstreams into focused child issues rather than implementing it as one change. Start with the shared package list, LibreSign's vendor-bin/behat toolchain, and nextcloud-docker-development; use the checklist as acceptance criteria, including Behat 4 execution, isolated deterministic parallel workers, preserved compatibility coverage, measured performance, and canonical documentation.

Written by the indexing model from the issue text.

Description

enhancement

Context

LibreSign's Behat integration suite is a critical part of the project's regression coverage, but the current infrastructure still reflects assumptions from older Behat versions and executes each environment serially.

The current stack includes:

  • Behat 3;
  • YAML Behat configuration;
  • Behat dependencies installed under tests/integration;
  • shared Behat packages tied to Behat 3-era APIs;
  • reusable contexts coupled to older Gherkin argument objects;
  • PHPUnit used as a runtime assertion dependency in shared Behat code;
  • LibreSign feature/context definitions that have accumulated without a single modern Behat 4 convention;
  • a suite that is not explicitly designed for concurrent execution;
  • complete Behat runs that can take more than ten minutes.

Behat 4 provides the boundary for a coordinated modernization rather than a dependency-only upgrade.

The supported baseline for this work is:

  • PHP >= 8.2;
  • Behat ^4.0;
  • supported Nextcloud versions only.

The LibreSign migration applies to main only. It must not be backported to existing stable branches.

Goal

Modernize the complete LibreSign Behat testing stack around Behat 4 and make the suite safe, maintainable and measurably faster through isolated parallel execution.

Workstreams

Shared Behat packages

Modernize and publish Behat 4-compatible breaking releases of the packages LibreSign depends on:

  • libresign/behat-builtin-extension;
  • libresign/nextcloud-behat;
  • libresign/mailpit-behat-extension;
  • libresign/behat-tsa-extension.

The shared package work also includes:

  • removing PHPUnit as a runtime assertion implementation from nextcloud-behat;
  • adopting DataTable and DocString;
  • auditing reusable generic steps for Behat 4 signatures and clearer contracts;
  • relying on Behat public APIs rather than unsupported internals.
LibreSign Behat toolchain

LibreSign should:

  • install Behat through vendor-bin/behat;
  • use PHP configuration instead of Behat YAML;
  • consume released Behat 4-compatible shared packages;
  • keep the Behat Composer environment resolvable for PHP 8.2;
  • prefer Behat's built-in JUnit support when it satisfies the existing CI reporting contract.
Test design

Define and apply:

  • a small semantic tag taxonomy;
  • ownership boundaries between generic Nextcloud steps and LibreSign domain steps;
  • modern Behat 4 context/feature conventions;
  • independent scenario setup and lifecycle rules.

Tags describe test semantics. They must not be manually maintained shard identifiers.

Parallel execution

Use LibreCodeCoop/nextcloud-docker-development as the isolated Nextcloud environment for the complete parallel Behat database matrix. The worker lifecycle must support SQLite, MySQL, MariaDB 10.6/10.11 and PostgreSQL so the migration does not leave part of the compatibility matrix on a separate legacy execution architecture.

Each worker must have isolated mutable state, including where applicable:

  • Nextcloud installation/configuration;
  • database;
  • data directory;
  • users/groups;
  • app configuration;
  • generated files;
  • mail state;
  • mock/local services;
  • temporary files;
  • reports/logs.

Sharding must be deterministic and independent from semantic tagging.

Architecture

Shared package modernization
├── behat-builtin-extension
├── nextcloud-behat
│   ├── standalone assertions
│   ├── DataTable / DocString
│   └── reusable step modernization
├── mailpit-behat-extension
└── behat-tsa-extension
        ↓
LibreSign vendor-bin/behat + Behat 4
        ↓
LibreSign context/feature modernization

Test taxonomy ────────────────┐
Scenario concurrency safety ─┤
Deterministic sharding ──────┼─> isolated parallel Behat execution
Docker worker lifecycle ─────┘
        ↓
final Behat 3 cleanup + canonical documentation

Test-design principles

Scenarios must:

  • be executable independently where intended for normal parallel execution;
  • not rely on execution order;
  • create or declare the state they require;
  • not depend on state left by another scenario;
  • avoid unnecessary implementation-detail steps;
  • reuse generic Nextcloud steps when behavior is generic;
  • keep LibreSign business-language steps in the LibreSign layer.

Parallel-execution principles

Parallelism must be introduced only after isolation prerequisites are proven.

Parallel workers belonging to one test run may share immutable caches/images, but must not share mutable application state.

Untrusted pull-request runs must not receive unnecessary repository write credentials or long-lived secrets, and mutable state must not be reused across unrelated test runs.

Performance

Record a serial baseline before selecting the final concurrency level.

Compare at least, where runner resources permit:

  • one worker;
  • two workers;
  • four workers.

Measure:

  • wall-clock duration;
  • environment bootstrap duration;
  • Behat execution duration;
  • CPU/memory pressure where practical;
  • total runner usage;
  • failures/retries;
  • evidence of flakiness or state leakage.

Do not select the final worker count without measurement.

Compatibility

  • Target PHP >= 8.2.
  • Target Behat ^4.0.
  • New breaking releases of the shared testing packages do not retain Behat 3 compatibility.
  • LibreSign changes target main.
  • Do not backport this migration to stable branches.

Done when

  • All required shared Behat packages have Composer-installable Behat 4-compatible breaking releases.
  • LibreSign main executes its integration suite using Behat 4.
  • Behat tooling is isolated under vendor-bin/behat and remains resolvable for PHP 8.2.
  • YAML Behat configuration is no longer required.
  • Shared contexts use supported/public Behat APIs.
  • PHPUnit is not used merely as the runtime assertion library of shared Behat contexts.
  • Reusable step APIs use modern Behat argument types where applicable.
  • Generic Nextcloud reusable steps and LibreSign domain steps have documented ownership boundaries and have been modernized accordingly.
  • The test taxonomy is documented and applied.
  • Scenarios intended for normal parallel execution are independent of execution order and shared mutable state.
  • nextcloud-docker-development supports the complete Behat database matrix: SQLite, MySQL, MariaDB 10.6/10.11 and PostgreSQL.
  • Multiple isolated Nextcloud test environments can run concurrently.
  • Deterministic sharding executes every intended test exactly once across workers.
  • Parallel CI is measurably faster than the serial baseline.
  • Parallel execution does not introduce measurable cross-worker state leakage or systematic flakiness.
  • Failure output identifies the failing worker/shard and remains diagnosable.
  • Existing database/Nextcloud/PHP compatibility coverage is preserved.
  • Legacy Behat 3 infrastructure is removed after the new path is proven.
  • The supported developer workflow is documented in LibreSign's canonical documentation.

Target

Branch: main

Milestone: Next Major (36)

Pull requests should implement focused child issues, not this epic.

Dominant language
PHP
Stars
844
Forks
176
Avg merge
14h 3m
Merged PRs (30d)
344

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from LibreSign/libresign

All issues in LibreSign/libresign

Similar issues

More PHP issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.