add a hook that runs before ALL test projects / test modules and after all test projects / test modules
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- csharp
- Domain
- devtools, testing-qa
Research direction
Start by tracing how dotnet test, dotnet run --project, and server mode discover and execute test projects or modules. Compare where a session-wide lifecycle hook could be introduced across those entry points. Done means defining and validating a design that runs the before and after hooks exactly once for the whole test session, including parallel projects.
Written by the indexing model from the issue text.
Description
Summary
currently at the moment some stuff is extremly hard with the M.T.P or even before. and that is running things before all tests projects did run and also after all test projects did run.
the problem is even amplified because its now possible to run tests in multiple ways, there is:
- dotnet test
- dotnet run --project TestProj.csproj
- server mode
probably even more cases i do not now about.
now if you do not have a single csproj that will run as a cordinater you are basically in a game over statement if developers use either one and you want to create a test solution that basically warms stuff up or builds grouped reports at the end.
its also impossible to correctly build solutions to limit parallel execution of tests since, test projects run in parallel and tests inside the projects can also run in parallel. in a big monorepo that gets ugly insanly fast.
things that are also not possible because that feature is missing is test sharding, if a global hook would exists, it might even be possible to provide a way to introduce new args that would allow these features.
also running a single testcontainer, for example a database is basically impossible, running all migrations once before all tests, basically impossible, you basically have no concurrency control at all.
there can be a "hacky" solution for the "start", but not for the finish. the start can be done by creating a file level lock (urgh) the end can be done with a file level lock, if you only need to merge files, but that breaks for stopping containers.
Background and Motivation
we basically want more concurrency control and starting containers and migrations once for the whole test session no matter how many test projects we have.
Proposed Feature
add a way of providing a before all/after all hook, that does run exactly once, but not twice, if you have multiple projects that contain tests. maybe even provide a way to announce possible new arguments, however that would be hard if there are multiple extensions that provide the same arg, so that would need further design work.
the hook should run before anything, maybe for the server mode it can be done when the server mode is launched, its not like test containers would need to be reloaded all the time.
Alternative Designs
provide a solution to stop running test projects in parallel, however that would fix some issues it would mean that we need to pay the bootup multiple times.
- Dominant language
- C#
- Stars
- 1.1k
- Forks
- 312
- Avg merge
- 9h 13m
- Merged PRs (30d)
- 411
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from microsoft/testfx
-
type/automation type/tech-debt
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
area/mtp-github-actions-report external/dotnet-test needs/triage
Difficulty 4/5 3-5 days Newbie friendliness 56/100
Maintainers usually reply within 1 day
-
area/native-aot area/parameterized-tests needs/triage type/regression
Difficulty 3/5 1-2 days Newbie friendliness 52/100
Maintainers usually reply within 1 day
-
area/mtp needs/triage
Difficulty 2/5 1-3 hours Newbie friendliness 25/100
Maintainers usually reply within 1 day
-
Allow a trailing CancellationToken parameter on MSTest test methodsPossibly taken @Evangelink claimed this 1 day ago. Openarea/mstest area/parameterized-tests needs/triage
Difficulty 4/5 3-5 days Newbie friendliness 22/100
Maintainers usually reply within 1 day
All issues in microsoft/testfx
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
nightscout/nocturne#2135 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
Devolutions/UniGetUI#5541 ·
Maintainers usually reply within 1 day
-
agentic-workflows Needs: Triage :mag:
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 1 day
-
[誤判定]Openlinux アプリ: ブラウザー 再現済み 要トリアージ 誤判定
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
yksr-melt/Meltype#853 · 2 comments ·
Maintainers usually reply within 1 day