MonoGame/MonoGame

Manual Test Framework

Open

#3,793 opened on Apr 30, 2015

View on GitHub
 (18 comments) (0 reactions) (0 assignees)C# (13,886 stars) (3,071 forks)batch import
Feature RequestHelp WantedTesting

Description

We should write a framework for manually run unit tests to augment the existing automated unit test framework.

These manual tests would be specifically for features like audio and input which are most easily validated by people. The trick would be to make sure the test is quick to do and hard to get wrong. Some core ideas:

  • The test app should be built and distributed as part of the automated build process.
  • The tests should execute in a fixed order.
  • You must complete one test to move to the next one.
  • Should take very little time to complete the tests... less than a few minutes.
  • The results should contain very basic machine info.
  • The tests should work on all our platforms... including phones.
  • Once all the tests are completed it should display the results, spit out a text file, and display a button to copy the results to your clipboard for easy pasting into GitHub.

The tests themselves should be crafted so that the user has to do the least work possible. We should ask the user to do something... verify the result wasn't an accident by repeating it... store the result of the test and move on to the next one. We should not depend on the user to judge themselves if the test passed or not.

The framework for this should be a standard MonoGame Game with some sort of TestScreen class that is overloaded for different tests. The framework would then just find them via reflection, sort them in some consistent order, and run them recording results.

The usage for these would be in validating PRs. If a PR changes something that is covered in the manual tests we need people to run thru them on all the target platforms before approval and merge. We would also run them once before we do a release.

Thoughts?

Contributor guide

Tech stack
csharp
Domain
testingtooling
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
C# knowledgeMonoGame basicsReflection in C#Unit testing concepts
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30
Research direction
This issue requests a manual test framework. To start, examine the existing automated test framework in the repository (likely under Tests/). Look for any previous attempts or discussions in the 18 comments. The proposed structure involves a 'TestScreen' class and reflection to discover tests. A good first step is to outline the required classes and interfaces, and perhaps prototype a simple test screen. Reach out to maintainers for clarification on expected behavior across platforms.