trailofbits/manticore

Cleanup Manticore workspace when a test passes

Open

#1617 opened on Feb 21, 2020

View on GitHub
 (3 comments) (0 reactions) (0 assignees)Python (3,469 stars) (481 forks)batch import
good first issuetesting

Description

It's nice if testing doesn't produce unnecessary artifacts after running with success but keeps relevant info on fail for further inspection.

Pytest should have a nice way of creating fixtures that can post-process a test based on whether it failed or not https://docs.pytest.org/en/latest/example/simple.html#making-test-result-information-available-in-fixtures

If a test fails, then keep the Manticore workspace, and, if possible, print the associated failed test's workspace directory in the pytest output to know where the user should look.

If a test passes, then remove the Manticore workspace.

Since some of our tests are structured differently, the exact implementation will likely look different for different tests.

@reaperhulk Do you have any comments, suggestions, or references for doing something like this?

Contributor guide

Tech stack
python
Domain
testingdeveloper experience
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Pythonpytest
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
75
Research direction
Investigate the existing test setup in the Manticore repository to understand how workspaces are created and destroyed. Look at pytest fixture patterns, particularly using the request object to determine test outcome as referenced in the pytest documentation. Then implement a fixture that conditionally removes the workspace on success and prints the path on failure. Note that the implementation may vary across different test files as mentioned by the maintainer.