Formalising Testing Framework and Approach

Introducing a greater breakdown and categorisation of our various testing scripts and procedures to enable further scaling and automation beyond our current CI and compliance.

Below is a first list of, I think, the key categories that we need to focus on. Not all will be fully automated in the first instance and priorities will vary naturally.

Listed below for comments/input and then more granular plans to be formulated around each as needed.
Focus on Interoperability Test for 3.1 Milestone as discussed and planned.

I. Compliance

• Development Process Compliance Tests
  • e.g. branch naming, issue and PR template checks, label checking etc.
• Code Compliance Tests
  • Review BHoM_Test repo and Compliance_Test , Coverage_Test etc.
  • Creation of Wiki - explaining BHoM Compliance which can hyperlink to each encoded Compliance Test for clarity and transparency for all developers.

II. Functionality

• Data-driven Unit Tests
  • datasets required of serialised method base, input & expected output as per our BH.oM.Testing.UnitTest
  • Generation of dataset as per previous gh approach to be revisited and polished
• Interoperability Tests
  • In the first instance - scripts testing each of the implemented BHoM_Adapter functionality
• Issue Tests
  • As currently created to enable review of each PR.
    Issue_Tests guidelines and best practice to be developed further to enable associated Regression Tests where appropriate
• Regression Tests
  • Ensuring fixed bugs do not reappear. Ensuring previous working features do not break for subsequent Betas
  • Regression Tests are not necessarily a separate type of test - but keen to include as a discrete category to be be included in our process of ongoing development of tests and integrated into process of raising and successfully closing PR.
• Macro Tests
  • any combination of the above
• Backwards Compatibility Tests
  • Tests targeting deserialisation of previous versions of BHoM.
  • Potential to include creation as part of deprecation
• Performance tests
• Define metric of speed and checking that subsequent PRs are not X times slower to run tests

III. System

• Installer Testing
  • Works on end user machine
  • Metrics and licencing etc. information is correct.
  • Validation that installer accurately recreates BHoM environment on master say
• Environment Testing
  • Both code developer and end user (see above)

IV. User Testing

• Beta Testing
  • Formalised and focused user testing as part of Beta Release process
  • Comms around best practice raising of issues/ideas from wider user community beyond release, and with Beta in the wild. To all encourage engagement
• Sample Testing
  • Checking key samples are all still working for each release