HEP: Re-location e2e test cases and redesign test suites

Why do we need it?

As the evoluation of Higress, more and more e2e test cases are needed. Currently, all test cases are in the test/e2e/conformance/tests folder. And "feature" are used to categorize them.

Now, we have three major feature groups: core, discovery and wasm. Since Higress adds the support to Gateway API, I think all these test cases need to be re-implemented based on Gateway API CRDs. If so, where to put these new cases and how to trigger them would be two immediate questions.

How could it be?

Here I share some of my ideas.

1. Should we use a new entry to run Gateway API test cases?

The current implementation of e2e test framework utitlizes several global variables to store test cases and load test data. If we create a new entry, it would be better to have a test case registry. So we will need to refactor the framework to support this, which will take some work.

I would perfer to have a unified entry for e2e tests, but use different "features" to distinguish these cases. Current features like "http", "dubbo", "wasm-go" are quite function oriented. We can add new features like "ingresss", "gateway-api" to mark the target configuration mechanism of each case.

2. How should we let the entry know which cases we'd like to run?

I think we can introduce a new group kind of term, suite. Each suite has a name and contains several features. For example, "ingress-stable" suite can contain features like "http", "dubbo", "nacos", "consul", and "eureka", and "ingress-experimental" suite can contain features like "wasm-go" and "wasm-cpp".

The entry can accept the suite name from command line args, so user can specifiy which suite he'd like to run through the command line.

3. Do we have more fine-grain control inside each suite?

Yes. So far we have wasm-plugin-type and wasm-plugin name flags defined for wasm features. We can add flag support to suite, to let each suite decide what features or configurations shall be applied to the test runner.

Anything else?

Of course this would be a big change, maybe including moving or renaming existed test cases. But I think it would improve the extensibility of our e2e test framework. Please share your opinions below. Thanks.

TODO

• #566
• #551