Unify testing framework

Test flakiness and maintenance cost remains one of the larger issues for Etcd project. Proposals like https://github.com/etcd-io/etcd/issues/13167 to track flakiness and contributions to address singular tests have definitely helped however to fully resolve the problem we also need to address the root cause. Etcd projects currently maintains multiple disconnected ways to run tests (unit, integration, e2e, functional) that mostly verify the same scenarios. Testing features on different level is itself desired as it allows to isolate failures and speed up resolution, however not in a way that is currently done in Etcd. Almost zero test scenario and test framework is reused. This means that we have multiplied test code lines without any benefit.

Goal:

• Reduce number of test code without affecting test coverage
• Simplify process of adding new integration/e2e test
• Make it easier to isolate test failures by comparing test results from different methods

I propose to unify testing by identifying common test scenarios, test framework operations and making them accessable the same way no matter which test method is used (unit, integration, e2e, functional). For example simple put&get test scenario goes through very same stages no matter if this is integration or e2e cluster: create cluster, execute put, execute get, compare results, cleanup. This test should be written once and executed on different layers, no matter if underneath framework runs a fake grpc client or starts whole process. With this we will reduce number of code lines as each test will be needed to be implemented once, hide all the differences between test types reducing knowledge needed to add new tests and improve deflaking process by comparing same test results from different methods (we can isolate test location, if both e2e and integration tests fail then feature is broken, if only one of them then the test framework)

At this moment I would be most effective to target unifying integration and e2e tests as they are most similar in both cluster setup and test scenarios. We should be able to identify minimal common interface for cluster creation and communication that would allow us to start rewriting tests.

Plan:

• Create new Cluster and Client interfaces in tests/framework.
• Find common function in tests/framework/e2e/cluster.go and tests/framework/integration/cluster.go and migrate them one by one.
• Deduplicate test scenarios between e2e and integration tests
• Create common tests located in tests/common that will be invoked by e2e and integration.

cc @ptabor @ahrtr @spzala

• tests/e2e/logging_test.go
• tests/e2e/reproduce_18667_test.go
• tests/e2e/ctl_v3_auth_test.go
• tests/e2e/member_no_proxy_test.go
• tests/e2e/zap_logging_test.go
• tests/e2e/v3_lease_no_proxy_test.go
• tests/e2e/v3_curl_maintenance_test.go
• tests/e2e/v3_curl_lock_test.go
• tests/e2e/v3_curl_lease_test.go
• tests/e2e/v3_curl_election_test.go
• tests/e2e/v3_curl_cluster_test.go
• tests/e2e/v3_curl_auth_test.go
• tests/e2e/v3_cipher_suite_test.go
• tests/e2e/v2store_deprecation_test.go
• tests/e2e/utl_migrate_test.go
• tests/e2e/utils.go
• tests/e2e/runtime_reconfiguration_test.go
• tests/e2e/reproduce_19406_test.go
• tests/e2e/reproduce_17780_test.go
• tests/e2e/promote_experimental_flag_test.go
• tests/e2e/metrics_test.go
• tests/e2e/main_test.go
• tests/e2e/leader_snapshot_no_proxy_test.go
• tests/e2e/http_health_check_test.go
• tests/e2e/graceful_shutdown_test.go
• tests/e2e/gateway_test.go
• tests/e2e/force_new_cluster_test.go
• tests/e2e/failover_test.go
• tests/e2e/etcd_release_upgrade_test.go
• tests/e2e/etcd_mix_versions_test.go
• tests/e2e/etcd_grpcproxy_test.go
• tests/e2e/etcd_config_test.go
• tests/e2e/doc.go
• tests/e2e/discovery_v3_test.go
• tests/e2e/defrag_no_space_test.go
• tests/e2e/ctl_v3_test.go
• tests/e2e/ctl_v3_snapshot_test.go
• tests/e2e/ctl_v3_role_test.go
• tests/e2e/ctl_v3_move_leader_test.go
• tests/e2e/ctl_v3_member_test.go
• tests/e2e/ctl_v3_member_no_proxy_test.go
• tests/e2e/ctl_v3_lock_test.go
• tests/e2e/ctl_v3_lease_test.go
• tests/e2e/ctl_v3_elect_test.go
• tests/e2e/ctl_v3_defrag_test.go
• tests/e2e/ctl_v3_completion_test.go
• tests/e2e/ctl_v3_auth_security_test.go
• tests/e2e/ctl_v3_auth_no_proxy_test.go
• tests/e2e/ctl_v3_auth_cluster_test.go
• tests/e2e/cmux_test.go
• tests/e2e/cluster_downgrade_test.go