Health checker: add configurable no_traffic_interval delay

Title: Health checker: add configurable no_traffic_interval delay

Description:

What issue is being seen?

For a cluster whose health checker configuration is -

...
"health_checks": [
  {
    "grpc_health_check": {},
    "timeout": "1s",
    "interval": "10s",
    "interval_jitter": "3s",
    "initial_jitter": "3s",
    "unhealthy_threshold": 3,
    "healthy_threshold": 3,
  }
]
...

1. Server A's Envoy comes up t0.
2. A's health checker makes its first HC request for Server B at t1, it passes and gets registered in the cluster members to serve traffic to.
3. A starts traffic to B at t1+20s (this happens for us if the workload is a kafka consumer which takes about 20-30s during which rebalancing takes place)
4. B marks itself unhealthy at t1+25s and runs for 20 seconds before exiting. It expects all its clients to get updated about its unhealthy status during this duration.
5. After the first health check in step 2, A chooses to send the next health check request at t1+60s at which time B is unavailable so after unhealthy_threshold of 3 and interval of 3, A removes B from its cluster member at around t+80s. (60 + 2*10 seconds).
6. Requests are fired to B from t1+25 to t1+80 which should have stopped at latest by t1+35s. All requests beyond t1+45s are timed out with 504 as a result.

While going through the documentation I found a config "no_traffic_healthy_internal" available from version v1.16.0 which can override the "no_traffic_interval" if the upstream host is healthy, that does solve the above use-case but it also overwhelms the other upstream healthy clusters with unnecessary health check traffic where actual traffic wasn't required.

I believe that health checker should ignore zero connection used stats check after some amount of configurable graceful time has elapsed.

Logs

Server A's health checker fail logs

{"health_checker_type":"GRPC","host":{"socket_address":{"protocol":"TCP","address":"192.168.11.73","resolver_name":"","ipv4_compat":false,"port_value":16005}},"cluster_name":"Bservice","add_healthy_event":{"first_check":true},"timestamp":"2021-05-28T10:07:48.007Z"}
{"health_checker_type":"GRPC","host":{"socket_address":{"protocol":"TCP","address":"192.168.11.73","resolver_name":"","ipv4_compat":false,"port_value":16005}},"cluster_name":"Bservice","health_check_failure_event":{"failure_type":"NETWORK","first_check":false},"timestamp":"2021-05-28T10:08:51.908Z"}
{"health_checker_type":"GRPC","host":{"socket_address":{"protocol":"TCP","address":"192.168.11.73","resolver_name":"","ipv4_compat":false,"port_value":16005}},"cluster_name":"Bservice","health_check_failure_event":{"failure_type":"NETWORK","first_check":false},"timestamp":"2021-05-28T10:09:04.618Z"}
{"health_checker_type":"GRPC","host":{"socket_address":{"protocol":"TCP","address":"192.168.11.73","resolver_name":"","ipv4_compat":false,"port_value":16005}},"cluster_name":"Bservice","eject_unhealthy_event":{"failure_type":"NETWORK"},"timestamp":"2021-05-28T10:09:19.532Z"}
{"health_checker_type":"GRPC","host":{"socket_address":{"protocol":"TCP","address":"192.168.11.73","resolver_name":"","ipv4_compat":false,"port_value":16005}},"cluster_name":"Bservice","health_check_failure_event":{"failure_type":"NETWORK","first_check":false},"timestamp":"2021-05-28T10:09:19.532Z"}

Server B's had marked itself unhealthy at "2021-05-28T10:08:18.275051694Z" and was stopped at "2021-05-28T10:08:38.XXXZ"

Envoy version Checked on all envoy version > v1.15.3

Notes I have collected the logs from production envoys, I can make a repro of this soon and update the thread. On the surface it looks like this makes the decision of overriding the interval to no_traffic_interval