Restore should use new connections after receiving non-success status codes

Details about Problem

NuGet product used (NuGet.exe | VS UI | Package Manager Console | dotnet.exe): dotnet

NuGet version (x.x.x.xxx): --

dotnet.exe --version (if appropriate): SDK Version: 3.1.301

VS version (if appropriate): --

OS version (i.e. win10 v1607 (14393.321)): Mac 10.15

Worked before? No

Detailed repro steps so we can see the same problem

Sample from our logs:

Retrying 'FindPackagesByIdAsync' for source 'redacted'.
  Response status code does not indicate success: 503 (Service Unavailable).
Retrying 'FindPackagesByIdAsync' for source 'redacted'.
  Response status code does not indicate success: 503 (Service Unavailable).
Retrying 'FindPackagesByIdAsync' for source 'redacted'.
  Response status code does not indicate success: 503 (Service Unavailable).
Retrying 'FindPackagesByIdAsync' for source 'redacted'.
  Response status code does not indicate success: 503 (Service Unavailable).
Retrying 'FindPackagesByIdAsync' for source 'redacted'.
  Response status code does not indicate success: 503 (Service Unavailable).
...

1. Point NuGet at a feed that fails restore requests.
2. Observe (via packet capture or similar) that new connections are not created after failed requests, and instead pooled connections continue to be used.

The reason it's important to not use a pooled connection after failure is because modern websites are often behind load balancers that pin your TCP connection to a single machine. If that machine gets into a bad state it could be removed from the load balancer, but your connection would still go directly to that machine since removing from the LB usually only stops connection initiation.

Closing the connection after failures is good client hygiene to ensure that you get load balanced to a different machine with a higher chance of success. In the above example, NuGet was attempting to request packages repeatedly over the same connection, but that connection was load balanced to a bad node.