Do not retry HTTP responses that indicate unsupported request semantics

Start in lib/saluki-io/src/net/util/retry/classifier/http.rs, especially default_should_retry and the tests around default_classifier_retries_5xx_and_most_4xx. Review the existing permanent and retryable status tables, then run the classifier tests. Done means 404, 405, 415, and 501 are permanent while 408, 429, and transient 5xx responses remain retryable, with custom predicates still covered.

Human Intro

So, 403's are perhaps special in that we use them to signal a need for api_key refresh. But the other types listed below, namely, 404, 405, 415, and 501 do seem like reasonable additions to the "do not retry" list. As for 403's, they are logically something we should not retry but we need to trace the effect that would have on api_key.

Note: this came up during an investigation of increased retries, but I do not think there is any evidence that this was a root cause of that problem. This was more of a dive-by "by the way" fix from the LLM.

The following was written by GPT-5.6 Sol:

Background

default_should_retry treats 400, 401, 403, and 413 as permanent failures. It retries every other 4xx and 5xx response.

The tests encode this policy and currently assert that 404 is retryable in default_classifier_retries_5xx_and_most_4xx.

Some responses describe a request that cannot succeed when repeated unchanged against the same endpoint:

• 404 Not Found;
• 405 Method Not Allowed;
• 415 Unsupported Media Type;
• 501 Not Implemented.

Retrying these responses can keep a misrouted or incompatible request in the retry queue indefinitely.

Problem

The default classifier should not retry responses that indicate a stable mismatch between the request and endpoint.

We should retain retries for statuses that may recover, including 408, 429, and transient 5xx responses. Callers should still be able to add narrowly scoped predicates when an endpoint has a documented exception.

Implementation Ideas

We could extend the permanent-status match in default_should_retry with 404, 405, 415, and 501. Before adding 501, it would be worth confirming whether any supported endpoint uses it during a transient rollout state.

An alternative is to express the policy as named status groups so the distinction between request-semantics failures and transient service failures remains visible in the code.

The existing OR-composed custom predicates should remain available for endpoint-specific behavior.

Testing Ideas

Move 404 out of the retryable test table and add 404, 405, 415, and 501 to the permanent-status table. The test should fail before the classifier changes.

Keep explicit assertions that 408, 429, 500, 502, 503, and 504 remain retryable. Preserve the test showing that a custom predicate can opt a normally permanent status back into retries.