An UPDATE for a record that is not in the zone fails permanently

Read the recordSet() and submitChanges code paths described in the issue, then inspect PR #84, which is already open against this issue. Trace how tolerated not-found results reach change submission and how empty refs affect UPDATE and DELETE. Done means the proposed cases are handled without breaking TXT target values; the issue provides no test file or test command.

Summary

When ExternalDNS asks this provider to update a record that is not present in the zone, the change fails with could not build record: not found — and fails again on every reconcile afterwards. Nothing converges, because an UPDATE never creates.

ExternalDNS does not promise that an UpdateNew endpoint already exists: its TXT registry can emit an UpdateNew for an ownership TXT that is absent from the zone.

Reproduction

An endpoint this instance owns, whose ownership records are incomplete:

1. The A record foo.example.com exists.
2. Its legacy ownership TXT <prefix>foo.example.com exists and names this instance as owner.
3. Its type-prefixed ownership TXT <prefix>a-foo.example.com is absent.

The desired A record has the same target and the same external-dns/resource label as the existing one, so the TXT values generated for UpdateOld and UpdateNew are identical and nothing else about the endpoint is changing.

Ownership still resolves through the legacy record, so the endpoint is managed, and the registry marks it with providerSpecificForceUpdate to backfill the missing type-prefixed TXT — which then travels in UpdateNew. The provider cannot apply it:

level=error msg="Failed to apply changes: encountered errors:
  [could not build record (&{UPDATE <prefix>a-foo.example.com 0 IN TXT  heritage=external-dns,external-dns/owner=cluster-a,external-dns/resource=ingress/default/foo []}): not found ...]"

The same hostnames fail on every cycle, indefinitely.

An interrupted write batch, or records left by an older implementation, can leave a zone in that state. The TXT registry keeps support for reading legacy ownership records and forces an update to backfill a missing type-prefixed one; that migration behaviour is already present in 0.16.1. Although 0.18 stopped generating legacy TXT records while keeping support for reading them, that change alone does not explain when this failure starts.

Why it can take the process down

submitChanges collects per-change errors and returns them together, so the failed changes come back as HTTP 500. That is retryable, the webhook provider wraps it in provider.SoftError, and the controller logs it and keeps going. Survivable on its own.

What is not survivable is the cost. recordSet issues a WAPI lookup per change, including for every change that is going to fail, and slow lookups make the batch exceed the webhook client's HTTP timeout. In 0.23.0 that timeout is the sum of the configured read and write timeouts — 15 seconds with the defaults — unless the request context expires earlier:

level=fatal msg="failed to do run once: Post \"http://localhost:8888/records\": context deadline exceeded"

That transport error is not wrapped as a provider.SoftError, so it terminates reconciliation and ExternalDNS exits. Repeated, this is CrashLoopBackOff.

Both 0.16.1 and 0.23.0 distinguish soft errors from fatal ones. What differs is the client: 0.16.1 builds &http.Client{}, with no overall timeout and no context on the request, so a slow batch simply finishes and comes back as the retryable 500. 0.23.0 applies the timeout above. A batch that is slow and failing can therefore start exiting the process after an upgrade, having been merely noisy before. Confirming that for a given deployment means checking its effective timeouts and its actual request durations.

Root cause

Two defects, one symptom.

recordSet() leaks a tolerated not-found. The lookup branches are written to tolerate it — that is what isNotFoundError is for, and why they do not abort:

err = p.client.GetObject(obj, "", queryParams, &res)
if err != nil && !isNotFoundError(err) {
    metrics.FailedApiCallsTotal.Inc()
    return
}

but err is a named return and is never cleared, so the not-found leaves the function anyway. buildRecord passes it up and submitChanges renders it as could not build record (...): not found.

submitChanges has no upsert path. With the above fixed, getRefID answers "" and no error for an absent record, and UpdateObject(obj, "") would be called with an empty ref.

Proposed fix

• Clear err after a tolerated NotFoundError in the five recordSet branches.
• UPDATE with an empty ref creates instead, rebuilding the record in its create shape — that path sets the View, which Infoblox needs to find the parent zone and rejects on an update.
• DELETE with an empty ref becomes a no-op: its desired state is already met. (Needed to avoid a regression once err is cleared.)
• recordSet works on a copy of the endpoint. Its TXT branch normalises Targets in place with strconv.Unquote, which is harmless while it runs once per change; the upsert calls it a second time, and without the copy a TXT whose value is itself a quoted string loses a layer.

PR: #84

Versions

Reproduced on v1.7.1 and on main (2f02eed) — the affected code is identical in both. ExternalDNS 0.16.1 and 0.23.0, Infoblox WAPI 2.6.1.