Create a Terraform mirror protocol datasource

How are you running Renovate?

Self-hosted Renovate

If you're self-hosting Renovate, tell us what version of Renovate you run.

35.32.2

If you're self-hosting Renovate, select which platform you are using.

Bitbucket Server

Was this something which used to work for you, and then stopped?

I am trying to get this working for the first time

Describe the bug

The logic for the Terraform Module and Provider datasource is flawed in that it uses the service discovery API that is only available on the official Terraform registries. This API is not to be used with custom registries. The documentation states this on Hashicorp here

The Provider Network Mirror protocol does not use the service discovery indirection, because a network mirror location is only a physical location and is never used as part of the identifier of a dependency in a Terraform configuration.

There are two problems with the logic in the datasource:

• First, the equality check to see if the registry is hashicorp and therefore has the extended API, does not account for configuration that might change the default registry URLs
• The service discovery API is still called even if the result is not required

See the logic here: https://github.com/renovatebot/renovate/blob/main/lib/modules/datasource/terraform-provider/index.ts#L65-L81

Given this configuration of renovate:

{
  "packageRules": [
    "matchDatasources": [
       "terraform-provider"
    ],
    "defaultRegistryUrls": [
      "https://mycustommirror.com"
    ]
  ]
}

First, the equality check in the lines I mentioned, just check the position of the registry URL in the list, instead of actually checking the contents are hashicorp. Second, service discovery is still called even if it is not required because queryRegistryExtendedApi should not be called.

The specific problem is here:

    const serviceDiscovery = await this.getTerraformServiceDiscoveryResult(
      registryUrl
    );

which is being called regardless of the registryUrl. This should only be called if the registry URL is from hashicorp, not from mirrors.

Custom solutions like Artifactory that implement the Terraform specification, rightly do not provide a .well-known/terraform.json, so any hosted registry will not work.

Relevant debug logs

{"name":"renovate","hostname":"renovate-hn7-mzgtc","msg":"GET https://myregistry.com/.well-known/terraform.json = (code=ERR_NON_2XX_3XX_RESPONSE, statusCode=404 retryCount=0, duration=16)"}

Have you created a minimal reproduction repository?

I have linked to a minimal reproduction in the description above