Tanka Loki to allow user configurable kv store backend instead of hard coded as 'consul'

Is your feature request related to a problem? Please describe. No

Describe the solution you'd like Allow user to specify already deployed or another kv store such as etcd in main.libsonnet file used in ingestor, distributor, and ruler components. Currently it is hard coded as consul in config.libsonnet, see below

      ingester: {
        chunk_idle_period: '15m',
        chunk_block_size: 262144,
        max_transfer_retries: 60,

        lifecycler: {
          ring: {
            heartbeat_timeout: '1m',
            replication_factor: $._config.replication_factor,
            kvstore: {
              store: 'consul',
              consul: {
                host: 'consul.%s.svc.cluster.local:8500' % $._config.namespace,
                http_client_timeout: '20s',
                consistent_reads: true,
              },
            },
          },
...
      distributor: {
        // Creates a ring between distributors, required by the ingestion rate global limit.
        ring: {
          kvstore: {
            store: 'consul',
            consul: {
              host: 'consul.%s.svc.cluster.local:8500' % $._config.namespace,
              http_client_timeout: '20s',
              consistent_reads: false,
              watch_rate_limit: 1,
              watch_burst_size: 1,
            },
          },
        },
      },
...
      ruler: if $._config.ruler_enabled then {
        rule_path: '/tmp/rules',
        enable_api: true,
        alertmanager_url: 'http://alertmanager.%s.svc.cluster.local/alertmanager' % $._config.namespace,
        enable_sharding: true,
        enable_alertmanager_v2: true,
        ring: {
          kvstore: {
            store: 'consul',
            consul: {
              host: 'consul.%s.svc.cluster.local:8500' % $._config.namespace,
            },
          },
        },

The option should allow user to specify related tls_config options too to authenticate with etcd, which is very likely to enabled mTLS.

Example user editable fields in main.libsonnet.

kvstore: {
  store: 'etcd',
  etcd: {
    host: etcd.db.svc.cluster.local:8888,
    scheme: https
    tls_config: {
      ca_cert_secret_name: <k8s Secret with Opaque type> 
      client_cert_secret_name: <k8s Secret with TLS type> [Auto read the tls.key and tls.crt in this secret]
}

Describe alternatives you've considered None, stick with consul.

Additional context None