[Bug]: Jaeger Query Tags Search Causes Nested Search Error

What happened?

When searching for tags in the UI, a "failed to create query: [nested] failed to find nested object under path [tags]" error is thrown by jaeger's query. I would think ignore_unmapped would be used here to prevent this error, but it isn't in the query.

Steps to reproduce

1. Use collector with the following configuration:

es.use-ilm: False
es.index-date-separator: .
es.server-urls: <intake nodes>
es.tags-as-fields.all: True
collector.zipkin.host-port: 9411
es.max-span-age: 744h0m0s
es.use-aliases: True
es.create-index-templates: False
es.username: "jaeger"
es.password: <pw>

2. Use query with the following configuration:

query.max-clock-skew-adjustment: 300s
es.index-date-separator: .
es.server-urls: https://<cluster LB>:9200
es.tags-as-fields.all: True
es.use-ilm: False
es.max-span-age: 744h0m0s //tried with smaller values as well
es.use-aliases: True
es.tls.skip-host-verify: True
es.create-index-templates: False
es.username: "jaeger"
es.password: <pw>

3. Ingest spans
4. Attempt to search for any tags
5. See error

Expected behavior

I expect the nested error to be ignored and the search to work.

Relevant log output

{"level":"debug","ts":1662507212.5653389,"caller":"zapgrpc/zapgrpc.go:129","msg":"POST /jaeger-span-read/_search?ignore_unavailable=true&rest_total_hits_as_int=true HTTP/1.1\r\nHost: <host>:9200\r\nUser-Agent: elastic/6.2.37 (linux-amd64)\r\nTransfer-Encoding: chunked\r\nAccept: application/json\r\nAuthorization: Basic <>\r\nContent-Type: application/json\r\nAccept-Encoding: gzip\r\n\r\n400\r\n{\"aggregations\":{\"traceIDs\":{\"aggregations\":{\"startTime\":{\"max\":{\"field\":\"startTime\"}}},\"terms\":{\"field\":\"traceID\",\"order\":[{\"startTime\":\"desc\"}],\"size\":20}}},\"query\":{\"bool\":{\"must\":[{\"range\":{\"startTimeMillis\":{\"from\":1662503612261,\"include_lower\":true,\"include_upper\":true,\"to\":1662507212261}}},{\"match\":{\"process.serviceName\":{\"query\":\"<Service> Client\"}}},{\"bool\":{\"should\":[{\"bool\":{\"must\":{\"regexp\":{\"tag.screen_height\":{\"value\":\"1080\"}}}}},{\"bool\":{\"must\":{\"regexp\":{\"process.tag.screen_height\":{\"value\":\"1080\"}}}}},{\"nested\":{\"path\":\"tags\",\"query\":{\"bool\":{\"must\":[{\"match\":{\"tags.key\":{\"query\":\"screen_height\"}}},{\"regexp\":{\"tags.value\":{\"value\":\"1080\"}}}]}}}},{\"nested\":{\"path\":\"process.tags\",\"query\":{\"bool\":{\"must\":[{\"match\":{\"process.tags.key\":{\"query\":\"screen_height\"}}},{\"regexp\":{\"process.tags.value\":{\"value\":\"1080\"}}}]}}}},{\"nested\":{\"path\":\"logs.fields\",\"query\":{\"bool\":{\"must\":[{\"match\":{\"logs.fields.key\":{\"query\":\"screen_height\"}}},{\"regexp\":{\"logs.fields.value\":{\"value\":\"1080\"}}}]}}}}]}}]}},\"size\":0}\r\n0\r\n\r\n\n"}
{"level":"debug","ts":1662507212.573889,"caller":"zapgrpc/zapgrpc.go:129","msg":"HTTP/1.1 400 Bad Request\r\nContent-Length: 905\r\nConnection: keep-alive\r\nContent-Type: application/json; charset=UTF-8\r\nDate: Tue, 06 Sep 2022 23:33:32 GMT\r\nX-Elastic-Product: Elasticsearch\r\n\r\n{\"error\":{\"root_cause\":[{\"type\":\"query_shard_exception\",\"reason\":\"failed to create query: [nested] failed to find nested object under path [tags]\",\"index_uuid\":\"3_fWA2cPSgiezCxBAi55fw\",\"index\":\"jaeger-span-write-<metadata>\"}],\"type\":\"search_phase_execution_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[{\"shard\":0,\"index\":\"jaeger-span-write-<metadata>\",\"node\":\"D2OukQsjT_yWLVGhSZMUwg\",\"reason\":{\"type\":\"query_shard_exception\",\"reason\":\"failed to create query: [nested] failed to find nested object under path [tags]\",\"index_uuid\":\"3_fWA2cPSgiezCxBAi55fw\",\"index\":\"jaeger-span-write-<metadata>\",\"caused_by\":{\"type\":\"illegal_state_exception\",\"reason\":\"[nested] failed to find nested object under path [tags]\"}}}]},\"status\":400}\n"}
{"level":"error","ts":1662507212.5740166,"caller":"app/http_handler.go:487","msg":"HTTP handler, Internal Server Error","error":"search services failed: elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]","stacktrace":"github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleError\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:487\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).search\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:236\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2046\ngithub.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5\n\tgithub.com/opentracing-contrib/go-stdlib@v1.0.0/nethttp/server.go:154\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2046\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2046\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\tgithub.com/gorilla/mux@v1.8.0/mux.go:210\ngithub.com/jaegertracing/jaeger/cmd/query/app.additionalHeadersHandler.func1\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/additional_headers_handler.go:28\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2046\ngithub.com/gorilla/handlers.CompressHandlerLevel.func1\n\tgithub.com/gorilla/handlers@v1.5.1/compress.go:141\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2046\ngithub.com/gorilla/handlers.recoveryHandler.ServeHTTP\n\tgithub.com/gorilla/handlers@v1.5.1/recovery.go:78\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2878\nnet/http.(*conn).serve\n\tnet/http/server.go:1929"}

Screenshot

No response

Additional context

{
  "_index": "jaeger-span-write-<metadata>",
  "_type": "_doc",
  "_id": "P3AGFYMBB_hNjC7OD2UC",
  "_version": 1,
  "_score": 1,
  "_source": {
    "traceID": "4563888ef57951d46bbf8c77c41d2cf7",
    "spanID": "0000017af27bfe5b",
    "flags": 2,
    "operationName": "ojButton Click",
    "references": [],
    "startTime": 1662504984000000,
    "startTimeMillis": 1662504984000,
    "duration": 1185000,
    "tags": [],
    "tag": {
      ...
      "screen_height": "1080",
      "screen_orientationType": "landscape-primary",
      "screen_width": "1920" ...
    },
    "logs": [],
    "process": {
      "serviceName": "<Service> Client",
      "tags": []
    }
  }
}

Jaeger backend version

v1.26

SDK

No response

Pipeline

Java -> Jaeger Collector (zipkin format via :9411) -> Intake ELK cluster -> CCR -> Query ELK Cluster -> Jaeger Query/UI

Stogage backend

Elasticsearch 7.16.3

Operating system

Oracle Linux Server 7.9

Deployment model

No response

Deployment configs

No response