Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[ML] Inconsistent categorization_filters content for ML jobs in Kibana

Open
#917 3 comments 0 reactions 1 assignee View on GitHub

@lcawl is already working on this.

Since Jan 10, 2020.

Assessment

This issue has not been assessed yet.

Description

:ml

Original comment by @lcawl:

On 5.4 and 5.5 builds from s3, there is disparity with respect to the categorization_filters content.

For example, when I run the following API in the Dev Tools tab in Kibana:

PUT _xpack/ml/anomaly_detectors/it_ops_logs
{
  "description": "IT Ops application logs",
  "analysis_config": {
    "categorization_field_name": "message",
    "bucket_span":"30m",
    "detectors": [
      {
      "function": "count",
      "by_field_name": "mlcategory",
      "detector_description": "Unusual message counts"
    }],
    "categorization_filters":"\\[statement:.*\\]"
  },
  "analysis_limits": {
      "categorization_examples_limit": 5
  },
  "data_description": {
    "time_field": "time",
    "time_format": "epoch_ms"
  }
}

It returns the following:

...
"analysis_config": {
    "bucket_span": "30m",
    "categorization_field_name": "message",
    "categorization_filters": [
      """\[statement:.*\]"""
    ]
...

Ditto when I run GET _xpack/ml/anomaly_detectors/it_ops_logs from the Dev Tools tab.

If I run it from the command line, however, I get the following:

curl -u elastic:changeme -XGET 'localhost:9200/_xpack/ml/anomaly_detectors/it_ops_logs'
{"count":1,"jobs":[{"job_id":"it_ops_logs","job_type":"anomaly_detector","job_version":"5.5.0","description":"IT Ops application logs","create_time":1497295377972,"analysis_config":{"bucket_span":"30m","categorization_field_name":"message","categorization_filters":["\\[statement:.*\\]"],"detectors":[{"detector_description":"Unusual message counts","function":"count","by_field_name":"mlcategory","detector_rules":[],"detector_index":0}],"influencers":[]},"analysis_limits":{"categorization_examples_limit":5},"data_description":{"time_field":"time","time_format":"epoch_ms"},"model_snapshot_retention_days":1,"results_index_name":"shared"}]}Lisas-MBP:~ lcawley$ 

Note that the categorization_filters content matches what I specified in the PUT command in this case.

The "categorization_filters":"""[statement:.*]""" does not actually seem to be invalid (I can successfully create a job in the Dev Tools with that syntax, it's just a bit confusing that it's returning a different syntax in Dev Tools vs command line.

For interest's sake, to get this to work in the advanced job wizard I must use the following syntax:
filter1
.... the categorization_filters property then has the desired format in the Edit JSON tab:
filter2

Dominant language
Java
Stars
105
Forks
249
PR merge metrics
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from elastic/stack-docs

All issues in elastic/stack-docs

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.