[bug-hunter] unifiedlogs custom dataset is ignored due to hardcoded event.dataset

Start with packages/unifiedlogs/elasticsearch/ingest_pipeline/default.yml:23-26, then compare it with packages/unifiedlogs/agent/input/unifiedlogs.yml.hbs and packages/unifiedlogs/manifest.yml:27-35. Run the reproduction script from the issue and verify that event.dataset matches the configured data_stream.dataset, including elastic_agent.test.

Impact

Users who configure a custom dataset for the unifiedlogs input (for example elastic_agent.test) still get events with event.dataset: unifiedlogs.log. This creates deterministic field mismatch (data_stream.dataset != event.dataset) and breaks dataset-based filters, dashboards, and detection/content that relies on event.dataset.

Reproduction Steps

1. Save and run this script from repo root:

import yaml
from copy import deepcopy

pipeline_path = 'packages/unifiedlogs/elasticsearch/ingest_pipeline/default.yml'
manifest_path = 'packages/unifiedlogs/manifest.yml'
input_tpl_path = 'packages/unifiedlogs/agent/input/unifiedlogs.yml.hbs'

with open(pipeline_path, 'r', encoding='utf-8') as f:
    pipeline = yaml.safe_load(f)

with open(manifest_path, 'r', encoding='utf-8') as f:
    manifest = yaml.safe_load(f)

vars_list = manifest['policy_templates'][0]['vars']
assert any(v.get('name') == 'data_stream.dataset' for v in vars_list)

with open(input_tpl_path, 'r', encoding='utf-8') as f:
    tpl = f.read()
assert 'dataset: \{\{data_stream.dataset}}' in tpl

doc = {
    'data_stream': {'dataset': 'elastic_agent.test'},
    'message': '{"eventMessage":"hello"}'
}

for p in pipeline.get('processors', []):
    if 'set' in p:
        s = p['set']
        field = s.get('field')
        value = s.get('value')
        if field and value is not None:
            cur = doc
            parts = field.split('.')
            for part in parts[:-1]:
                if part not in cur or not isinstance(cur[part], dict):
                    cur[part] = {}
                cur = cur[part]
            cur[parts[-1]] = deepcopy(value)

expected = doc['data_stream']['dataset']
actual = doc.get('event', {}).get('dataset')
print(f'data_stream.dataset={expected}')
print(f'event.dataset={actual}')
if actual != expected:
    print('BUG REPRODUCED: event.dataset is hardcoded and does not follow configured dataset')
    raise SystemExit(1)

2. Run:

python /tmp/gh-aw/agent/repro_unifiedlogs_dataset.py

Expected vs Actual

Expected: event.dataset should match the configured data_stream.dataset (e.g. elastic_agent.test).

Actual: event.dataset is always unifiedlogs.log.

Observed output:

data_stream.dataset=elastic_agent.test
event.dataset=unifiedlogs.log
BUG REPRODUCED: event.dataset is hardcoded and does not follow configured dataset

Failing Test

The reproduction script above is a minimal failing test (exits non-zero when mismatch is present).

Evidence

• packages/unifiedlogs/elasticsearch/ingest_pipeline/default.yml:23-26 unconditionally sets:
  • field: event.dataset
  • value: unifiedlogs.log
• packages/unifiedlogs/agent/input/unifiedlogs.yml.hbs:2-3 wires user-configurable data_stream.dataset.
• packages/unifiedlogs/manifest.yml:27-35 exposes data_stream.dataset as required user var.
• Commit c2f386e55078c0ae340661793d2d0e6613b3780d introduced dataset configurability, but ingest pipeline still hardcodes event.dataset.

[!NOTE]

🔒 Integrity filtering filtered 2 items

Integrity filtering activated and filtered the following items during workflow execution.
This happens when a tool call accesses a resource that does not meet the required integrity or secrecy level of the workflow.

• issue:elastic/integrations#unknown (search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• issue:#unknown (search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)

---

What is this? | From workflow: Bug Hunter

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

• expires on Apr 8, 2026, 11:38 AM UTC