[azure-ai-ml] Reserved system tags cannot be added, changed, replaced, set to null, or deleted after run creation.

Start with the reproduction in the issue and inspect azure/ai/ml/operations/_job_ops_helper.py:334 to confirm how jobs.stream surfaces the server-side failure. Compare the successful and failing AutoML runs, including the ai-ml-automl:41 environment and child-job tag behavior. Done means identifying a service-side or runtime fix that lets trial child jobs complete without the reserved-tag validation error.

• Package Name:
• azure-ai-ml
• Package Version:
• 1.32.0
• Operating System:
• Windows 11 Pro 25H2
• Python Version:
• 3.10

Describe the bug
Every AutoML regression trial (child job) submitted via azure.ai.ml.automl.regression() started failing on 2026-09-26 with the following service-side error, while the exact same job definition completed successfully on 2026-09-25:

{
    "error": {
        "code": "ValidationError",
        "message": "Reserved system tags cannot be added, changed, replaced, set to null, or deleted after run creation.",
        "message_parameters": {},
        "details_uri": "https://aka.ms/azureml-run-troubleshooting",
        "target": "Tags",
        "details": []
    },
    "time": "0001-01-01T00:00:00.000Z"
}

Important observations:

1. Job submission succeeds. ml_client.jobs.create_or_update(job) returns a valid job and a RunId is assigned. The exception surfaces on the client only because ml_client.jobs.stream(...) re-raises the failure of the server-side run as JobException (azure/ai/ml/operations/_job_ops_helper.py:334).
2. Setup and featurization child jobs complete successfully. The Command type setup child job and the automl.fe featurization child job both finish with status Completed, and the parent job's outputs/featurization/** and outputs/_automl_internal/featurized_cv_split_*.pkl artifacts are all produced.
3. All model-training trial child jobs fail after ~30-38 seconds, all with the identical error above (target: Tags).
4. The training itself actually succeeds inside the failing child jobs. Downloading the outputs of a failed trial child job shows a fully produced model and all downstream artifacts:

   automl_driver.py
   predicted_true
   residuals
   outputs/model.pkl
   outputs/internal_cross_validated_models.pkl
   outputs/featurization_summary.json
   outputs/engineered_feature_names.json
   outputs/pipeline_graph.json
   outputs/scoring_file_v_1_0_0.py
   outputs/scoring_file_v_2_0_0.py
   outputs/scoring_file_pbi_v_1_0_0.py
   outputs/generated_code/script.py
   outputs/generated_code/script_run_notebook.ipynb
   outputs/mlflow-model/{MLmodel, model.pkl, conda.yaml, python_env.yaml, requirements.txt}
   

   A failed trial ran for 34.36s versus 39.74s for a successful trial on the previous day, i.e. the run reaches the very end of its lifecycle. This strongly suggests the failure happens when the AutoML runtime writes its final system tags (e.g. _aml_system_codegen, _aml_system_automl_is_child_run_end_telemetry_event_logged) back to the run at completion time — a code path entirely inside the service-managed AutoML runtime, not in user code.
5. No relevant logs are produced. Neither the parent nor the failing child jobs contain logs/ or user_logs/ folders; only outputs/ artifacts exist. The only error information available is the message above shown in the Studio Overview blade.

I have verified that nothing on the client side changed between the working and failing runs(see "Additional context" for the full diff).

To Reproduce
Steps to reproduce the behavior:

1. Create an AutoML regression job with custom featurization, cross validation and ensembling:

   from azure.ai.ml import MLClient, Input
   from azure.ai.ml.automl import regression
   from azure.ai.ml.constants import AssetTypes
   
   job = regression(
       training_data=Input(type=AssetTypes.MLTABLE, path="<data-asset-id>"),
       target_column_name="<target>",
       primary_metric="spearman_correlation",
       n_cross_validations=5,
       compute="<cpu-cluster>",
       experiment_name="<experiment-name>",
       tags={"kaisai_id": "<id>", "dataset_name": "<name>"},
   )
   job.set_featurization(
       mode="custom",
       column_name_and_types={
           "<col_a>": "Categorical",
           "<col_b>": "Numeric",
           # ... 15 columns total
       },
   )
   job.set_limits(
       timeout_minutes=180,
       trial_timeout_minutes=10,
       max_trials=50,
       max_concurrent_trials=5,
       max_cores_per_trial=-1,
       enable_early_termination=True,
   )
   job.set_training(
       enable_stack_ensemble=True,
       enable_vote_ensemble=True,
   )
   
   returned_job = ml_client.jobs.create_or_update(job)
   ml_client.jobs.stream(returned_job.name)
   

2. Region: Japan East. Curated environment resolved for the trial child jobs:
   azureml://registries/azureml/environments/ai-ml-automl/versions/41.
3. The setup and featurization child jobs complete, then every trial child job fails after roughly 30-38 seconds with ValidationError / target: Tags, and the parent job ends as Failed.

Expected behavior
The AutoML trial child jobs should complete successfully and be reported as Completed, as they did with an identical job definition and identical curated environment version on the previous day. The AutoML runtime should not be rejected by the Run History service when writing its own reserved system tags at run completion.

Screenshots

Additional context
I diffed the serialized job definition printed by print(returned_job) for a successful run (2026-09-25) against the failing run (2026-09-26). The only differences were auto-generated values:

• name / display_name / id / Studio endpoint URL (service-assigned run name)
• mlflow.source.git.commit (unrelated commits in the same repository)
• the values of tags.dataset_name / tags.kaisai_id and training_data.path (different dataset for a different day)

Everything else was byte-identical, including featurization, limits, training, n_cross_validations, primary_metric, queue_settings, the tags key set, and the auto-injected properties (azureml.git.dirty, mlflow.source.git.*).

I also verified the following, which rules out user-supplied tags as the cause:

• The successful trial child job from 2026-09-25 shows Tags: No tags in its Overview blade, i.e. AutoML does not propagate the parent job's user tags to trial child jobs. There is therefore no path by which tags={"kaisai_id": ..., "dataset_name": ...} on the parent could collide with reserved tags on a child run.
• Neither kaisai_id nor dataset_name is in a reserved namespace (azureml.*, _azureml.*, _aml_system*).
• The curated environment version is identical (ai-ml-automl:41) between the working and failing runs, so this is not caused by a runtime image update.
• The same compute cluster was used in both cases.

The failure was reproduced twice in a row on 2026-09-26 (two separate parent jobs, same dataset), so it does not appear to be a one-off transient.

Given that the tag write originates from the service-managed AutoML runtime and the error is raised by the Run History service, this looks like a regression in the service-side reserved-tag validation (or in the tags the AutoML runtime emits at run completion) rather than something a user can work around from the SDK. Filing here for visibility since I cannot open a technical support ticket on my current support plan.