GKEStartJobOperator impersonation chain not working with Kubernetes RBAC

(2 条评论) (1 个反应) (0 位负责人)Python (16,781 个派生)batch import

area:providersgood first issuekind:bugprovider:google

仓库指标

星标: (44,809 个星标)
PR 合并指标: (平均合并 7天 18小时) (30 天内合并 834 个 PR)

描述

Apache Airflow version

Other Airflow 2 version (please specify below)

If "Other Airflow 2 version" selected, which one?

2.10.5

What happened?

I am trying to run the GKEStartJobOperator using impersonation chain from airflow. I have setup RBAC in the gke cluster to give that iam account permissions using the method shown here https://cloud.google.com/kubernetes-engine/docs/how-to/role-based-access-control#rolebinding. However, when running GKEStartJobOperator it seems to call the GKE cluster with the unqiue id of the service account in the impersonation chain rather than the email. This causes the RBAC to fail as it cant match the User. It works if i put the unqiue id directly as a user subject.

What you think should happen instead?

The follow error happens User \”<unique_id>\” cannot create resource \"jobs\" in API group \"batch\" in the namespace \"airflow-jobs\": requires one of [\"container.jobs.create\"] permission(s). I expected it to use the email in the impersonation chain rather than the unique id.

How to reproduce

set up rbac as follows

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: airflow-jobs
  name: job-creator
rules:
  - apiGroups: [ "batch" ]
    resources: [ "jobs", ]
    verbs: [ "create" , "list", "get", "update", "patch", "delete"]
  - apiGroups: [ "batch" ]
    resources: [ "jobs/status" ]
    verbs: [ "get" ]
  - apiGroups: [ "" ]
    resources: [ "pods" ]
    verbs: [ "list", "get"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: job-creator-binding
  namespace: airflow-jobs
subjects:
  - kind: User
    name: <iam-service-account>
    namespace: airflow-jobs
roleRef:
  kind: Role
  name: job-creator
  apiGroup: rbac.authorization.k8s.io

create a dag with operator configure with impersonation chain

job_task = GKEStartJobOperator(
        task_id="job_task",
        project_id=“<project-id>,
        location="europe-west1",
        cluster_name=“<gke-cluster-name>“,
        namespace="airflow-jobs",
        image="perl:5.34.0",
        cmds=["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"],
        name="test-job",
        ttl_seconds_after_finished=60,
        wait_until_job_complete=True,
        deferrable=True,
        impersonation_chain=“<iam service account>”,
    )

Operating System

linux

Versions of Apache Airflow Providers

apache-airflow==2.10.5+composer apache-airflow-providers-apache-beam==6.0.4 apache-airflow-providers-celery==3.10.0 apache-airflow-providers-cncf-kubernetes==10.1.0 apache-airflow-providers-common-compat==1.6.0 apache-airflow-providers-common-io==1.5.4 apache-airflow-providers-common-sql==1.26.0 apache-airflow-providers-dbt-cloud==4.3.3 apache-airflow-providers-fab==1.5.3 apache-airflow-providers-ftp==3.12.3 apache-airflow-providers-google==15.1.0 apache-airflow-providers-hashicorp==4.1.1 apache-airflow-providers-http==5.2.2 apache-airflow-providers-imap==3.8.3 apache-airflow-providers-mysql==6.2.2 apache-airflow-providers-openlineage==2.2.0 apache-airflow-providers-postgres==6.1.3 apache-airflow-providers-redis==4.0.2 apache-airflow-providers-sendgrid==4.0.1 apache-airflow-providers-smtp==2.0.3 apache-airflow-providers-sqlite==4.0.2 apache-airflow-providers-ssh==4.0.1 apache-airflow-providers-standard==1.0.0

Deployment

Google Cloud Composer

Deployment details

No response

Anything else?