GKEStartJobOperator impersonation chain not working with Kubernetes RBAC

(2 則留言) (1 個反應) (0 位負責人)Python (16,781 個分叉)batch import

area:providersgood first issuekind:bugprovider:google

倉庫指標

星標: (44,809 顆星)
PR 合併指標: (平均合併 7天 18小時) (30 天內合併 834 個 PR)

描述

Apache Airflow version

Other Airflow 2 version (please specify below)

If "Other Airflow 2 version" selected, which one?

2.10.5

What happened?

I am trying to run the GKEStartJobOperator using impersonation chain from airflow. I have setup RBAC in the gke cluster to give that iam account permissions using the method shown here https://cloud.google.com/kubernetes-engine/docs/how-to/role-based-access-control#rolebinding. However, when running GKEStartJobOperator it seems to call the GKE cluster with the unqiue id of the service account in the impersonation chain rather than the email. This causes the RBAC to fail as it cant match the User. It works if i put the unqiue id directly as a user subject.

What you think should happen instead?

The follow error happens User \”<unique_id>\” cannot create resource \"jobs\" in API group \"batch\" in the namespace \"airflow-jobs\": requires one of [\"container.jobs.create\"] permission(s). I expected it to use the email in the impersonation chain rather than the unique id.

How to reproduce

set up rbac as follows

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: airflow-jobs
  name: job-creator
rules:
  - apiGroups: [ "batch" ]
    resources: [ "jobs", ]
    verbs: [ "create" , "list", "get", "update", "patch", "delete"]
  - apiGroups: [ "batch" ]
    resources: [ "jobs/status" ]
    verbs: [ "get" ]
  - apiGroups: [ "" ]
    resources: [ "pods" ]
    verbs: [ "list", "get"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: job-creator-binding
  namespace: airflow-jobs
subjects:
  - kind: User
    name: <iam-service-account>
    namespace: airflow-jobs
roleRef:
  kind: Role
  name: job-creator
  apiGroup: rbac.authorization.k8s.io

create a dag with operator configure with impersonation chain

job_task = GKEStartJobOperator(
        task_id="job_task",
        project_id=“<project-id>,
        location="europe-west1",
        cluster_name=“<gke-cluster-name>“,
        namespace="airflow-jobs",
        image="perl:5.34.0",
        cmds=["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"],
        name="test-job",
        ttl_seconds_after_finished=60,
        wait_until_job_complete=True,
        deferrable=True,
        impersonation_chain=“<iam service account>”,
    )

Operating System

linux

Versions of Apache Airflow Providers

apache-airflow==2.10.5+composer apache-airflow-providers-apache-beam==6.0.4 apache-airflow-providers-celery==3.10.0 apache-airflow-providers-cncf-kubernetes==10.1.0 apache-airflow-providers-common-compat==1.6.0 apache-airflow-providers-common-io==1.5.4 apache-airflow-providers-common-sql==1.26.0 apache-airflow-providers-dbt-cloud==4.3.3 apache-airflow-providers-fab==1.5.3 apache-airflow-providers-ftp==3.12.3 apache-airflow-providers-google==15.1.0 apache-airflow-providers-hashicorp==4.1.1 apache-airflow-providers-http==5.2.2 apache-airflow-providers-imap==3.8.3 apache-airflow-providers-mysql==6.2.2 apache-airflow-providers-openlineage==2.2.0 apache-airflow-providers-postgres==6.1.3 apache-airflow-providers-redis==4.0.2 apache-airflow-providers-sendgrid==4.0.1 apache-airflow-providers-smtp==2.0.3 apache-airflow-providers-sqlite==4.0.2 apache-airflow-providers-ssh==4.0.1 apache-airflow-providers-standard==1.0.0

Deployment

Google Cloud Composer

Deployment details

No response

Anything else?