Kubernetes: ServiceAccount with AWS IAM Role for Kubernetes Pod

---

We have Grafana Loki for logs and need to connect an AWS IAM Role with AWS IAM Policy, which gives access to an AWS S3 bucket where Loki’s chunks and indexes will be stored.

IAM roles for Kubernetes Pods will work in the same way as IAM roles to EC2 instances: a process inside a Pod makes a request to the AWS API, and the AWS SDK or AWS CLI that makes the request also makes the AssumeRole request with the IAM Role to be used (see AWS: rotation of user IAM keys, EC2 IAM Roles and Jenkins ).

To check how it will work in Kubernetes, let’s create a test IAM Role, ServiceAccount with the annotation of this role, and a Pod with this ServiceAccount that will use this Role.

Looking ahead — it works in Loki in some alternative reality, that is, even when you connect an already tested ServiceAccount to the Loki Pod — it crashes with errors. But eventually, I made it works even there (about setting up Loki itself with AWS S3 a little later in a separate post).

Documentation — IAM roles for service accounts.

IAM OIDC identity provider verification

In our case, the EKS cluster is deployed using the Terraform aws_eks_cluster module and the OpenID Connect (OIDC) provider should already be configured.

Go to the cluster page, and find the OpenID Connect provider URL :

Or from the terminal:

$ aws -rofile development --egion us-west-2 eks describe-cluster --name dev_data_services --query “cluster.identity.oidc.issuer” --output text
https://oidc.eks.us-west-2.amazonaws.com/id/537***A10

Copy the URL without https://, and find it in the IAM > Identity providers :

Okay, that’s it.

Creating a Kubernetes ServiceAccount with an AWS IAM role

First, let’s create an IAM policy that gives permissions to the AWS S3 bucket, then an IAM Role with a TrustedPolicy that allows you to perform the AssumeRole, using the OIDC identity provider (IDP) of the cluster.

AWS IAM policy

Create a test policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::development-dev-loki-object-store",
                "arn:aws:s3:::development-dev-loki-object-store/*"
            ]
        }
    ]
}

Add it to the AWS IAM:

$ aws — profile development iam create-policy --policy-name test-iam-sa-pod-policy --policy-document file://test-iam-sa-pod-policy.json

Move on to the role.

AWS IAM role and TrustedPolicy

Find an ARN of your Identity Provider:

The OIDC identity provider URL has already been found earlier:

$ aws --profile development --region us-west-2 eks describe-cluster --name dev_data_services --query “cluster.identity.oidc.issuer” --output text
https://oidc.eks.us-west-2.amazonaws.com/id/537***A10

Creating an IAM Role with the AWS CLI

Create a file with the TrustedPolicy, in its Principal filed specify the ARN of the IDP, and in the Condition - the OIDC URL of the cluster, which will be allowed to perform requests to the sts.amazonaws.com :

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::638 ***021:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/537*** A10"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.us-west-2.amazonaws.com/id/537***A10:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}

Create a role with this TrustedPolicy:

$ aws --profile development iam create-role --role-name test-iam-sa-pod-role --assume-role-policy-document file://test-iam-sa-role-trusted-policy.json

Connect the IAM Policy which we created above with permissions to the S3 to this IAM Role:

$ aws --profile development iam attach-role-policy --role-name test-iam-sa-pod-role --policy-arn=arn:aws:iam::638***021:policy/test-iam-sa-pod-policy

Create an IAM Role from the AWS Console

Or we can also do it through the Amazon Console page  -  choose the type of Web identity, Identity Provider of the cluster, and Audience :

Add the IAM Policy for S3:

Save it:

Copy the ARN of the role:

Creating a Kubernetes ServiceAccount

Create a ServiceAccount manifest, in its annotations set the ARN of the created role:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: test-iam-sa-pod-service-account
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::638***021:role/test-iam-sa-pod-role

And let’s go to the Kubernetes Pod.

Running a Kubernetes Pod with a ServiceAccount

Create a manifest of the Pod with the serviceAccountName of the ServiceAccount, so the complete file will look like this:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: test-iam-sa-pod-service-account
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::638***021:role/test-iam-sa-pod-role
---
apiVersion: v1
kind: Pod
metadata:
  name: test-iam-sa-pod
  labels:
    app: test-iam-sa-app
spec:
  containers:
    - name: test-iam-sa
      image: amazon/aws-cli
      command: ["/bin/bash", "-c", "--"]
      args: ["while true; do sleep 30; done;"]
  serviceAccountName: test-iam-sa-pod-service-account

In the Pod, we will use a Docker image with AWS CLI with the sleep command, so it will stay Running after startup.

Deploy the ServiceAccount and Pod:

$ kubectl apply -f test-iam-sa-pod.yaml
serviceaccount/test-iam-sa-pod-service-account created
pod/test-iam-sa-pod created

Go into it:

$ kubectl exec -ti test-iam-sa-pod — bash

And check access to the basket:

bash-4.2# aws s3 ls development-dev-loki-object-store
PRE test/

Done.

Originally published at RTFM: Linux, DevOps, and system administration.

---