AWS: Kubernetes  -  AWS Secrets Manager and Parameter Store integration

---

Storing access data in Kubernetes Secrets has an important drawback, because they are only available within the Kubernetes cluster itself.

To make them available to external services, we can use Hashicorp Vault and integrate it with Kubernetes using solutions such as vault-k8sor use services from AWS - Secrets Manager or Parameter Store.

Integrating AWS Secrets Manager and Parameter Store into Kubernetes will give us the ability to create a new type of resource — SecretProviderClass which we can connect to Kubernetes Pods as files or environment variables.

For this, we will need AWS Secrets and Configuration Provider (ASCP) and Kubernetes Secrets Store CSI Driver.

AWS Secrets and Configuration Provider vs Hashicorp Vault

I haven’t used Vault for a long time, but regarding the question “What to use”, it’s a choice between the “setup, configure, and manage Hashicorp Vault yourself” (installation of the Helm chart and configuration of accesses) or “use a ready-made solution from AWS” (in fact, you only need to configure IAM roles).

Also, keep in mind that using AWS services (surprise!) is paid, so if you plan to have thousands of secrets, it’s probably better to use Vault.

In addition, Vault itself provides much more opportunities, for example, the generation of temporary tokens for services, plus, as far as I remember — Kubernetes Pods can receive parameters from Vault without the need to create Kubernetes Secrets, while when using AWS Secrets and Configuration Provider and Kubernetes Secrets Store, CSI Driver for connecting variables will create Kubernetes Secrets.

However, our project already uses Secrets Manager and Parameter Store, so I don’t see a point in Vault (yet), so let’s integrate our existing secrets into an AWS Elastic Kubernetes Service cluster.

AWS Secrets Manager vs Parameter Store

You can read more about the difference between them here — AWS — Difference between Secrets Manager and Parameter Store (Systems Manager), here just briefly.

Common features:

• both use AWS KMS to encrypt data

• both are Key/Value Store

• both support versioning

Key differences:

• Cost :

  • Secrets Manager: Charges $0.40 for each secret and $0.05 for every 10,000 API requests

  • Parameter Store: for the Standard, it does not take money for storage, with higher throughput it costs $0.05 for every 10,000 API requests, with Advanced parameters — $0.05 for storage and $0.05 for every 10,000 API requests

• Secrets rotation :

  • Secrets Manager: has a built-in rotation mechanism and integrates it with services (RDS, DocumentDB, etc)

  • Parameter Store: You must implement the rotation yourself

• Cross-account Access :

  • Secrets Manager: Supports

  • Parameter Store: Not supported

• Cross-Regions Replication :

  • Secrets Manager: Supports

  • Parameter Store: Not supported

• Data limits :

  • Secrets Manager: up to 10KB per secret

  • Parameter Store: 4KB for each record (8KB for Advanced Parameters)

• Quantity limits:

  • Secrets Manager: 500,000 per region and account

  • Parameter Store: 10,000 per region and account

Installing Secrets Store CSI Driver

So, for integration, we need two services — Secrets Store CSI Driver and AWS Secrets and Configuration Provider.

First, we’ll add the Secrets Store CSI Driver.

With its help, we will be able to connect secrets/parameters from AWS as files or variables to Kubernetes Pods.

Add a Helm chart and install it with the syncSecret.enabled=true parameter to create Kubernetes Secrets and synchronize them with AWS secrets during data rotation (see Sync as Kubernetes Secret ):

$ helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
$ helm -n kube-system install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver — set syncSecret.enabled=true

Check Pods:

$ kubectl -n kube-system get pod | grep secret
csi-secrets-store-secrets-store-csi-driver-kzmcx 3/3 Running 0 31s
csi-secrets-store-secrets-store-csi-driver-t7bqc 3/3 Running 0 31s

Installing AWS Secrets and Configuration Provider

Add a repository and install the Helm chart:

$ helm repo add aws-secrets-manager https://aws.github.io/secrets-store-csi-driver-provider-aws
$ helm install -n kube-system secrets-provider-aws aws-secrets-manager/secrets-store-csi-driver-provider-aws

Check Pods:

$ kubectl -n kube-system get pod | grep secret
csi-secrets-store-secrets-store-csi-driver-kzmcx 3/3 Running 0 9m
csi-secrets-store-secrets-store-csi-driver-t7bqc 3/3 Running 0 9m
secrets-provider-aws-secrets-store-csi-driver-provider-awskq5g8 1/1 Running 0 23s
secrets-provider-aws-secrets-store-csi-driver-provider-awsksq9d 1/1 Running 0 23s

And look at the CSIDriver:

$ kubectl get csidriver
NAME ATTACHREQUIRED PODINFOONMOUNT STORAGECAPACITY TOKENREQUESTS REQUIRESREPUBLISH MODES AGE
ebs.csi.aws.com true false false <unset> false Persistent 46h
efs.csi.aws.com false false false <unset> false Persistent 4d
secrets-store.csi.k8s.io false true false <unset> false Ephemeral 10m

Next, we will configure IAM for IRSA.

IAM Policy and IAM Role for ServiceAccount

In order for Kubernetes Pods to be able to access AWS SecretManager and Parameter Store, we will use IRSA — create a ServiceAacount that will use an IAM Role with an IAM Policy that will have permissions to call Secrets Manager and Parameter Store (see AWS: EKS, OpenID Connect, and ServiceAccounts).

Describe an IAM policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "secretsmanager:DescribeSecret",
                "secretsmanager:GetSecretValue",
                "ssm:DescribeParameters",
                "ssm:GetParameter",
                "ssm:GetParameters",
                "ssm:GetParametersByPath"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

Create it in the IAM:

$ aws iam create-policy --policy-name ascp-iam-policy --policy-document file://ascp-policy.json
{
“Policy”: {
“PolicyName”: “ascp-policy”,
“PolicyId”: “ANPAXFIUAIGSBPFEDKZZT”,
“Arn”: “arn:aws:iam::492***148:policy/ascp-iam-policy”,
…

Describe the Trust policy for the IAM Role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Principal": {
        "Federated": "arn:aws:iam::492 ***148:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/2DC*** 124"
      },
      "Condition": {
        "StringEquals": {
          "oidc.eks.us-east-1.amazonaws.com/id/2DC***124:aud": "sts.amazonaws.com",
          "oidc.eks.us-east-1.amazonaws.com/id/2DC***124:sub": "system:serviceaccount:default:ascp-test-serviceaccount"
        }
      }
    }
  ]
}

Create the Role itself with this trust policy:

$ aws iam create-role --role-name ascp-iam-role --assume-role-policy-document file://ascp-trust.json
{
“Role”: {
“Path”: “/”,
“RoleName”: “ascp-iam-role”,
“RoleId”: “AROAXFIUAIGSLDCB3L4AR”,
“Arn”: “arn:aws:iam::492***148:role/ascp-iam-role”,

…

Attach the ascp-iam-policy policy to that role:

$ aws iam attach-role-policy --role-name ascp-iam-role --policy-arn=arn:aws:iam::492***148:policy/ascp-iam-policy

Now we can create a SecretProviderClass and a Pod that will use it.

Create a SecretProviderClass

Let’s add a SecretProviderClass, which will receive a string from the Secrets Manager and a string from the Parameter Store, and then we will connect them to a Kubernetes Pod.

Create a secret in Secrets Manager:

$ aws secretsmanager create-secret --name ascp-secret-test-string --secret-string "secretLine"
{
“ARN”: “arn:aws:secretsmanager:us-east-1:492***148:secret:ascp-secret-test-string-DNweNg”,
“Name”: “ascp-secret-test-string”,
“VersionId”: “9d4f490d-edcc-4ee0-b43d-5b4e25fa271b”
}

Create a record in the Parameter Store:

$ aws ssm put-parameter --name ascp-ssm-test-param --value "paramLine" --type "String"
{
“Version”: 1,
“Tier”: “Standard”
}

Next, describe a SecretProviderClass with two objects - in the parameters.objects.objectName set a name of the object in the Secrets Manager or Parameter Store and in the objectType set where we get this object from:

--- 
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata: 
  name: aspc-test-secret-class
spec:         
  provider: aws
  parameters:
    objects: |
        - objectName: "ascp-test-string"
          objectType: "secretsmanager"
        - objectName: "ascp-ssm-test-param"
          objectType: "ssmparameter"

Let’s go to the Pod.

Attaching a SecretProviderClass to a Pod as a file

Add a ServiceAccount with the IAM role we created earlier, and a Pod with this ServiceAccount:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ascp-test-serviceaccount
  namespace:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::492***148:role/ascp-iam-role
---
apiVersion: v1
kind: Pod
metadata:
  name: ascp-test-pod
spec:
  containers:
    - name: my-aws-cli
      image: amazon/aws-cli:latest
      command: ['sleep', '36000']
      volumeMounts:
      - name: ascp-test-secret-volume
        mountPath: /mnt/ascp-secret
        readOnly: true
  restartPolicy: Never
  serviceAccountName: ascp-test-serviceaccount
  volumes:
  - name: ascp-test-secret-volume
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: aspc-test-secret-class

Deploy it:

$ kubectl apply -f ascp-test.yaml
serviceaccount/ascp-test-serviceaccount created
secretproviderclass.secrets-store.csi.x-k8s.io/aspc-test-secret-class created
pod/ascp-test-pod created

Check the Pod:

$ kk describe pod ascp-test-pod
…
Mounts:
/mnt/ascp-secret from ascp-test-secret-volume (ro)
…
Volumes:
…
ascp-test-secret-volume:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: secrets-store.csi.k8s.io
FSType:
ReadOnly: true
VolumeAttributes: secretProviderClass=aspc-test-secret-class
…

And a content of the /mnt/ascp-secret directory:

$ kk exec -ti ascp-test-pod -- ls -l /mnt/ascp-secret
total 8
-rw-r — r — 1 root root 10 Jul 17 09:32 ascp-secret-test-string
-rw-r — r — 1 root root 9 Jul 17 09:32 ascp-ssm-test-param

And files there:

$ kk exec -ti ascp-test-pod -- cat /mnt/ascp-secret/ascp-secret-test-string
secretLine
$ kk exec -ti ascp-test-pod -- cat /mnt/ascp-secret/ascp-ssm-test-param
paramLine

Attaching a SecretProviderClass to a Pod as an environment variable

Mounting secrets as a file may be a good solution for some .env files, but what about environment variables? For example to pass a DB_PASSWORD.

To do so, we can add a secretObjects to the SecretProviderClass - then Kubernetes Secrets Store CSI Driver will create a Kubernetes Secret, which we can connect to the Pod:

---
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aspc-test-secret-class
spec:
  provider: aws
  parameters:
    objects: |
        - objectName: "ascp-secret-test-string"
          objectType: "secretsmanager"
        - objectName: "ascp-ssm-test-param"
          objectType: "ssmparameter"
  secretObjects:
    - secretName: aspc-test-kube-secret
      type: Opaque
      data:
        - objectName: ascp-secret-test-string
          key: kube-secret-key

Here:

• secretObjects.secretName: a name of the Kubernetes Secret to be created

• secretObjects.secretName.data.objectName: must be the same as parameters.objects.objectName

• secretObjects.secretName.data.key: a key for the Kubernetes Secret - data.kube-secret-key

Deploy, and check Kubernetes Secret:

$ kk get secret aspc-test-kube-secret -o yaml
apiVersion: v1
data:
kube-secret-key: c2VjcmV0TGluZQ==
…

And a value of the kube-secret-key:

$ echo c2VjcmV0TGluZQ== | base64 -d
secretLine

Now, let’s connect it to the Pod — add spec.containers.env with the valueFrom.secretKeyRef:

---
apiVersion: v1
kind: Pod
metadata:
  name: ascp-test-pod
spec:
  containers:
    - name: my-aws-cli
      image: amazon/aws-cli:latest
      command: ['sleep', '36000']
      env:
      - name: SECRET
        valueFrom:
          secretKeyRef:
            name: aspc-test-kube-secret
            key: kube-secret-key
      volumeMounts:
      - name: ascp-test-secret-volume
        mountPath: /mnt/ascp-secret
        readOnly: true
  restartPolicy: Never
  serviceAccountName: ascp-test-serviceaccount
  volumes:
  - name: ascp-test-secret-volume
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: aspc-test-secret-class

Deploy, and check:

$ kk exec -ti ascp-test-pod -- printenv | grep SECRET
SECRET=secretLine

Or:

$ kk exec -ti ascp-test-pod -- bash
bash-4.2# echo $SECRET
secretLine

During this, we have to attach the volumes and volumeMounts, as it was done for mounting Secrets as a file.

Creating a SecretProviderClass from JSON

If data in Secrets Manager and Parameter Store is stored in JSON, then we should use the jmesPath in our SecretProviderClass.

Let’s create another secret in Secrets Manager from JSON:

$ aws secretsmanager create-secret --name ascp-secret-test-json --secret-string '{"username":"admin", "password":"foobar"}'
{
“ARN”: “arn:aws:secretsmanager:us-east-1:492***148:secret:ascp-secret-test-json-iOtcBf”,
“Name”: “ascp-secret-test-json”,
“VersionId”: “32666608–6416–46cf-8b93-cf090eef1bc5”
}

Check it:

Update our SecretProviderClass:

---
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aspc-test-secret-class
spec:
  provider: aws
  parameters:
    objects: |
        - objectName: "ascp-secret-test-string"
          objectType: "secretsmanager"
        - objectName: "ascp-ssm-test-param"
          objectType: "ssmparameter"
        - objectName: "ascp-secret-test-json"
          objectType: "secretsmanager"
          jmesPath:
              - path: "username"
                objectAlias: "ascp-test-username"
              - path: "password"
                objectAlias: "ascp-test-password"
  secretObjects:
    - secretName: aspc-test-kube-secret
      type: Opaque
      data:
        - objectName: ascp-secret-test-string
          key: kube-secret-key
    - secretName: aspc-test-kube-secret-json
      type: Opaque
      data:
        - objectName: ascp-test-username
          key: kube-secret-user
        - objectName: ascp-test-password
          key: kube-secret-pass

Here:

• in the parameters.objects.objectName: "ascp-secret-test-json" use jmesPath, which parses our secret and receives the values ​​of two fields - username and password, for which it creates two objectAlias- ascp-test-username and ascp-test-password

• in the secretObjects.secretName: aspc-test-kube-secret-json we add a data with two objectName, in which we use objectAlias from the parameters

Update our Kubernetes Pod — add two secretKeyRef with the kube-secret-user and kube-secret-user keys from the aspc-test-kube-secret-json Secret:

---
apiVersion: v1
kind: Pod
metadata:
  name: ascp-test-pod
spec:
  containers:
    - name: my-aws-cli
      image: amazon/aws-cli:latest
      command: ['sleep', '36000']
      env:
      - name: SECRET
        valueFrom:
          secretKeyRef:
            name: aspc-test-kube-secret
            key: kube-secret-key
      - name: USER
        valueFrom:
          secretKeyRef:
            name: aspc-test-kube-secret-json
            key: kube-secret-user
      - name: PASS
        valueFrom:
          secretKeyRef:
            name: aspc-test-kube-secret-json
            key: kube-secret-pass
      volumeMounts:
      - name: ascp-test-secret-volume
        mountPath: /mnt/ascp-secret
        readOnly: true
  restartPolicy: Never
  serviceAccountName: ascp-test-serviceaccount
  volumes:
  - name: ascp-test-secret-volume
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: aspc-test-secret-class

Deploy and check Kubernetes Secrets:

$ kk get secret
NAME TYPE DATA AGE
aspc-test-kube-secret Opaque 1 2s
aspc-test-kube-secret-json Opaque 2 2s

And a value from the aspc-test-kube-secret-json:

$ kk get secret aspc-test-kube-secret-json -o yaml
apiVersion: v1
data:
kube-secret-pass: Zm9vYmFy
kube-secret-user: YWRtaW4=
…

And environment variables in the Pod:

$ kk exec -ti ascp-test-pod --printenv | grep 'SECRET\|USER\|PASS'
SECRET=secretLine
USER=admin
PASS=foobar

Done.

Originally published at RTFM: Linux, DevOps, and system administration.

---