AWS: CDK — an overview and Python examples

The AWS Cloud Development Kit (AWS CDK) allows you to describe an infrastructure using the programming languages TypeScript, JavaScript, Python, Java, C#, or Go.

Under the hood, CDK creates a CloudFormation stack with the resources described in your code.

The answer to the question “Our CDK, when is Terraform?” can be found here — 4 ultimate reasons to prefer AWS CDK over Terraform.

But since I haven’t used CDK yet, I won’t say anything about the advantages and disadvantages.

The only thing that you can pay attention to now is that, first of all, there are no state files like in Terraform, which, of course, are useful, but add a little pain to management. The second thing is that CloudFormation itself has its own shortcomings and “this is not a bug, but a feature”, but we have the opportunity to see all resources in the AWS Console web interface.

In general, I came to AWS CDK because it is already being used on the new project, so before taking Terraform to the project, need to check with what is already there.

UPD: Well, no, I’ll say it anyway. It looks like it’s not bad and interesting, because “Yahoo, finally Python!”, but:

after all, the HCL code looks much more concise and understandable
there are many more examples and banal Google search results for Terraform, which means that the speed of finishing an IaC task is faster
well… I got to the point where I had to create an SES domain, and… And nothing. Very unexpectedly, nothing really found in the standard Constructs, and the only more or less construct from Construct Hub had examples only for TypeScript, even in PyDoc. Such a pleasure, to be honest

Besides the CDK for AWS itself, there are also cdk8s for Kubernetes and CDKTF for Terraform.

Okay, let’s go to see the AWS CDK.

Key concepts

You can start with the AWS documentation Getting started with the AWS CDK, or watch a good 20-minute video tutorial Getting Started with AWS CDK and Python.

So, the main concepts we will use when working with AWS CDK:

App: the App is a kind of “container” where we describe our application, and can contain one or more Stacks (which will then be formed in CloudFormation Stacks). See Apps.
Stack: CloudFormation Stack or change set will be formed from a Stack. In the Stack itself, at the code level, we describe exactly which resources in this stack will be created, and add describe resources using Constructs. See Stacks.
Construct The basic “building blocks” in the CDK that describe the components that need to be built in AWS. See Construct.

Let’s dwell on Construct in a little more detail, because they are divided into three main groups:

AWS CloudFormation-only or L1 (“layer 1”): Here, we have resources that are described and supported by CloudFormation itself, and all such resources have names prefixed with Cfn, for example, for AWS S3 buckets there is the CfnBucket. All these resources are in the module aws-cdk-lib.
Curated or L2: These constructs were developed by the AWS CDK team to simplify infrastructure management. Typically, these will include the L1 resources with some default values and security policies. Resources are in the aws-cdk-lib and ready for use in production, but if a resource is a separate module, then it is either still under development or experimental.
Patterns or L3: Patterns include several resources that allow you to build the entire architecture for a specific use case. As with L2 resources, production-ready modules are included in the module aws-cdk-lib, and those that are under development are in separate modules.

In addition to the AWS Construct Library, there is also the Construct Hub, where you can find modules from AWS partners.

Installing the AWS CDK

To work with the AWS CDK, even if you will write in Python, you need Node.js, since all programming languages on the CDK will work through the Node.js backend.

To work with CDK, we have a CLI through which we can create new Apps, generate CloudFormation templates, perform a diff between our code and existing CloudFormation stacks, and much more.

Install the CDK itself — backend and CLI:

$ npm install -g aws-cdk
added 1 package, and audited 2 packages in 1s

Check the CLI:

$ cdk --help
Usage: cdk -a <cdk-app> COMMAND
Commands:
cdk list [STACKS..] Lists all stacks in the app [aliases: ls]
cdk synthesize [STACKS..] Synthesizes and prints the CloudFormation
template for this stack [aliases: synth]
cdk bootstrap [ENVIRONMENTS..] Deploys the CDK toolkit stack into an AWS
environment
cdk deploy [STACKS..] Deploys the stack(s) named STACKS into your
AWS account
…

For the sake of curiosity — where does the file cdk itself leads:

$ which cdk
/home/setevoy/.nvm/versions/node/v16.18.0/bin/cdk

Yup, Node.js.

Authentication with AWS

Documentation — Authentication and access:

AWS SSO: Using an SSO session via the AWS CLI config ( ~/.aws/config), see IAM Identity Center authentication
AWS EC2 Instance IAM Role: if the CDK code will run within the AWS environment, for example from EC2, then IAM roles for Amazon EC2 can be connected to the instance
AWS IAM User Access/Secret keys: well, the usual access keys for AWS CLI with a profile in files ~/.aws/config and ~/.aws/credentials

Creating a project

`cdk init`

With the cdk init we can generate a template of files and directories.

To get help on a specific command, for example for init - add --help after it, that is cdk init --help:

Among the interesting options here can be the following:

--verbose: more detailed output
--debug: even more detailed
--role-arn: use IAM Role
--language: the programming language that will be used when creating a project
--list: get a list of available templates

Let’s try list:

$ cdk init — list
Available templates:
* app: Template for a CDK Application
└─ cdk init app — language=[csharp|fsharp|go|java|javascript|python|typescript]
* lib: Template for a CDK Construct Library
└─ cdk init lib — language=typescript
* sample-app: Example CDK Application with some constructs
└─ cdk init sample-app — language=[csharp|fsharp|go|java|javascript|python|typescript]

Here, we can create a template for the App, a library for the Construct Library, or create a sample-app, that is an example App with some Constructcs already included.

Create the directory of our project:

$ mkdir cdk-example && cd cdk-example

Run init sample-app:

$ cdk init sample-app — language=python
Applying project template sample-app for python
…
Initializing a new git repository…
hint: Using ‘master’ as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: git config — global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of ‘master’ are ‘main’, ‘trunk’ and
hint: ‘development’. The just-created branch can be renamed via this command:
hint:
hint: git branch -m <name>
Please run ‘python3 -m venv .venv’!
Executing Creating virtualenv…
✅ All done!

Let’s look at the structure of the files and directories of the project:

$ tree .
.
| — README.md
| — app.py
| — cdk.json
| — cdk_example
| | — __init__.py
| ` — cdk_example_stack.py
| — requirements-dev.txt
| — requirements.txt
| — source.bat
` — tests
| — __init__.py
` — unit
| — __init__.py
` — test_cdk_example_stack.py
4 directories, 11 files

Python virtualenv and installing AWS CDK modules

source.bat – a script for Windows to create a Python virtualenv that should call the file .venv\Scripts\activate.bat:

$ tail -1 source.bat
.venv\Scripts\activate.bat

But since I’m doing it on Linux, I haven’t the .venv\Scripts directory at all, instead, I have a set of scripts in .venv/bin/ (well, it also looks somehow... AWS CDK seems like a serious project, but such a small thing as scripts were done with some... carelessness? ):

$ ls -1a .venv/bin/
.
..
Activate.ps1
activate
activate.csh
activate.fish
pip
pip3
pip3.11
python
python3
python3.11

For Linux, we will use the .venv/bin/activate script, which is a set of shell commands for creating and setting environment variables:

$ . .venv/bin/activate
(.venv)

To check that we are really in a virtualenv, you can check the value of the variable $VIRTUAL_ENV, which has the path to the directory of the current virtual environment where the installed libraries will be:

$ echo $VIRTUAL_ENV
/home/setevoy/Scripts/AWS_CDK/.venv

Next, install dependencies — modules aws-cdk-lib and constructs:

$ pip install -r requirements.txt
…
Collecting aws-cdk-lib==2.78.0
Using cached aws_cdk_lib-2.78.0-py3-none-any.whl (41.0 MB)
Collecting constructs<11.0.0,>=10.0.0
Using cached constructs-10.2.18-py3-none-any.whl (58 kB)
…

The app.py file

Let’s check the content of the app.py, which is the base of our project, because it is from where CDK will start its work:

#!/usr/bin/env python3

import aws_cdk as cdk

from cdk_example.cdk_example_stack import CdkExampleStack

app = cdk.App()
CdkExampleStack(app, "cdk-example")

app.synth()

The first thing we do import aws_cdk as cdk to import the module aws_cdk dir, which is in the directory .venv:

$ find . -name aws_cdk
./.venv/lib/python3.11/site-packages/aws_cdk

Next, with the from cdk_example.cdk_example_stack import CdkExampleStack we import the CdkExampleStack(Stack) class from the cdk_example_stack.py module:

from constructs import Construct
from aws_cdk import (
    Duration,
    Stack,
    aws_iam as iam,
    aws_sqs as sqs,
    aws_sns as sns,
    aws_sns_subscriptions as subs,
)

class CdkExampleStack(Stack):

    def __init__ (self, scope: Construct, construct_id: str, **kwargs) -> None:
        super(). __init__ (scope, construct_id, **kwargs)

        queue = sqs.Queue(
            self, "CdkExampleQueue",
            visibility_timeout=Duration.seconds(300),
        )

        topic = sns.Topic(
            self, "CdkExampleTopic"
        )

        topic.add_subscription(subs.SqsSubscription(queue))

And in the cdk_example_stack.py we can already see the resources that will be created - SQS, and SNS with a Subscription.

You can view the resources in the PyDoc:

>>> import aws_cdk as cdk
>>> help (cdk.App())

Where the class App will be described:

Help on App in module aws_cdk object:
class App(Stage)
| App(*args: Any, **kwargs) -> Any
|
| A construct which represents an entire CDK app. This construct is normally the root of the construct tree.
|
| You would normally define an ``App`` instance in your program’s entrypoint,
| then define constructs where the app is used as the parent scope.
…

OK, let’s move on.

The cdk list (or cdk ls) will return us a list of resources in the current directory/project:

$ cdk ls
cdk-example

Next, we can try with the cdk synth, which will generate a CloudFormation template, which will be used during deployment:

$ cdk synth
Resources:
CdkExampleQueue7618E31B:
Type: AWS::SQS::Queue
Properties:
VisibilityTimeout: 300
UpdateReplacePolicy: Delete
DeletionPolicy: Delete
Metadata:
aws:cdk:path: cdk-example/CdkExampleQueue/Resource
CdkExampleQueuePolicy839151B5:
Type: AWS::SQS::QueuePolicy
…

Working with the AWS CDK

AWS Account CDK Bootstrap

Before deploying the project, we need to configure our AWS account (or region) for AWS CDK. For this, we use the cdk bootstrap command, which will create a CloudFormation stack with the resources necessary for the CDK to work - an S3 bucket, roles and policies in IAM, an ECR repository, and records in the AWS Systems Manager Parameter Store. See bootstrapping.

Let’s start:

$ cdk -v bootstrap
…
⏳ Bootstrapping environment aws://264***286/eu-central-1…
…
CDKToolkit | 0/12 | 1:43:06 PM | REVIEW_IN_PROGRESS | AWS::CloudFormation::Stack | CDKToolkit User Initiated
CDKToolkit | 0/12 | 1:43:11 PM | CREATE_IN_PROGRESS | AWS::CloudFormation::Stack | CDKToolkit User Initiated
CDKToolkit | 0/12 | 1:43:16 PM | CREATE_IN_PROGRESS | AWS::IAM::Role | FilePublishingRole
CDKToolkit | 0/12 | 1:43:16 PM | CREATE_IN_PROGRESS | AWS::IAM::Role | LookupRole
CDKToolkit | 0/12 | 1:43:16 PM | CREATE_IN_PROGRESS | AWS::SSM::Parameter | CdkBootstrapVersion
CDKToolkit | 0/12 | 1:43:16 PM | CREATE_IN_PROGRESS | AWS::IAM::Role | CloudFormationExecutionRole
CDKToolkit | 0/12 | 1:43:16 PM | CREATE_IN_PROGRESS | AWS::ECR::Repository | ContainerAssetsRepository
CDKToolkit | 0/12 | 1:43:16 PM | CREATE_IN_PROGRESS | AWS::IAM::Role | ImagePublishingRole
CDKToolkit | 0/12 | 1:43:16 PM | CREATE_IN_PROGRESS | AWS::S3::Bucket | StagingBucket
…
CDKToolkit | 11/12 | 1:44:02 PM | CREATE_COMPLETE | AWS::IAM::Role | DeploymentActionRole
CDKToolkit | 12/12 | 1:44:04 PM | CREATE_COMPLETE | AWS::CloudFormation::Stack | CDKToolkit
[13:44:09] Stack CDKToolkit has completed updating
✅ Environment aws://264***286/eu-central-1 bootstrapped.

Check the S3 bucket:

$ aws s3 ls
2023–05–10 13:43:42 cdk-hnb659fds-assets-264***286-eu-central-1

`cdk deploy`

And now we can execute the cdk deploy, which will create a CloudFormation Stack with our SQS/SNS/Subscription:

$ cdk deploy
…
cdk-example: assets built
This deployment will make potentially sensitive changes according to your current security approval level ( — require-approval broadening).
Please confirm you intend to make the following modifications:
IAM Statement Changes
┌───┬────────────────────────┬────────┬─────────────────┬───────────────────────────┬────────────────────────────────────────────────────────┐
│ │ Resource │ Effect │ Action │ Principal │ Condition │
├───┼────────────────────────┼────────┼─────────────────┼───────────────────────────┼────────────────────────────────────────────────────────┤
│ + │ ${CdkExampleQueue.Arn} │ Allow │ sqs:SendMessage │ Service:sns.amazonaws.com │ “ArnEquals”: { │
│ │ │ │ │ │ “aws:SourceArn”: “${CdkExampleTopic}” │
│ │ │ │ │ │ } │
└───┴────────────────────────┴────────┴─────────────────┴───────────────────────────┴────────────────────────────────────────────────────────┘
(NOTE: There may be security-related changes not in this list. See https://github.com/aws/aws-cdk/issues/1299)
Do you wish to deploy these changes (y/n)? y
…

Respond Y:

…
Do you wish to deploy these changes (y/n)? y
cdk-example: deploying… [1/1]
[0%] start: Publishing 20e979ce16c7aba5e874330247d9054b841ea313261b523b47a50fc4cd1d6662:current_account-current_region
[100%] success: Published 20e979ce16c7aba5e874330247d9054b841ea313261b523b47a50fc4cd1d6662:current_account-current_region
cdk-example: creating CloudFormation changeset…
[███████████████████▎······································] (2/6)
1:48:50 PM | CREATE_IN_PROGRESS | AWS::CloudFormation::Stack | cdk-example
1:48:54 PM | CREATE_IN_PROGRESS | AWS::SQS::Queue | CdkExampleQueue
…

CDK will locally generate a template file cdk.out/cdk-example.template.json for the CloudFormation, and then will load it into the CDK bucket that was created at runtime cdk bootstrap:

$ aws s3 ls cdk-hnb659fds-assets-264***286-eu-central-1
2023–05–10 13:48:45 5750 20e979ce16c7aba5e874330247d9054b841ea313261b523b47a50fc4cd1d6662.json

Check the CloudFormation stack:

$ aws cloudformation list-stacks
{
“StackSummaries”: [
{
“StackId”: “arn:aws:cloudformation:eu-central-1:264***286:stack/cdk-example/3c4e4db0-ef20–11ed-9672–0a9a3483d50e”,
“StackName”: “cdk-example”,
“CreationTime”: “2023–05–10T10:48:45.099000+00:00”,
…

Or from the AWS Console:

Meanwhile, the deployment is complete:

…
✅ cdk-example
✨ Deployment time: 91.52s
Stack ARN:
arn:aws:cloudformation:eu-central-1:264***286:stack/cdk-example/3c4e4db0-ef20–11ed-9672–0a9a3483d50e
✨ Total time: 97.99s

`cdk diff`

Ok, we saw how it all works on everything ready, now let’s try to create something of our own, for example — an S3 basket.

Add aws_s3 as s3 to imports, remove sns/sqs, I am, and Duration.

Also, remove the SQS and SNS resources from the class CdkExampleStack and add the creation of the basket, can take an example from the PyPI documentation:

from constructs import Construct
from aws_cdk import (
    Stack,
    aws_s3 as s3
)

class CdkExampleStack(Stack):

    def __init__ (self, scope: Construct, construct_id: str, **kwargs) -> None:
        super(). __init__ (scope, construct_id, **kwargs)

        bucket = s3.Bucket(self, "MyEncryptedBucket",
            encryption=s3.BucketEncryption.KMS
        )

And let’s see what will be returned to us with cdk diff:

Red - is what will be deleted, green + is what will be created (it reminded me terraform plan a lot).

Okay, start the deployment:

$ cdk deploy
✨ Synthesis time: 6.38s
cdk-example: building assets…
…
Do you wish to deploy these changes (y/n)? y
cdk-example: deploying… [1/1]
[0%] start: Publishing 5bb8c7fc8643769d69d5eb9712af36c955f9b509cc05d26740d035e9d7225a16:current_account-current_region
[100%] success: Published 5bb8c7fc8643769d69d5eb9712af36c955f9b509cc05d26740d035e9d7225a16:current_account-current_region
cdk-example: creating CloudFormation changeset…
[████████████▉·············································] (2/9)
11:29:47 AM | UPDATE_IN_PROGRESS | AWS::CloudFormation::Stack | cdk-example
11:31:54 AM | CREATE_IN_PROGRESS | AWS::S3::Bucket |
MyEncryptedBucket
…

Let’s see how it looks in the UI:

Done.

Let’s clean up after ourselves — delete the stack.

`cdk destroy`

Check the stack name with the list:

$ cdk ls
cdk-example

And run cdk destroy with the name of the stack to completely remove it:

$ cdk destroy cdk-example
Are you sure you want to delete: cdk-example (y/n)? y
cdk-example: destroying… [1/1]
✅ cdk-example: destroyed

Check the Console

But the S3 bucket and KMS key were not deleted. Why?

AWS CDK RemovalPolicy

Because aws_cdk.core has the RemovalPolicy, which by default has the Retain value.

This policy controls what happens to resources that have been removed from CloudFormation control:

if a resource is deleted from the template
a resource needs to be replaced by creating a new one, so CloudFormation creates a new one and removes the old one from its control, but leaves the resource itself
a CloudFormation stack removed

The last point worked for us.

Okay, let’s repeat the experiment — create a stack again, but now will add the removal_policy to parameter to the basket to delete it when deleting the stack, and auto_delete_objects=True to delete all objects in it, because otherwise the basket cannot be deleted.

In addition, the KMS Key for the basket must be created as a separate object and have removal_policy passed to it, and then this Key must be passed as an argument to the encryption_key basket parameter:

from constructs import Construct
from aws_cdk import (
    Stack,
    RemovalPolicy,
    aws_s3 as s3,
    aws_kms as kms
)

class CdkExampleStack(Stack):

    def __init__ (self, scope: Construct, construct_id: str, **kwargs) -> None:
        super(). __init__ (scope, construct_id, **kwargs)

        my_key = kms.Key(self, "MyKey",
            enable_key_rotation=True,
            removal_policy=RemovalPolicy.DESTROY
        )

        bucket = s3.Bucket(self, "MyEncryptedBucket",
            encryption=s3.BucketEncryption.KMS,
            encryption_key=my_key,
            removal_policy=RemovalPolicy.DESTROY,
            auto_delete_objects=True
        )

Run synth:

$ cdk synth
Resources:
MyKey6AB29FA6:
Type: AWS::KMS::Key
…
UpdateReplacePolicy: Delete
DeletionPolicy: Delete
…
MyEncryptedBucket9A8D2FE1:
Type: AWS::S3::Bucket
…
UpdateReplacePolicy: Delete
DeletionPolicy: Delete
…

Now it should work — we can see both resources have the DeletionPolicy: Delete.

Let’s deploy:

$ cdk deploy
…
cdk-example: creating CloudFormation changeset…
✅ cdk-example
✨ Deployment time: 180.7s
Stack ARN:
arn:aws:cloudformation:eu-central-1:264***286:stack/cdk-example/1f5353d0-efe4–11ed-a627–02deb26b4a5c
✨ Total time: 187.53s

And delete it:

$ cdk destroy
Are you sure you want to delete: cdk-example (y/n)? y
cdk-example: destroying… [1/1]
✅ cdk-example: destroyed

I didn’t have time to take a screenshot, but the CDK was running an AWS Lambda which deleted objects in the S3 bucket, and maybe the bucket itself.

Well, that’s all for now.

The Workshop has more examples, and some more advanced ones, so I’d recommend it if you plan to the AWS CDK.

Did you like the article? Buy me a coffee!

Originally published at RTFM: Linux, DevOps, and system administration.