Matt Bacchi

Using the VSCode Claude Code Extension with Bedrock and Claude Sonnet 4.5

Thu, 01 Jan 2026 19:00:00 -0500

Lots of folks use the Claude IDE, or the Claude Code VSCode extension. Unfortunately, your prompts and completions are used (by default) to train Claude models. [0]

AWS Bedrock, on the other hand, doesn’t use your prompts and completions to train any AWS models or give them to 3rd parties. [1]

For these reasons (privacy, data sovereignty) I’m more inclined to use Bedrock as an LLM in my IDE. Today we’ll go over how to set up the VSCode Claude Code extension with AWS Bedrock and use the Claude Sonnet 4.5 foundation model.

Overview

In order to use the Claude Code VSCode extension with Bedrock and the Claude Sonnet 4.5 model, we need to perform these tasks:

Setup AWS IAM permissions to allow Bedrock usage
Install and configure Claude Code VSCode extension
Integrate AWS credentials, configure extension
Test our setup works

To do this, we’ll use Terraform to create our AWS IAM user and policies (but you could use the AWS console.)

Then we’ll integrate this all together and verify it works as expected.

Unfortunate Missing Features and a Bug/Regression with the Claude Code Extension

AWS Bedrock enables generating short lived API tokens via their SDK. [2] Claude Code does support two methods for automatic AWS credential refresh, but Bedrock API tokens is not one of them.

If it did support this feature it would be the best solution from a security perspective. Tokens would expire in 12 hours and when expired and the extension is used it would automatically refresh them.

Instead it only supports AWS SSO (or rather AWS Identity Center) for it’s awsAuthRefresh option, or AWS IAM credentials for its awsCredentialsExport refresh methods. [3] This deficiency is a poor decision, or at least an oversight by the Claude Code development team.

Unfortunately, a more egregious issue is that they claim the above awsCredentialsExport refresh method is functional when it is not. Whether it’s a regression or bug, or maybe never worked, I couldn’t get it working within a couple hours. (Including time spent conversing with the Claude Code VSCode extension using a working AWS Profile and it couldn’t suggest a workaround to this problem.) In addition to all these setbacks, using the Claude Code settings file (~/.claude/settings.json) didn’t work either so I have to use the VSCode settings file to set all Claude Code extension configuration options.

Since using AWS Identity Center for a personal account is overkill, refreshing Bedrock API tokens in Claude Code is not supported, and the AWS credential export method to automatically refresh my credentials for the Claude Code VSCode extension is not functional, I’ll settle for the lowest common denominator and use the AWS Profile in my configuration. I don’t love this method because it uses a long lived AWS IAM user credential (access key and secret access key.) But I can’t improve security due to the poor state of affairs in the Claude Code VSCode extension.

Create IAM User and Bedrock Permissions

Create an IAM user and attach IAM policy to the user with this Terraform:


resource "aws_iam_user" "bedrock_user" {
  name = "bedrock-user"
}

resource "aws_iam_access_key" "bedrock" {
  user = aws_iam_user.bedrock_user.name
}

data "aws_iam_policy_document" "bedrock" {
  statement {
    effect = "Allow"
    actions = [
      "bedrock:InvokeModel",
      "bedrock:ListFoundationModels",
      "bedrock:ListInferenceProfiles",
      "bedrock:InvokeModelWithResponseStream",
      #   "bedrock:CallWithBearerToken" # required if using Bedrock API token which we're not doing here
    ]
    resources = ["*"]
  }
  statement {
    effect = "Allow"
    actions = [
      "aws-marketplace:ViewSubscriptions",
      "aws-marketplace:Subscribe"
    ]
    resources = ["*"]
    condition {
      test     = "StringEquals"
      variable = "aws:CalledViaLast"
      values   = ["bedrock.amazonaws.com"]
    }
  }
}

resource "aws_iam_user_policy" "bedrock" {
  name   = "bedrock"
  user   = aws_iam_user.bedrock_user.name
  policy = data.aws_iam_policy_document.bedrock.json
}

Setup AWS Profile

I typically use aws-vault to manage my AWS credentials, for enhanced security and to use short lived credentials. But again we’re going to have to use the standard AWS method of storing long lived credentials in our ~/.aws/credentials file, and then access that profile from the Claude Code extension.

Above in the IAM user creation section, we only created the IAM user and policy. Now you will need to create an Access Key to use as the credential in your profile. Go to the AWS Console and create an access key for this user, follow the instructions to create it for the CLI use case.

After creating it, copy your Access Key and Secret Access Key somewhere for safe keeping (I use a password manager for this purpose.)

To setup your profile in the ~/.aws/credentials file, use the command:

`1`	`aws configure --profile YOUR_AWS_PROFILE_NAME`

It will prompt you to provide the AWS Access Key and Secret Access Key, which will be added to the ~/.aws/credentials file using the profile name you specified in the command (choose wisely!)

Enable Bedrock Claude Sonnet 4.5 and Validate Access

After creating the above IAM user and policy, and setting up the profile, we’ll login to the AWS console. Choose the AWS region that you normally use, but be aware that these Bedrock foundation models aren’t available in every single global AWS region. I used region us-west-2 but us-east-1 and us-east-2 are also supported.

Anthropic requires first-time customers to submit use case details before invoking a model once per account or once at the organization’s management account. [4] This is a rather antiquated policy in the cloud era, but required nonetheless.

Go to the Bedrock section of the AWS console, then to “Chat/Text Playground” and select Anthropic Claude Sonnet 4.5. You’ll be presented with a dialog to fill out and enable the foundation model.

Install and Configure VSCode

Install the VSCode extension:

`1`	`code --install-extension anthropic.claude-code`

Identify your Bedrock Claude Sonnet 4.5 inference profile ARN:

aws bedrock list-inference-profiles  --region us-west-2 --profile YOUR_AWS_PROFILE_NAME --no-cli-pager | jq '.inferenceProfileSummaries | .[] | select(.inferenceProfileId | match("us.anthropic.claude-sonnet-4-5-20250929-v1:0")) | .inferenceProfileArn'

NOTE: This assumes you’re in the US, if using another global region use the global Anthropic Claude Sonnet profile name in the above command.

Add the following to your VSCode user settings.json file (usually this is ~/.config/Code/User/settings.json):

{
  "claudeCode.selectedModel": "us.anthropic.claude-sonnet-4-5-20250929-v1:0",
  "claudeCode.environmentVariables": [
    {
      "name": "AWS_PROFILE",
      "value": "YOUR_AWS_PROFILE_NAME"
    },
    {
      "name": "AWS_REGION",
      "value": "YOUR_AWS_REGION_FROM_ABOVE"
    },
    {
      "name": "BEDROCK_MODEL_ID",
      "value": "INFERENCE_PROFILE_ARN_FROM_ABOVE"
    },
    {
      "name": "CLAUDE_CODE_USE_BEDROCK",
      "value": "1"
    }
  ],
  "claudeCode.disableLoginPrompt": true,
}

Did it work?

You’ll probably have to restart your VSCode session if it was running. Then, open the Claude window and type a question or request and you should see a successful response like below:

Conclusion

We did it! I’m not super impressed with the missing/broken/overlooked features of the Claude Code VSCode extension related to AWS IAM credentials. But it works fine for the time being and I’ll revisit this issue and report back when these issues are resolved and we can all begin using short lived credentials with the extension.

Resources

I borrowed some information from this incredibly detailed blog post by Vasko Kelkocev. [5]

But even though that blog was written in October 2025, it was already out of date by the time I found it. I had to add more IAM permissions to get the extension to work with Bedrock (specifically bedrock:InvokeModelWithResponseStream), and there were some other issues with the configuration I had to play with. Thanks for the great blog Vasko.

Cover photo by Fotis Fotopoulos on Unsplash

Firecracker Virtualization Overview

Thu, 11 Dec 2025 17:00:00 -0500

Firecracker is an open source virtualization technology created by Amazon Web Services (AWS) which underpins their AWS Lambda Functions as a Service (FaaS) serverless product.

Firecracker was open sourced in 2018 [0], making it possible for anyone to use this extremely fast and reliable system for their own projects and use cases.

I’ve been researching the ecosystem lately and am impressed at the flexibility architected into the Firecracker code which enables it to be used in many ways. We might have expected them to design it such that it only works in their very tightly controlled environment. But the fact that it’s not specialized to just the AWS Lambda use case means that it can be leveraged by anyone from AWS scale to a home lab running a single VM.

Let’s explore the capabilities of Firecracker and the various methods of using it.

Firecracker High Level Overview

There are many good quick start documents 1 and blogs describing how to install and start a single Firecracker MicroVM instance. Because of these great resources readily available, I won’t describe that here.

The basic requirements are:

Firecracker binary
kernel
rootfs
networking configuration (optional)

When you start a firecracker VM by executing the firecracker program, you are running a single VM instance which can be managed through the firecracker process you launched it with. You can send subsequent commands to perform additional management tasks to the firecracker process using the API via a unix socket. When you are done with the VM you can and should stop it “gracefully”.

Multiple firecracker processes can be executed at one time, which translates to running multiple VMs. Presumably this is how AWS runs millions of Lambda functions.

Configuration

There are a couple ways to configure your VM:

Sending HTTP requests via the API socket [2]
Using the configuration file [3]

If you use the configuration file to set the VM guest kernel and rootfs, you can still use the socket to send API requests. These configuration methods seem to conflict, but they appear to me to be part of the design of a highly flexible system.

The HTTP API is enhanced by what appears to be an official Go SDK [4] that you can use to manage your Firecracker VM instances. A likely scenario is that AWS uses this Go SDK to provision and control their Lambda functions via this HTTP API provided by the Firecracker process.

Initialization

Firecracker is also flexible in how it can be initialized. Options include:

command line execution
systemd
jailer

The first two options are obviously common to many Linux tools. But the diverse use cases that AWS themselves have for a multi-purpose virtualization technology like Firecracker make it likely that they designed it to be versatile. This adaptable capability extends from the configuration methods to initialization, and likely beyond.

Enhanced Isolation using Jailer

The jailer tool [5] which is specifically designed for firecracker further isolates each firecracker process to its own chroot jail directory. This security mechanism has been used for years in Unix systems and affords a greater sense of security in a multi-tenant environment.

We won’t go into detail on this for now, and it isn’t necessary for your home lab unless you’re running untrusted code (i.e. not written by you.)

Demonstration

To quickly show what we’ve discussed above, we’ll demonstrate with a small bash script that starts a Firecracker micro VM using previously downloaded kernel and rootfs. Here’s the script:

#!/usr/bin/env bash

firecracker="/usr/bin/firecracker"

uuid=$(/usr/bin/uuidgen)

nohup $firecracker --api-sock /tmp/$uuid.socket &

echo "created firecracker process running on socket: /tmp/$uuid.socket"

curl --unix-socket /tmp/$uuid.socket -i \
    -X PUT 'http://localhost/boot-source' \
    -H 'Accept: application/json' \
    -H 'Content-Type: application/json' \
    -d '{
        "kernel_image_path": "/tmp/firecracker-test/hello-vmlinux.bin",
        "boot_args": "console=ttyS0 reboot=k panic=1 pci=off"
    }'

curl --unix-socket /tmp/$uuid.socket -i \
    -X PUT 'http://localhost/drives/rootfs' \
    -H 'Accept: application/json' \
    -H 'Content-Type: application/json' \
    -d '{
        "drive_id": "rootfs",
        "path_on_host": "/tmp/firecracker-test/hello-rootfs.ext4",
        "is_root_device": true,
        "is_read_only": false
    }'

curl --unix-socket /tmp/$uuid.socket -i \
    -X PUT 'http://localhost/actions' \
    -H  'Accept: application/json' \
    -H  'Content-Type: application/json' \
    -d '{
        "action_type": "InstanceStart"
    }'

echo "Started VM instance."

Running it we get the following output:

user@temphost:~/data/firecracker-test$ ./startvm.sh 
created firecracker process running on socket: /tmp/b214c708-e506-455b-a98d-647939a0ef0d.socket
nohup: appending output to 'nohup.out'
HTTP/1.1 204 
Server: Firecracker API
Connection: keep-alive

HTTP/1.1 204 
Server: Firecracker API
Connection: keep-alive

HTTP/1.1 204 
Server: Firecracker API
Connection: keep-alive

Started VM instance.
user@temphost:~/data/firecracker-test$ tail nohup.out 
 [ ok ]
 * Mounting persistent storage (pstore) filesystem ...
 [ ok ]
Starting default runlevel
[    1.088111] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x2879b124109, max_idle_ns: 440795245440 ns

Welcome to Alpine Linux 3.8
Kernel 4.14.55-84.37.amzn2.x86_64 on an x86_64 (ttyS0)

localhost login:

You can see that we have a login prompt in the nohup.out file that indicates the VM is running and ready to accept logins. We didn’t setup any SSH keys in this attempt, nor networking, so we’re SOL on logging in. But that’s OK because we were just trying to show how straightforward this can be.

Notice we created a random UUID to use as the firecracker unix socket, which is how you can communicate with it via the HTTP API. After starting the firecracker process, we then make curl requests to configure the kernel, rootfs, and then initiate the InstanceStart action.

Granted, this is a very rudimentary example. But it shows how easily firecracker can be used if you understand the fundamentals.

Firecracker Ecosystem Discrepancies Compared to Lambda

In multiple AWS re:Invent talks, Lambda architecture has been detailed by AWS employees at a high level. One example is in Julian Wood’s 2022 session A closer look at AWS Lambda (SVS404-R). At 10:36 in the video, he references the Lambda data plane, which consists of a host of interrelated services for synchronous vs. asynchronous Lambda invocations. These services are:

Synchronous

Frontend Invoke Service
Counting Service
Assignment Service
Worker Hosts
Placement Service

Asynchronous

Poller Fleet
Queue Manager
State Manager
Stream Tracker
Leasing Service

The Cold Truth

These administrative services used to run millions or billions of AWS Lambda invocations are obviously not available for Firecracker. It makes sense that AWS would release their Firecracker product as open source, but they’re not likely to ever provide the facade that props Firecracker up in their environment, enabling AWS Lambda to be highly performant and reliable. But this is something I’m interested in. I think some of these features that streamline the functionality of Firecracker in the Lambda use case could be built to make it easier to manage for those of us who think this tool is exciting and wish it came with batteries included.

Conclusion

Over the next few weeks (and months) I’ll be playing with this some more. I’ll likely be posting as I do this to give a different perspective on firecracker than the standard quick start blogs out there already.

Hope you got something out of this today!

Cover photo by NASA Hubble Space Telescope on Unsplash

Installing Nvidia Datacenter GPU Manager on Amazon Linux 2023

Sat, 01 Feb 2025 08:00:00 -0500

A short post discussing the installation of Nvidia Datacenter GPU Manager on Amazon Linux 2023. I recently had to figure this out using sketchy documentation, so I’m hoping this helps some folks out there doing similar head scratching. NOTE: This blog post does not include all steps to install Nvidia drivers. I assume you already have driver package(s) required for your application installed.

What is the Datacenter GPU Manager

Nvidia Datacenter GPU Manager is a suite of tools used for managing and monitoring Nvidia GPUs in the data center.

In our environment we use Nvidia Datacenter GPU Manager as a prerequisite for the DCGM Prometheus metrics exporter used to send metrics to Datadog, named dcgm-exporter. I won’t go into detail on installing dcgm-exporter, the instructions in the Github README are fairly straightforward to follow.

Amazon Linux 2 End Of Life

The Amazon Linux 2 (AL2) EOL date is 6/30/2025 (June 30, 2025). The replacement is Amazon Linux 2023 (AL2023) [3].

Installing DCGM on AL2023

Installing Nvidia Datacenter GPU Manager on AL2 was documented, but applying that documentation to AL2023 wasn’t clear what value to use for the <distro> [4].

In order to understand what distribution Amazon Linux 2023 most closely relates to, you have to look in the Amazon Linux 2023 User Guide. It describes that AL2023 is “sourced from multiple versions of Fedora…including CentOS 9.”

That means when following the Nvidia installation instructions, they suggest enabling the repository with this command:

`1`	`sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/<distro>/x86_64/cuda-<distro>.repo`

Because AL2023 is derived from Fedora/CentOS 9, the <distro> parameter is going to be rhel9. So our command to install the repository instead becomes:

`1`	`sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo`

Now you can run the install command for the datacenter-gpu-manager package:

`1`	`sudo dnf install -y datacenter-gpu-manager`

Conclusion

The documentation was not clear which distribution should be used for AL2023. Replace <distro> with rhel9 in the repo installation command and we’re now able to install the datacenter-gpu-manager package.

Cover photo by Christian Wiediger on Unsplash

Switching to the Terraform S3 Backend with Native State File Locks

Tue, 07 Jan 2025 08:00:00 -0500

Terraform is a flexible, cloud agnostic infrastructure as code (IaC) tool. As it constructs infrastructure resources, it builds a ledger used to track resources that have successfully been created as well as additional metadata (such as id.) Terraform stores this state in a binary formatted file with the extension .tfstate.

What is the Terraform S3 Backend

The Terraform state file described above by default is stored in the same directory as the Terraform infrastructure definition files you wrote. But with this state on your local computer it is vulnerable to being lost or overwritten, and it cannot be shared with or managed by other team members. Using a distributed storage mechanism to store this state file is straightforward with Terraform, and they provide many backend options. For AWS users, the Terraform S3 Backend allows storing this state file in AWS S3.

What is State File Locking

So now we know this state file is stored in distributed object storage (AWS S3,) and more than one user can manage resources within it. But to safely manage this state file, we require a locking mechanism (often called a mutex in computing) that disallows multiple users from attempting to write to it at once. We would have a mess if we allowed more than one user to write to it at the same time, potentially losing the resources that were created and intended to be stored in this state file.

Terraform state locking capability has been available for the S3 backend for quite some time. But unfortunately it has required an additional DynamoDB table to be created that tracked the state file locking status.

Until now.

This DynamoDB table is an extra resource that seemed tangential to the Terraform state backend process and complicated the process of configuring your backend. That requirement has been rendered obsolete with a recent feature that was added to AWS S3, conditional writes.

AWS S3 Conditional Writes

In August, AWS announced the addition of the S3 Conditional Writes feature. This feature of AWS S3 compels S3 clients to check for the existence of an object before writing it, and if it already exists to fail. If the file exists the S3 client returns a 412 Precondition Failed error response.

S3 Conditional Write Support Added in Terraform v1.10

Support for S3 Conditional Writes was added to Terraform release v1.10. (If you want to see some great background and architecture detail from the developer Bruno Schaatsbergen about the implementation look here.)

Thankfully this is completely transparent to the Terraform user (unless it returns an error attempting to lock the state file.)

Configuring the S3 Backend to Use Native State File Locking

The Terraform documentation describes the new configuration parameter use_lockfile to enable S3 state locking. It also currently describes the old DynamoDB method as still available. (It’s common for software to support both an old and new related feature for some time until all users can migrate to the new methodology.)

This means you can actually use both locking mechanisms at the same time. But this is both unnecessary overkill, and could lead to confusion and problems. I would recommend that you replace your old DynamoDB locking configuration with S3 state locking immediately. It will be cheaper (without having to pay for an extra DynamoDB table or reads/writes to that table,) and less error prone.

Here’s how to change your Terraform backend configuration.

Terraform Configuration Specifics

The old DynamoDB method used a configuration parameter named dynamodb_table.

The new S3 state locking method uses a configuration parameter named use_lockfile.

Both are covered in the current Terraform documentation.

Version Constraints

We also recommend that when you switch to the S3 native state locking method, you set the Terraform configuration parameter required_version to the minimum version that supports S3 native state file locking. If you have users with an earlier version of Terraform, they won’t be able to use this feature and will see errors if the use_lockfile parameter is enabled. Setting the required_version to v1.10 at a minumum makes your configuration more resilient and doesn’t let someone attempt to create or update resources using an older Terraform version. Think of it as a prerequisite.

This Terraform version constraints configuration is documented here, and looks like this:

`1`	`required_version = "~> 1.10"`

Sample Configuration

With all this background information about the configuration parameters, here’s a sample Terraform configuration with both the old and new parameters present:

provider "aws" {
  region = "us-west-2"
}

terraform {
  backend "s3" {
    encrypt        = true
    bucket         = "tfstate-lock-test-0bhfxn8x1"
    region         = "us-west-2"
    key            = "example/terraform-state-lock-test.tfstate"
    dynamodb_table = "tfstate-lock-test"
    use_lockfile   = true
  }

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "5.82.2"
    }

  # This sets the version constraint to a minimum of 1.10 for native state file locking support
  required_version = "~> 1.10"
}

In order to switch away from using the old DynamoDB locking method, remove that dynamodb_table configuration parameter.

State File Locking in Action

We now know how to configure Terraform S3 native state file locking, but how does it perform and what will we see if you cannot get the mutex to lock the file?

I’ve tested both methods and will show you the output from each when state file locking fails.

Error from DynamoDB State File Locking

The old DyamoDB state file locking method would return an error such as the below:

$ terraform apply plan.out 
Acquiring state lock. This may take a few moments...
╷
│ Error: Error acquiring the state lock
│ 
│ Error message: operation error DynamoDB: PutItem, https response error StatusCode: 400, RequestID: CP9U4IRC04OBONKIQHM4LUOLLJVV4KQNSO5AEMVJF66Q9ASUAAJG, ConditionalCheckFailedException: The conditional request failed
│ Lock Info:
│   ID:        39f64263-4ad8-a563-faf7-f28f8a042a00
│   Path:      tfstate-lock-test-0bhfxn8x1/example/terraform-state-lock-test.tfstate
│   Operation: OperationTypeApply
│   Who:       user@hostname
│   Version:   1.10.3
│   Created:   2025-01-08 03:38:13.121564614 +0000 UTC
│   Info:      
│ 
│ 
│ Terraform acquires a state lock to protect the state from being written
│ by multiple users at the same time. Please resolve the issue above and try
│ again. For most commands, you can disable locking with the "-lock=false"
│ flag, but this is not recommended.

Error from S3 Backend Native State File Locking

The new S3 backend native state file locking method will return an error that looks like this:

$ terraform apply plan2.out
╷
│ Error: Error acquiring the state lock
│ 
│ Error message: operation error S3: PutObject, https response error StatusCode: 412, RequestID: N0BGGAFN8V1N2WCQ, HostID: 4G32Xus/86u8CehvNbvzv8NoqiyTvsBWGYBXYK6E8Vn0E4+wom+6Jm6WFVUFSaCE7C1TBP5Vauo=, api error PreconditionFailed: At least one of the pre-conditions you specified did not hold
│ Lock Info:
│   ID:        837482e8-441e-9ff6-d30b-333ee83d8fc4
│   Path:      tfstate-lock-test-0bhfxn8x1/example/terraform-state-lock-test.tfstate
│   Operation: OperationTypeApply
│   Who:       user@hostname
│   Version:   1.10.3
│   Created:   2025-01-08 03:43:11.691479913 +0000 UTC
│   Info:      
│ 
│ 
│ Terraform acquires a state lock to protect the state from being written
│ by multiple users at the same time. Please resolve the issue above and try
│ again. For most commands, you can disable locking with the "-lock=false"
│ flag, but this is not recommended.

Dealing with Stale State File Locks

NOTE: After publishing this blog, I was asked whether the terraform force-unlock command still worked. I tested this and can say it does perform as expected with the old S3 DynamoDB state file locking mechanism. Here’s an example session showing this:

$ terraform apply plan2.out
╷
│ Error: Error acquiring the state lock
│ 
│ Error message: operation error S3: PutObject, https response error StatusCode: 412, RequestID: NGQM2VGSTSDWPCZF, HostID: dzPZArnTy31oVeuVLI8Dm61HXnuL6M3R2tlWFe2suztP0zkh4Bwv/eJFBLqVfitAI40I5BvIeds=, api error PreconditionFailed: At least one of the pre-conditions you specified did not hold
│ Lock Info:
│   ID:        bde40e3b-2bfb-f577-fea5-44923c9d5275
│   Path:      tfstate-lock-test-0bhfxn8x1/example/terraform-state-lock-test.tfstate
│   Operation: OperationTypeApply
│   Who:       user@hostname
│   Version:   1.10.3
│   Created:   2025-01-08 16:55:35.808464751 +0000 UTC
│   Info:      
│ 
│ 
│ Terraform acquires a state lock to protect the state from being written
│ by multiple users at the same time. Please resolve the issue above and try
│ again. For most commands, you can disable locking with the "-lock=false"
│ flag, but this is not recommended.
╵

You can remove the lock, but only do this if you know the lock is stale. To do this, first note the lock ID above, then run the force-unlock command:

$ terraform force-unlock bde40e3b-2bfb-f577-fea5-44923c9d5275
Do you really want to force-unlock?
  Terraform will remove the lock on the remote state.
  This will allow local Terraform commands to modify this state, even though it
  may still be in use. Only 'yes' will be accepted to confirm.

  Enter a value: yes

Terraform state has been successfully unlocked!

The state has been unlocked, and Terraform commands should now be able to
obtain a new lock on the remote state.

Delete Your Old DynamoDB Tables

Now that you’ve switched from using the old Terraform DynamoDB locking to the new S3 native state file locking, you can remove the old DynamoDB table used to track these locks!

Yay, one less resource to manage and be charged by AWS for.

Summary

Hopefully you see the advantage of using the new Terraform S3 backend native state file locking mechanism, and how to configure it for your environment.

Happy Terraforming!

A Survey of Serverless Sustainability Trends

Sun, 29 Dec 2024 08:00:00 -0500

As we bring 2024 to a close, and after an invigorating week at AWS re:Invent, many will be writing their year in review summaries. I’ve decided to dedicate those column inches to the state of serverless sustainability today. The observant among us are quite aware of how the artificial intelligence (AI) craze has wormed it’s way into every product, industry and conversation over the past year. It’s been making headlines as shuttered power plants like Three Mile Island are reopened. And it’s on a collision course with the world’s climate change goals. (AWS also announced in October they signed agreements for 3 small modular nuclear reactor projects for their own data centers.)

Running servers in data centers is an energy intensive process. Every search for a restaurant or game score requires a response from a server. Running a busy webserver requires the same power as one that gets 5 hits per day. Serverless technologies like AWS Lambda, S3, DynamoDB can be used to reduce that constant power usage when there is no demand.

Today we’ll discuss the current state of affairs with serverless sustainability, and perhaps a little about sustainable computing in general.

What is Sustainability

In the context of technology and computing in particular, sustainability equates primarily to power usage and carbon emissions. The computing industry has been on an upward trend in energy use since, well, the beginning of the computer era. During that time, increases in computational capabilities (faster processors) have driven higher demands for electricity. The next (current?) era of computing is going to require rethinking these needs by reducing the power requirements of computers and data centers, recycling hardware, and becoming better climate citizens. (AWS discusses some of their sustainability improvements related to the circular economy in this re:Invent talk “Advancing sustainable AWS infrastructure to power AI solutions” here.)

AWS defines sustainability as one of the 6 pillars of their Serverless Applications Lens for the AWS Well-Architected Framework. Their Well-Architected Framework documentation covering Sustainability indicates their focus on environmental sustainability in this pillar. It’s important that they make that distinction, and even more telling that they are making a large commitment to improving energy efficiency and carbon emissions in their own operations. This document (and others they’ve released [PDF]) lay out their goals as well as identify how AWS cloud customers can optimize their operations to become more sustainable.

Sustainable Workloads

As is common when working with AWS, they assert customers approach cloud sustainability by using the shared responsibility model. This is akin to their approach for security in the cloud, so it makes sense that they want users to take our part seriously.

Region Selection

One way to make your application more sustainable is to use the lowest carbon intense AWS region available, taking into account other business requirements such as latency. If you are a small company with only customers in one country or region, you may not be able to take advantage, but if business requirements don’t disallow using a more remote location you can use this technique. Obviously some AWS regions are in locations that have a higher percentage of renewable energy entering it’s power grid. Deciding to use one of these regions should be based on latency of course too, but carbon intensity must be part of the decision.

How, then, do we choose a region for sustainability? A good reference is this AWS Architecture Blog: How to select a Region for your workload based on sustainability goals. They suggest using the site electricitymaps.com to identify carbon intensity and renewal energy percentage for each regional electricity grid.

On the electicitymaps 24 hour climate impact map, hover over the PJM region (which covers Virginia, home to the us-east-1 AWS region) we see the carbon intensity (as of 12/21/2024) is 409g CO²/kWh, and the renewable energy mix is 8%. If we now look at the California power grid (home to the AWS us-west-1 region), we see 139g CO²/kWh, and the renewable energy mix is 67%.

Over-provisioned Capacity and Time Shifting

Earlier this year I ran across a podcast from the Green Software Foundation called Environment Variables. In one episode the guest Kate Goldenring talked about the sustainability merits of serverless. She pointed out a statistic from the Sysdig 2023 Cloud Native Security and Usage Report that showed 69% of requested CPU resources go unused. That means as an industry we’re over provisioning by more than two thirds (2/3). The goal should be higher hardware utilization according to Kate.

This proves what we’ve been saying about how running applications on servers (such as EC2) or containers is less efficient than serverless. Provisioning for peak capacity is what makes these “always on” deployments wasteful because traffic is rarely at it’s expected peak usage. Serverless technologies that utilize multi-tenancy are intentionally more sustainable than traditional deployment methods because they enable higher hardware utilization.

It’s idealistic to assume we can perfectly utilize hardware via multi-tenancy, but one approach Meta has begun using in their internal private cloud functions product called XFaaS (similar to AWS Lambda) is a technique called “time shifting”. Instead of running all workloads immediately, they will delay some operations that are not time sensitive. This increases the average utilization of their hardware due to scheduling these delay tolerant functions during off-peak times, ultimately lowering their peaks and raising their troughs. That equates to a higher average utilization.

This introduces an alternative approach to managing sustainability, seldom discussed in the serverless realm as far as I’ve seen. In many event driven systems events don’t require immediate processing, and are delay tolerant by design. This characteristic lends itself well to having the processing delayed until utilization in your system is below a certain threshold (i.e. off hours.) Of course this is only possible for data that isn’t expected to be immediately consistent. This type of processing emerged in the mainframe computing era, and was called “batch processing”. It’s still in use today, did you ever wonder why your bank statement can take up to 24 hours to reflect purchases? They’re processing payments either overnight or in some batch processing methodology.

Time Shifting on Kubernetes

Although I don’t consider Kubernetes to be serverless by definition, I happened across a Kubernetes admission controller recently that accepts delay tolerant workloads. This paradigm is atypical for Kubernetes (which usually processes a request as soon as it’s received,) but the pattern hints at the potential future of all workload processing systems.

You may have a request sent to your backend system that doesn’t require immediate processing, but rather is capable of running whenever is most efficient to do so. The research paper describing the Cucumber and a (2 year old but unmodified) reference implementation are available. While this is more geared towards running workloads when excess solar power is being generated, the concept could be extended to many types of situations such as how heavily loaded your servers are.

Summary

You’ve just read a few of the trends that have been developing in the sustainable serverless computing arena. Hope this has provided some new information for you to go delve into on your own. Thanks and have a great 2025!

Cover photo by Samuel Faber from Pixabay

Limitations of AWS EC2 Image Builder Lifecycle Policies

Mon, 12 Aug 2024 08:00:00 -0500

EC2 Image Builder Lifecycle Policies is a fairly new AWS feature that enables automatic cleanup of old EC2 Image Builder pipeline artifacts. If you haven’t used EC2 Image Builder, it’s quite handy for creating customized AWS Machine Images (AMIs) for your EC2 deployments.

This could be a really valuable tool that simplifies the AMI build process. But there are a number of artifacts left behind after your AMI is built that potentially cost money and require manual effort to clean up.

Enter EC2 Image Builder Lifecycle Policies. This feature is advertised as a way to automatically tidy these artifacts. Unfortunately it’s almost unusable if you build a large number of AMIs constructed from numerous recipes.

Below I’ll describe a high frequency use case that illustrates the limitations of EC2 Image Builder Lifecycle Policies. You can make your own conclusions about the utility of this tool as it stands today.

Standard use case

The process of building AWS EC2 Amazon Machine Images (AMIs) can be rather involved. You start with a base image, add required packages and user data, then build, test, and finally deploy your custom AMI. The deployment step isn’t even straightforward, as it potentially requires disk volumes, network interfaces, and additional resources which necessitate experimentation. Even after you’ve done all this, managing the state of these AMIs and their related resources still isn’t complete. These resources all need to be cleaned up. And because AMIs tend to be created frequently to meet security objectives, the entire process can occur at a high cadence.

Why so many AMIs?

We all know how frequently CVEs (Common Vulnerabilities and Exposures) can be released, and how quickly management expects them to be mitigated. Lower severity security updates come out at an even more unrelenting pace. For production infrastructure at enterprise scale, updates need to be routinely applied and be extremely reliable. This process is simplified by automating it all with an EC2 Image Builder pipeline executed on a regular schedule.

But after these AMIs and related resources have been superseded by newer ones they need to be removed. Until now there was no method provided by AWS to perform this action. You needed to develop your own tooling for this purpose.

EC2 Image Builder Lifecycle Policies

Lifecycle Policies adds a necessary (heretofore missing) feature to EC2 Image Builder. Having AWS remove your unused resources is a great idea. But the execution leaves something to be desired. For the sake of illustration, let’s also define what “related resources” means here.

Every AMI built via EC2 Image Builder requires an image recipe and image pipeline be created. An image recipe consists of AWS managed components or custom (user generated) components to perform an AMI build. The pipeline runs your recipes which generate an output image upon success which references the output AMI, infrastructure configuration, distribution settings, and a CloudWatch log stream (which includes pipeline output for debugging purposes.)

When you modify the image recipe, a new recipe version is created and the previous versions become obsolete. The now unused recipe and it’s related resources (such as the output image as well as the AMI) remains active in AWS indefinitely and won’t be removed until you perform that action. (I’ve highlighted this specific point for a reason that will become obvious later.)

If you thought you merely needed to delete the AMI when you were done with it, you would leave all these related Image Builder resources in your account. (Granted they don’t cost a lot, but it makes sense to remove unused resources, if only to avoid hitting quota limits at some point in the future.)

While EC2 Image Builder Lifecycle Policies makes this last cleanup bit somewhat more straightforward, it cannot be said it’s “a one size fits all” solution.

Limitations of the current Lifecycle Policies service

To create a lifecycle policy for AMI image resources, the documentation says you must “apply lifecycle rules to image resources based on the recipe that created them, select up to 50 recipe versions for the policy.”

This is unwieldy because you have to add new recipe versions explicitly each time you create a new image recipe. If you use Infrastructure as Code (IaC) tools to do this (such as Terraform, CloudFormation or CDK), it becomes slightly less manual, but still requires manual effort (or at least in depth awareness of the process.) But even if you add these recipes to your Lifecycle Policy manually, there’s a limit of 50 recipe versions in the policy. This isn’t as irritating if you only have a dozen recipes in total, the scope of the problem is intensified with scale.

Also, I’m referring to lifecycle policies for AMI image resources, I don’t currently use EC2 Image Builder to create container image resources.

The limitations as I see them are:

The maximum number of recipes allowed in the LifecyclePolicyResourceSelection API documentation which can be used as selection criteria is 50. This 50 recipe limit is quite an arbitrary number, and when a user has a large number of image recipes it is both an extremely small limit, and would become a burden to add all recipes to this rule scope parameter.

I’ve verified this is not adjustable in the AWS Service Quotas self service quota increase dashboard in the AWS Console (or the Service Quotas API).
In the EC2 Image Builder Lifecycle Policies AWS Console, when selecting the recipes to apply the Rule Scope, the drop down list only displays a few current recipes. It shows a spinner and indicates it is “Loading Recipes” (see screenshot) but the additional recipes never show up.

Conclusion

While EC2 Image Builder Lifecycle Policies has the potential to be a great service, it is currently hampered by the user experience (UX.) Adding functionality to make adding recipes to policies easier would be a major improvement for early adopters and power users.

S3 Bucket Takeover Neutralization

Tue, 30 Jan 2024 07:00:00 -0500

There’s been a recent uptick in the number of S3 buckets that have become “hijacked” or “taken over”.

How can an attacker take over your S3 bucket? What can you do to prevent this on your buckets?

We’ll answer these questions and provide details on how you can neutralize the threat of S3 bucket takeovers.

What is Hijacking?

How can your S3 bucket get hijacked?

It should be noted that the term “hijacked” is a misnomer. S3 buckets which have ended up in this state were managed poorly by the owner. The original owner had to choose to give up ownership of the S3 bucket, prior to it being claimed by a bad actor. If you have an active AWS S3 bucket in your account, this cannot occur unless you relinquish control. These are categorized as software supply chain attacks, where the attacker hijacks open source software or software updates, eventually compromising target computers.

AWS uses a global namespace for S3 buckets. If a bucket name “mattbacchi” is available and I requisition it, no one else in the world can use that bucket name. A global namespace has some downsides, namely that everyone competes for the same bucket names, and a single bucket name cannot be owned by more than one AWS account.

This surge in exploit/hijack/takeover incidents ultimately has this global namespace to blame. If you owned an S3 bucket but then deleted it, someone else could subsequently recreate it in their account (after a short period of time.)

If the deleted bucket was public and contained trusted files (such as npm packages), someone with sinister intentions could recreate the bucket and masquerade as you. If they named their tainted files the same as your trusted files, users might download the malicious content assuming it was your original content. This is how the above poisoned “bignum” npm package exploit worked. The S3 bucket owned by someone the “bignum” software developer trusted was abandoned, subsequently an attacker recreated it with malicious content.

These attacks cannot occur if S3 buckets are managed with better data lifecycle hygiene.

Preventing S3 Bucket Takeover

In order to neutralize this threat, I recommend never deleting your S3 buckets.

As described above, the only S3 buckets vulnerable to this attack vector are those that have been deleted and are no longer active. The logical defense is to never relinquish your S3 buckets, instead you maintain the bucket and name forever.

Won’t this get expensive, you may ask? AWS bills customers for S3 based on storage. If you aren’t storing any data in the bucket, it will never generate a charge. In this way we retain an S3 bucket name so that it doesn’t fall into the wrong hands.

Of course, this may not be necessary for all S3 buckets you own. I recommend using this technique for any buckets that are public and have been used in production software applications. The “bignum” npm package bucket above is a prime example, retaining that bucket would have prevented the attack.

Decommissioning Process

Developing an official process to manage this in your organization is important.

If you intend to retain your S3 buckets in order to prevent takeover, you must first develop documentation and communicate this policy. Also consider how you will implement identifiers or signposts indicating the bucket must be retained to administrators who may not be familiar with the process. I’ve personally used documentation, an AWS tagging scheme, and a couple other methods to prevent the bucket from being accidentally relinquished.

One method of making this obvious to infrastructure engineers (i.e. devops or platform engineers) is to create a zero byte file in the bucket that identifies it as persistent. I have suggested the file be named DO_NOT_DELETE_S3_BUCKET_PERSISTENT as a fairly large signpost for people to prevent them from deleting it. A sample command to do this via the AWS CLI is something like:

`1`	`touch DO_NOT_DELETE_S3_BUCKET_PERSISTENT; aws s3 cp DO_NOT_DELETE_S3_BUCKET_PERSISTENT s3://BUCKETNAME/`

Another method is to create an AWS tag on the bucket such as persistent=true.

Finally, if you want to use a more durable mechanism, add an S3 bucket policy that denies the action s3:DeleteBucket as well as s3:PutObject. This will prevent removing the bucket unless the user is an administrator, at which point they can remove the s3:DeleteBucket permission from the bucket policy. But they must manually perform this update, and hopefully that extra step makes them realize it shouldn’t be deleted.

Conclusion

Today we’ve discussed the cause of S3 bucket hijacking and some methods of preventing it. Hopefully this motivates you to perform some analysis of your S3 buckets and apply these suggestions in your environment.

Thanks for reading and have a great day!

Event Driven Processing of ip-ranges.json

Sun, 28 Jan 2024 08:00:00 -0600

Imagine you have a security group that needs to allow all IP addresses of AWS EC2 instances. Or imagine you have to allow IP addresses of Github Actions runners so that only your CI workers connect to your VPC. Both of those IP address ranges change regularly, and need to be updated (usually by hand.)

If we want to automate these security group updates, how could you figure out when these IP address ranges have changed? AWS has an SNS notification sent every time their ip-ranges.json list changes. The SNS notification can be used to initiate an automated procedure to update our security group.

What we’re describing is an event driven architecture. In event driven architectures, an event producer causes an event to be created. A downstream event consumer handles the event and may trigger further events.

In this 2 part blog post series, we’ll cover event driven architectures. In the first part of the example used to illustrate the mechanics, we’ll build the IP address range processing component. This piece of the puzzle processes the AWS ip-ranges.json file and inserts it into a DynamoDB table. The second blog in this series will insert the IP ranges into AWS security groups.

After the next blog post, you’ll be able to manage IP address ranges in your environment using Event Driven Architecture.

What is Event Driven Architecture?

The Wikipedia page for event driven architecture (or EDA) describes it as “a software architecture paradigm concerning the production and detection of events”. This is in contrast to software which is focused on its own state, and doesn’t concern itself with external state changes. Event driven architectures are often made up of loosely coupled components that act independantly on events that they’re concerned with.

Why use EDA for data processing?

Data processing is often well suited to the EDA model because it’s rarely necessary to syncronously perform a data processing task. If a user purchases something in a shopping cart, certainly the interatction with the bank or credit card needs to be syncronous and immediate. But if the website catalogs all purchases in a “top ten products” list from all user purchases, that processing can be done asyncrynously at a later, and possibly less busy time of the day. It’s common for data engineering teams to perform extract, transform, load (or ETL) jobs that process data off hours, to avoid contention for compute resources. This can all be done with an event driven model, where the list of purchased items in our above example would be sent to an event bus (such as AWS EventBridge or Apache Kafka.) Consumers of that type of event would then act on the event depending on their function.

As mentioned above, anyone can subscribe to SNS notifications for the ip-ranges.json list. When a change occurs to that list the SNS notification is sent out. It acts as the event producer in this case, causing anyone who is subscribed to the notification list to receive the event.

In our example today, we’ll setup an AWS Lambda function to process these events stemming from the SNS notification. Our function will be the event consumer in this scenario.

Data organization of the ip-ranges.json file

The ip-ranges.json file syntax consists of a creation date (to indicate last update time) and a list of IPv4 and IPv6 address ranges. These look like the following sample:

    {
      "ip_prefix": "52.4.0.0/14",
      "region": "us-east-1",
      "service": "EC2",
      "network_border_group": "us-east-1"
    },

We want to maintain all of this information, and in fact enhance it with the date stamp (or synctoken) which is included in the ip-ranges.json file as mentioned above.

DynamoDB Single Table Design

We’ll be inserting all of these IP address ranges into a DynamoDB table for future access and processing (triggered via events.) DynamoDB is great for serverless applications because it allows a large number of stateless connections via HTTP. Using a single DynamoDB table, we’ll store all of this data in a way that can be easily queried by creating composite keys that pre-join the data fields and make lookups extremely fast. This concept is discussed quite extensively, but Alex Debrie (an AWS Data Hero) has a blog post that covers these ideas quite well.

Since we want to enable queries that return IP prefixes based on AWS region and service. In order to do that, we’ll format the data with both a primary key (PK) and sort key (SK), as well as a synctoken that is completely separate from the IP prefixes. This synctoken will enable us to remove all IP prefixes that have the synctoken because we’ll be creating a global secondary index using the synctoken to allow fast queries of the items containing a particular synctoken. We can add non-normalized data to the same DynamoDB table because it isn’t an SQL database, it’s more like a key-value or wide-column store database, generally referred to as NoSQL.

Here is what our primary and secondary key layout will look like:

Lambda Function Overview

We’ll use a Lambda function that gets triggered by an SNS notification whenever the IP ranges JSON changes. This SNS subscription, Lambda Function, DynamoDB table and all the AWS infrastructure is configured using AWS Cloud Development Kit (CDK.) The Lambda Function itself does one thing. It downloads the JSON, cycles through every item, and adds it to the DynamoDB table.

AWS CDK configuration

The configuration used to create the Lambda Function and other infrastructure is in the Github repository here. Instructions on how to deploy the infrastructure are also in that repository.

Once deployed the Lambda Function will do nothing until the next time the SNS notification invokes it when the JSON file has changed. Below you can see a screenshot of recent invocations of the function in my environment.

Wrapping Up

In this first part of the blog series, you saw how we used Event Driven Architecture to respond to an event and perform some data processing. In the next section we’ll handle using these IP address ranges to update security groups allowing traffic for certain AWS services.

Thanks for reading and have a great day!

Resources

AWS blog post on “How to Automatically Update Your Security Groups for Amazon CloudFront and AWS WAF by Using AWS Lambda”

Cover photo by Dushyant Kumar on Unsplash

New and Improved Job Searching Resources in 2024

Thu, 11 Jan 2024 08:00:00 -0600

A couple years ago I shared a post that detailed various tech industry job search resources. There are a number of more modern sites available today that have remote tech roles and the ability to filter the results more effectively than places like LinkedIn.

Ventureloop

Ventureloop is a job site that lists primarily venture capital startups who are looking for candidates. If you’re interested in startups (and some of these can be very large and established companies such as DataDog to name just one) take a look at their search site here.

Otta

Otta requires you to sign up and fill out a questionaire of what you’re looking for in a company and role. For instance the size of the company, the technologies they use and whether you’re looking for a remote position are all things you can add to an allow or deny list. Then they filter roles and present them to you so that you can have roles that match what you’re looking for.

The company descriptions are detailed (and written by the Otta staff not just the company PR materials) and they have an indicator of how quickly they get back to candidates, as well as how much growth they’re expecting in the next year. All helpful information to have about a prospective new company.

Climatebase

If you’re interested in tech jobs (or any roles) in the climate tech industry, here’s your chance to do something for the environment.

Check it out here.

StillHiring Today

With the subtitle of “WHO THE FRIGGIN’ FRIG IS HIRING RN!?”, and using a giant airtable sheet listing actively hiring companies, StillHiring Today is a unique offering. They have a lot of entries but you have to weed through the table manually, and they don’t list positions just the company name and what types of open roles(by job function) they have. It’s still helpful to find companies that you are interested in working for, and know that they’re actively hiring.

Check out the list.

Cover photo by lucas Favre on Unsplash

Bundling Go Lambda Functions with the AWS CDK

Thu, 21 Dec 2023 18:22:00 -0500

Recently the Lambda Go runtime has changed from using the Go 1.x managed runtime to using the provided runtimes which have been historically used for custom runtimes (i.e. Rust.) The former go1.x runtime is being deprecated on January 8, 2024 (quite soon) and the new runtimes provided.al2023 or provided.al2 are expected to be used.

With the introduction of these new runtimes, all of our Go binaries must now be called bootstrap and be located at the root of the zip file used to deploy the function.

If we have two functions built by separate function code, in the go1.x runtime the naming convention would allow us to name them differently (i.e. create vs. list). Now we must name both function handlers bootstrap, potentially causing confusion during the bundling process.

In today’s blog post, we’ll describe how to bundle two different Go functions with the same name in a single CDK stack.

AWS CDK Deployment Using Typescript

I’ll be using Typescript as the language for the AWS CDK deployment. It might be strange to mix languages for CDK deployment (Typescript) and the Lambda function (Go), but CDK uses Typescript as its main language and most examples are written in that. The reason for using Go in this example is because many deployments need to use compiled languages for task duration and performance (not necessarily to improve cold start time.)

Source Organization

As described earlier, in the past we could have kept the names of the Go function source files in the same directory and compiled them separately to different file names. Now that the handler file name has to be the same (bootstrap), they can’t be output to the same directory any longer.

The layout of the files now looks like the following, with each function in its own subdirectory in the lib/functions directory:

Bundling Go Functions

Now that we have the function code in separate directories, we can bundle them using the Alpha CDK construct for a Go Function, shown below.

export class BasicGoLambdaStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const table = new TableV2(this, 'Table', {
        partitionKey: { name: 'id', type: AttributeType.STRING },
        removalPolicy: RemovalPolicy.DESTROY,
        });

    const createFunction = new GoFunction(this, 'CreateFunction', {
      entry: 'lib/functions/create',
      environment: {
        "DYNAMODB_TABLE": table.tableName
      },
      logRetention: RetentionDays.ONE_WEEK,
    });

    const listFunction = new GoFunction(this, 'ListFunction', {
      entry: 'lib/functions/list',
      environment: {
        "DYNAMODB_TABLE": table.tableName
      },
      logRetention: RetentionDays.ONE_WEEK,
    });
...

The above configuration is obviously fairly straightforward. But before I found the aws-lambda-go-alpha module, I attempted (mostly unsuccessfully) to use the aws-lambda-function Function construct which required using the Code.fromAsset method with the ILocalBundling interface. This was extremely uncooperative, I found multiple blogs and suggestions on how to configure this, and in the end I did get it working, but the solution I came up with was ugly and actually packaged the Go source file in the zip file with the bootstrap handler. Not cool.

It’s great that the CDK now supports Go Lambda functions as a full blown construct. While NodeJS functions have been supported for much longer it makes sense to have Go supported the same way, because the aggravation I endured packaging these Go handlers with the standard Lambda Function construct was a serious pain in the ass.

Verifying the Output

Now that we have these defined, we can package the artifacts using cdk synth and verify that the bootstrap binary was created for both functions.

We can see here that the first messages output from the cdk synth command are the two functions being bundled:

$ cdk synth
Bundling asset BasicGoLambdaStack/CreateFunction/Code/Stage...
Bundling asset BasicGoLambdaStack/ListFunction/Code/Stage...
Resources:
...

And when we look in the cdk.out directory which is where our intermediate CDK files are placed during packaging, we see that the two Lambda Function output directories itemized in the stack asset JSON file (called BasicGoLambdaStack.assets.json) have binaries in them (ignore the third .js asset, that’s for the cloudwatch logs):

$ tree cdk.out/
cdk.out/
├── asset.4e26bf2d0a26f2097fb2b261f22bb51e3f6b4b52635777b1e54edbd8e2d58c35
│   └── index.js
├── asset.b0bae29696f98febe5e6a655c4f61466ad71ebfa0071b46a15e917a1d307333c
│   └── bootstrap
├── asset.f153e459ffb70033083e7507aaa06c00f4b13a792fad71433c11b5f050078e74
│   └── bootstrap
├── BasicGoLambdaStack.assets.json
├── BasicGoLambdaStack.template.json
├── cdk.out
├── manifest.json
└── tree.json

4 directories, 8 files

After running cdk synth to verify things, we can deploy with cdk deploy.

Success

Now we can execute the functions and see them working as expected:

$ curl -X POST -H "Content-Type: application/json" https://someurl.execute-api.us-west-2.amazonaws.com/todos --data '{ "title": "stuff", "details": "really some stuff"}'
{"id":"80cf566a-937b-4aef-98ef-48b84ee75c01","title":"stuff","details":"really some stuff"}

$ curl -X GET https://someurl.execute-api.us-west-2.amazonaws.com/todos
[{"id":"80cf566a-937b-4aef-98ef-48b84ee75c01","title":"stuff","details":"really some stuff"}]

Summary

The new requirements for using the provided runtimes for Go Lambda Functions poses some minor challenges (especially if you have to migrate already running Go 1.x Lambda Functions) but thankfully the aws-lambda-go-alpha module makes it much more straightforward than bundling these manually.

I hope you’ve gotten a good understanding of how to bundle Go Lambda functions by reading this. I know I learned a lot, and this will come in handy as I use Go for Lambdas going forward.

Cover photo by Jesse De Meulenaere on Unsplash

A Gentle Introduction to AWS Lambda

Tue, 01 Feb 2022 08:00:00 -0600

You’ve probably heard of AWS Lambda and serverless by now. But what is Lambda all about? The short definition of AWS Lambda is a “Functions as a Service” (FaaS) technology. The longer and more complicated answer is that Lambda is a lightweight runtime that requires no infrastructure to be defined by the developer. FaaS allows developers to build software features quickly with less emphasis on the question of where and how they run. This allows for more focus on the business logic that users interact with. Functions as a Service is the most recent innovation in the long line of virtualization technologies. Compute virtualization started with physical servers, then moved to virtual machines (VMs), which grew into Infrastructure as a Service (IaaS). Platforms as a Service came next with companies like Heroku allowing developers to deploy without caring about infrastructure specifics. Now we’re arriving at FaaS, which promises to abstract all these details away. Here are a couple good descriptions of these origins and where we are as an industry today.

Lambda vs. Serverless Terminology

The terms serverless and AWS Lambda are often used interchangeably, but they aren’t exactly the same thing. Serverless refers to a collection of tools that require no infrastructure configurations from the developer, including databases, queues, functions and gateways as a service. AWS Lambda is only the FaaS component in this list. That means there are many more products such as API gateways, message queueing systems, notification tools, and serverless databases that fill out the serverless landscape. In this blog post we’ll only work with a couple of these, namely AWS Lambda and AWS API Gateway for our demonstration.

FaaS Platforms

There are many FaaS providers in the industry. From the big players like Google, IBM, and AWS to the smaller ones like the open source OpenWhisk and OpenFaaS, they all provide similar functionality (generally) with different developer experiences.

There are also Edge computing FaaS providers, which instead of executing the code in one central data center, run the code in a data center nearest the user, known as the “Edge”. The players in this area of serverless technology are companies like Cloudflare, Lambda@Edge (AWS), and Fastly.

AWS Lambda in Plain English

After reading the above definition of serverless, Functions as a Service, and AWS Lambda, your understanding might still be cloudy (no pun intended). I’ll attempt to explain it in more simple terms.

The description on the AWS documentation website says “Lambda is a compute service that lets you run code without provisioning or managing servers.” The essence of Lambda is that AWS built all the scaffolding to allow you and I to send them a single piece of code, which they will run for us.

We don’t need to build servers or container images in order to run this single piece of code. All we have to think about is writing code, and then a small SAM template file which defines the requirements of the deployment, and finally shipping it.

Is it Really a Single Function?

In order to wrap your brain around this idea of functions as a service, think of a single method or function in a small program. The function is called by another function, and may return a result.

Below you see a simple Python program in which there is a parent function (main()) calling a child function (childfunction()).

def childfunction(inputstring):
    if inputstring == "hello":
        return "world"
    return "hi"


def main():
    value = childfunction("hello")
    print(f"value: {value}")


if __name__ == "__main__":
    main()

The output of this program looks like this:

1
2

$ python hello.py
value: world

At it’s simplest form, AWS Lambda can be thought of as a single child function being called by a parent function. AWS owns the parent function, which executes your function (the child function) in it’s own virtual environment.

Your objective during the rest of this blog post is to write, deploy and execute a single Lambda function.

Let’s Write One Bite Sized Function

For this example, we’ll write a small function that does nothing particularly important, but demonstrates the mechanics of running a function in AWS Lambda. You are probably familiar with shell environment variables. In your terminal you can print out the environment variables defined in your session, or even create new environment variables.

The below terminal command prints out an environment variable showing the user name that you are logged in as:

1
2

$ echo $USER
mbacchi

What Will the Function Do?

The function we’ll deploy will print out the default runtime environment variables in the AWS Lambda execution environment.

Similar to the terminal session above, our Lambda function can access the default environment variables such as AWS_LAMBDA_FUNCTION_NAME, AWS_LAMBDA_LOG_GROUP_NAME, or TZ. But how do we see the output of print statements in a Lambda? Anything that gets printed to STDOUT will be logged to CloudWatch Logs using the log group name of the function.

How Does Lambda Work?

The concepts that you’ll need to understand are described in the AWS Lambda documentation.

Here are some of the most important details you’ll need to write your Lambda functions:

Invocation: The function is invoked by one of the triggers that AWS provides. It can also be triggered by cron like scheduling events.
Entrypoint: Your Lambda function needs to implement a handler which is the method (or function) that gets called by AWS when triggered.
Environment: The handler is passed an event object and context object from AWS that provides your function both data to be processed and context (information) about how it was called and other clues that you can use in your code.
Response: In certain cases you will want to syncronously return a value from your function, but that is optional. If you are writing an API backend you will return a value through API Gateway. If instead you are performing some scheduled data processing you may not return a value.

Background on Associated Serverless Components

We mentioned earlier how serverless isn’t just AWS Lambda, and in this project we’ll use API Gateway and CloudWatch Logs, but only to enable us to show how our Lambda function executes.

AWS API Gateway is, as the name suggests, an API gateway product. But it also allows for more than just API calls. For example it enables generic HTTP/HTTPS traffic, REST APIs and Websocket APIs. You can host a standard web server with API Gateway and Lambda, you aren’t limited to just APIs with API Gateway.

CloudWatch Logs is a logging product from AWS that allows all other AWS products to log their activity to a central location. In our example we’ll be looking at the output of our Lambda function to see the environment variables available in the Lambda itself during runtime.

We’re not going to cover these in any more detail because they’re not the main focus of this post.

Sample Lambda Function

For this example deployment, we’ll use a GitHub repository that has our AWS Lambda function defined. The Lambda function in this repo was written with the intention of being used for demonstration purposes, and as a simple function that can be deployed when testing out CI/CD or the many serverless deployment tools available today (such as Serverless Framework, Cloud Development Kit (CDK) etc.)

Deploying a Lambda Function with AWS SAM

AWS SAM is the Serverless Application Model, which allows you to write a template describing the Lambda function, and any related resources you need to run your Lambda. The SAM template uses the YAML markup language and is based on AWS Cloudformation. After reading the template, SAM performs the deployment for you, and you can watch Cloudformation events to identify what the outcome was.

We are going to use the template in the above repository to deploy the function. I won’t go into detail breaking down all of the items in the template for now. That information can be found in the documentation above or in other blogs. But I do want to highlight the two resources used here, the LambdaEnvVarsFunction which is of the type AWS::Serverless::Function and the LambdaLogGroup, which is the definition of our AWS::Logs::LogGroup CloudWatch log group, where our Lambda output will be collected.

Install SAM CLI

You’ll need to install a few components before running SAM. I would recommend running this in a Python virtual environment (venv), and I will provide basic steps here to set that up.

From your terminal run these commands (this is on a Linux system):

Step 1: Make sure you have Python 3.8 (you may have to install this first, on Fedora Linux the command would be sudo dnf install python3.8)
1 2

$ which python3.8 /usr/bin/python3.8

Step 2: Create a Python 3.8 virtual environment

1
2
3

$ python3.8 -m venv venv38
$ ls venv38
bin  include  lib  lib64  pyvenv.cfg

Step 3: Activate the virtual environment

$ . venv38/bin/activate
$ which python
~/FAKE_PATH/lambda-env-vars7/venv38/bin/python
$ which pip
~/FAKE_PATH/lambda-env-vars7/venv38/bin/pip

Step 4: Install SAM CLI NOTE: This installs Docker in the Python virtual environment

$ pip install aws-sam-cli
Collecting aws-sam-cli
  Using cached aws_sam_cli-1.37.0-py3-none-any.whl (5.0 MB)
Collecting boto3==1.*,>=1.18.32
  Using cached boto3-1.20.46-py3-none-any.whl (131 kB)
...

Now you can move on to the next step, where we run the Lambda function locally.

Running Your Lambda Function with SAM Local

Before we deploy to AWS itself, we want to test our code by invoking the Lambda function locally. What is this magic you ask? SAM provides a utility command called sam local invoke which allows your function to run locally in a Docker container to test that it runs as expected before deploying to AWS.

NOTE: We assume you haven’t left the terminal session in step 4 above. If you have, activate your Python virtual environment as in step 3 above.

Run sam local invoke as described in the Git repository (this uses test events supplied with Lambda function in the repository)

$ sam local invoke LambdaEnvVarsFunction --event events/event.json
Invoking app.lambda_handler (python3.8)
Skip pulling image and use local one: public.ecr.aws/sam/emulation-python3.8:rapid-1.37.0-x86_64.

Mounting /home/mbacchi/data/repos/mbacchi/lambda-env-vars7/.aws-sam/build/LambdaEnvVarsFunction as /var/task:ro,delegated inside runtime container
START RequestId: f97086cf-e8b6-4d9f-b2bc-b627ca5961e6 Version: $LATEST
Hello there this is the body: {'_HANDLER': 'app.lambda_handler', 'AWS_REGION': 'us-east-1', 'AWS_EXECUTION_ENV': 'AWS_Lambda_python3.8', 'AWS_LAMBDA_FUNCTION_NAME': 'LambdaEnvVarsFunction', 'AWS_LAMBDA_FUNCTION_MEMORY_SIZE': '128', 'AWS_LAMBDA_FUNCTION_VERSION': '$LATEST', 'AWS_LAMBDA_LOG_GROUP_NAME': 'aws/lambda/LambdaEnvVarsFunction', 'AWS_LAMBDA_LOG_STREAM_NAME': '$LATEST', 'LANG': 'en_US.UTF-8', 'TZ': ':/etc/localtime', 'LAMBDA_TASK_ROOT': '/var/task', 'LAMBDA_RUNTIME_DIR': '/var/runtime', 'PATH': '/var/lang/bin:/usr/local/bin:/usr/bin/:/bin:/opt/bin', 'LD_LIBRARY_PATH': '/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib', 'PYTHONPATH': '/var/runtime', 'AWS_LAMBDA_RUNTIME_API': '127.0.0.1:9001'}
END RequestId: f97086cf-e8b6-4d9f-b2bc-b627ca5961e6
REPORT RequestId: f97086cf-e8b6-4d9f-b2bc-b627ca5961e6	Init Duration: 0.11 ms	Duration: 100.72 ms	Billed Duration: 101 ms	Memory Size: 128 MB	Max Memory Used: 128 MB	
{"statusCode": 200, "body": "{\"_HANDLER\": \"app.lambda_handler\", \"AWS_REGION\": \"us-east-1\", \"AWS_EXECUTION_ENV\": \"AWS_Lambda_python3.8\", \"AWS_LAMBDA_FUNCTION_NAME\": \"LambdaEnvVarsFunction\", \"AWS_LAMBDA_FUNCTION_MEMORY_SIZE\": \"128\", \"AWS_LAMBDA_FUNCTION_VERSION\": \"$LATEST\", \"AWS_LAMBDA_LOG_GROUP_NAME\": \"aws/lambda/LambdaEnvVarsFunction\", \"AWS_LAMBDA_LOG_STREAM_NAME\": \"$LATEST\", \"LANG\": \"en_US.UTF-8\", \"TZ\": \":/etc/localtime\", \"LAMBDA_TASK_ROOT\": \"/var/task\", \"LAMBDA_RUNTIME_DIR\": \"/var/runtime\", \"PATH\": \"/var/lang/bin:/usr/local/bin:/usr/bin/:/bin:/opt/bin\", \"LD_LIBRARY_PATH\": \"/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib\", \"PYTHONPATH\": \"/var/runtime\", \"AWS_LAMBDA_RUNTIME_API\": \"127.0.0.1:9001\"}"}

Deploy to AWS

NOTE: Before deploying you need to setup your AWS credentials properly.

In order to deploy the Lambda function, we’ll follow the steps in the lambda-env-vars repository:

Step 1: Run sam build

$ sam build
Building codeuri: /home/mbacchi/data/repos/mbacchi/lambda-env-vars7/env_vars runtime: python3.8 metadata: {} architecture: x86_64 functions: ['LambdaEnvVarsFunction']
Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Test Function in the Cloud: sam sync --stack-name {stack-name} --watch
[*] Deploy: sam deploy --guided

Step 2: Run sam package (this creates an S3 bucket, which must be removed later)

$ sam package --output-template-file packaged.yaml --resolve-s3
  Creating the required resources...
  Successfully created!

    Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-61z4v277mwy0
    A different default S3 bucket can be set in samconfig.toml
    Or by specifying --s3-bucket explicitly.
Uploading to eb6c7534d67a5b044653e48f2802a548  452695 / 452695  (100.00%)

Successfully packaged artifacts and wrote output template to file packaged.yaml.
Execute the following command to deploy the packaged template
sam deploy --template-file /home/mbacchi/data/repos/mbacchi/lambda-env-vars7/packaged.yaml --stack-name <YOUR STACK NAME>

Step 3: Run sam deploy

$ sam deploy --template-file packaged.yaml --region us-east-2 --capabilities CAPABILITY_IAM --stack-name lambda-env-vars --resolve-s3

    Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-61z4v277mwy0
    A different default S3 bucket can be set in samconfig.toml
    Or by specifying --s3-bucket explicitly.

  Deploying with following values
  ===============================
  Stack name                   : lambda-env-vars
  Region                       : us-east-2
  Confirm changeset            : False
  Disable rollback             : False
  Deployment s3 bucket         : aws-sam-cli-managed-default-samclisourcebucket-61z4v277mwy0
  Capabilities                 : ["CAPABILITY_IAM"]
  Parameter overrides          : {}
  Signing Profiles             : {}

Initiating deployment
=====================
Uploading to 77524ed205c6d82d5487433831c24c0f.template  1635 / 1635  (100.00%)

Waiting for changeset to be created..

CloudFormation stack changeset
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Operation                                                  LogicalResourceId                                          ResourceType                                               Replacement                                              
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Add                                                      LambdaEnvVarsFunctionLambdaEnvVarsPermissionProd           AWS::Lambda::Permission                                    N/A                                                      
+ Add                                                      LambdaEnvVarsFunctionRole                                  AWS::IAM::Role                                             N/A                                                      
+ Add                                                      LambdaEnvVarsFunction                                      AWS::Lambda::Function                                      N/A                                                      
+ Add                                                      LambdaLogGroup                                             AWS::Logs::LogGroup                                        N/A                                                      
+ Add                                                      ServerlessRestApiDeploymentcaa6ada684                      AWS::ApiGateway::Deployment                                N/A                                                      
+ Add                                                      ServerlessRestApiProdStage                                 AWS::ApiGateway::Stage                                     N/A                                                      
+ Add                                                      ServerlessRestApi                                          AWS::ApiGateway::RestApi                                   N/A                                                      
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Changeset created successfully. arn:aws:cloudformation:us-east-2:592431548397:changeSet/samcli-deploy1643607514/85a31faf-1358-47ff-9eeb-4428020bb482


2022-01-30 22:38:46 - Waiting for stack create/update to complete

CloudFormation events from stack operations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ResourceStatus                                             ResourceType                                               LogicalResourceId                                          ResourceStatusReason                                     
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CREATE_IN_PROGRESS                                         AWS::IAM::Role                                             LambdaEnvVarsFunctionRole                                  -                                                        
CREATE_IN_PROGRESS                                         AWS::IAM::Role                                             LambdaEnvVarsFunctionRole                                  Resource creation Initiated                              
CREATE_COMPLETE                                            AWS::IAM::Role                                             LambdaEnvVarsFunctionRole                                  -                                                        
CREATE_IN_PROGRESS                                         AWS::Lambda::Function                                      LambdaEnvVarsFunction                                      -                                                        
CREATE_IN_PROGRESS                                         AWS::Lambda::Function                                      LambdaEnvVarsFunction                                      Resource creation Initiated                              
CREATE_COMPLETE                                            AWS::Lambda::Function                                      LambdaEnvVarsFunction                                      -                                                        
CREATE_IN_PROGRESS                                         AWS::Logs::LogGroup                                        LambdaLogGroup                                             -                                                        
CREATE_IN_PROGRESS                                         AWS::ApiGateway::RestApi                                   ServerlessRestApi                                          -                                                        
CREATE_IN_PROGRESS                                         AWS::ApiGateway::RestApi                                   ServerlessRestApi                                          Resource creation Initiated                              
CREATE_COMPLETE                                            AWS::ApiGateway::RestApi                                   ServerlessRestApi                                          -                                                        
CREATE_IN_PROGRESS                                         AWS::Logs::LogGroup                                        LambdaLogGroup                                             Resource creation Initiated                              
CREATE_IN_PROGRESS                                         AWS::ApiGateway::Deployment                                ServerlessRestApiDeploymentcaa6ada684                      -                                                        
CREATE_IN_PROGRESS                                         AWS::Lambda::Permission                                    LambdaEnvVarsFunctionLambdaEnvVarsPermissionProd           Resource creation Initiated                              
CREATE_IN_PROGRESS                                         AWS::Lambda::Permission                                    LambdaEnvVarsFunctionLambdaEnvVarsPermissionProd           -                                                        
CREATE_COMPLETE                                            AWS::Logs::LogGroup                                        LambdaLogGroup                                             -                                                        
CREATE_IN_PROGRESS                                         AWS::ApiGateway::Deployment                                ServerlessRestApiDeploymentcaa6ada684                      Resource creation Initiated                              
CREATE_COMPLETE                                            AWS::ApiGateway::Deployment                                ServerlessRestApiDeploymentcaa6ada684                      -                                                        
CREATE_IN_PROGRESS                                         AWS::ApiGateway::Stage                                     ServerlessRestApiProdStage                                 -                                                        
CREATE_IN_PROGRESS                                         AWS::ApiGateway::Stage                                     ServerlessRestApiProdStage                                 Resource creation Initiated                              
CREATE_COMPLETE                                            AWS::ApiGateway::Stage                                     ServerlessRestApiProdStage                                 -                                                        
CREATE_COMPLETE                                            AWS::Lambda::Permission                                    LambdaEnvVarsFunctionLambdaEnvVarsPermissionProd           -                                                        
CREATE_COMPLETE                                            AWS::CloudFormation::Stack                                 lambda-env-vars                                            -                                                        
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

CloudFormation outputs from deployed stack
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Outputs                                                                                                                                                                                                                                   
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Key                 LambdaEnvVarsApi                                                                                                                                                                                                      
Description         API Gateway endpoint URL for Prod stage for Lambda Env Vars function                                                                                                                                                  
Value               https://e4cusxvy7f.execute-api.us-east-2.amazonaws.com/Prod/env_vars/                                                                                                                                                 

Key                 LambdaLogGroup                                                                                                                                                                                                        
Description         Cloudwatch Log Group ARN                                                                                                                                                                                              
Value               /aws/lambda/lambda-env-vars-LambdaEnvVarsFunction-6r0qwkWGeZSp                                                                                                                                                        

Key                 LambdaEnvVarsFunctionIamRole                                                                                                                                                                                          
Description         Implicit IAM Role created for Lambda Env Vars function                                                                                                                                                                
Value               arn:aws:iam::592431548397:role/lambda-env-vars-LambdaEnvVarsFunctionRole-L05B77IT7WK0                                                                                                                                 

Key                 LambdaEnvVarsFunction                                                                                                                                                                                                 
Description         Lambda Env Vars Function ARN                                                                                                                                                                                          
Value               arn:aws:lambda:us-east-2:592431548397:function:lambda-env-vars-LambdaEnvVarsFunction-6r0qwkWGeZSp                                                                                                                     
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Successfully created/updated stack - lambda-env-vars in us-east-2

The sam deploy command has deployed the Lambda function to AWS and returned the API Gateway endpoint which we can use to invoke the Lambda with an API call. We need the API Gateway URL from the above output, look for the LambdaEnvVarsApi key. In our example above, this is:

https://e4cusxvy7f.execute-api.us-east-2.amazonaws.com/Prod/env_vars/

GET Request to Invoke the Lambda Function

Now that we have the API Gateway URL (also known as the endpoint,) we can use a tool like curl to perform an HTTP GET request against the endpoint. Because we defined the Events stanza with a Type of Api, and Method get, SAM has deployed an API Gateway resource to trigger our Function. (More detailed discussion on this implicit API Gateway resource here).

We will use the command curl to perform a GET request as below:

1
2

$ curl -X GET https://e4cusxvy7f.execute-api.us-east-2.amazonaws.com/Prod/env_vars/
{"_HANDLER": "app.lambda_handler", "AWS_REGION": "us-east-2", "AWS_EXECUTION_ENV": "AWS_Lambda_python3.8", "AWS_LAMBDA_FUNCTION_NAME": "lambda-env-vars-LambdaEnvVarsFunction-6r0qwkWGeZSp", "AWS_LAMBDA_FUNCTION_MEMORY_SIZE": "128", "AWS_LAMBDA_FUNCTION_VERSION": "$LATEST", "AWS_LAMBDA_LOG_GROUP_NAME": "/aws/lambda/lambda-env-vars-LambdaEnvVarsFunction-6r0qwkWGeZSp", "AWS_LAMBDA_LOG_STREAM_NAME": "2022/01/31/[$LATEST]c2b9b9c953154c8daa7c9b2c0637b70e", "LANG": "en_US.UTF-8", "TZ": ":UTC", "LAMBDA_TASK_ROOT": "/var/task", "LAMBDA_RUNTIME_DIR": "/var/runtime", "PATH": "/var/lang/bin:/usr/local/bin:/usr/bin/:/bin:/opt/bin", "LD_LIBRARY_PATH": "/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib", "PYTHONPATH": "/var/runtime", "AWS_LAMBDA_RUNTIME_API": "127.0.0.1:9001"}

On line 2, we can see the response JSON, which is what the Lambda function returned to us via the API Gateway. And we see all the environment variables that we had expected, just as in the sam local invoke command earlier.

But how can we observe the output of the print statement at line 50?

To do that we look at the CloudWatch log output. Browse to your AWS console, and search for the Cloudwatch log group name that was in the Outputs section of the sam deploy command output.

In our case the LambdaLogGroup name was: /aws/lambda/lambda-env-vars-LambdaEnvVarsFunction-6r0qwkWGeZSp

When I browse there in CloudWatch, I see the “Hello there” line in the output, which was sent to CloudWatch because we performed a print to standard output within the Lambda function code:

What Just Occurred? What’s Next?

Congratulations, you just deployed and invoked your first Lambda function.

Take some time to examine the output of the function in CloudWatch Logs, as well as what was returned during your curl command.

Challenge 1: If you’ve heard of the Postman tool, you can also call the API endpoint from there. Give it a shot.

Challenge 2: You can display the HTTP headers when calling the API endpoint. How would you see the headers with the curl command? How would you see them with Postman?

Challenge 3: Read more about Lambda functions, SAM, Serverless Framework, CDK. Build your own Lambda and deploy it. Let me know how it goes on Twitter @fshwsprr.

Clean Up AWS Resources

When you are done playing around with this Lambda function, run these commands to remove your resources:

aws cloudformation delete-stack --stack-name lambda-env-vars --region us-east-2
aws s3 rb s3://aws-sam-cli-managed-default-samclisourcebucket-61z4v277mwy0 --force NOTE: This requires you to look at the output of your sam package or sam deploy command above to get the name of the S3 bucket to remove!
Finally, you can deactivate your python virtual environment with the command: deactivate

Summary

If you’ve read this far, the important takeaways about AWS Lambda in my opinion, are:

The concept that Lambda functions are only one part of the cloud serverless landscape.
Understanding invocation options and how your single function gets called by AWS.
The fact that you can locally invoke the function in SAM Local before you deploy to AWS.
Anything you print inside your function will be written to CloudWatch Logs (assuming you created a Log Group for your Lambda.)

Thanks for reading! Enjoy your serverless and Lambda journey!

Cover photo by Avel Chuklanov on Unsplash

How to Avoid CIDR Conflicts in AWS Sagemaker Notebooks

Wed, 12 Jan 2022 08:00:00 -0600

Networking can sometimes be quite complicated. Despite the oft repeated joke that “It’s always DNS”, sometimes your problem is even more difficult to diagnose than DNS.

According to Wikipedia, Classless Inter-Domain Routing (or CIDR) “is the method for allocating IP addresses and for IP routing” on the internet and on private networks. If there are conflicts in two networks’ CIDR ranges, it can cause headaches that make DNS problems look like childs play.

This is a story about how I unknowingly created a CIDR conflict, and I hope it will be useful to help you avoid CIDR conflicts in Sagemaker Notebooks in the future.

What is AWS Sagemaker

AWS Sagemaker is a popular Machine Learning platform that provides preconfigured environments which allow you to begin training ML models quickly.

Sagemaker Notebooks provide the platform to build and train your models. Under the covers an AWS Sagemaker Notebook is an EC2 instance running a Jupyter notebook packaged with numerous libraries and algorithms. This prevents users from having to configure their own compute, storage and libraries themselves.

A benefit of Sagemaker Notebooks is that you can run generic Python commands alongside your more complex ML code within the Jupyter notebooks. For example, while testing out some connectivity issues recently I was able to use the ubiquitous Python Requests library to perform an HTTP GET request within the Sagemaker Notebook to verify I was able to communicate with a webserver I launched in another AWS account.

Creating a VPC

When I create new projects or resources in AWS accounts, I find it useful to create a new VPC just for the resources in that specific project. This provides network isolation, security and connectivity to the specific components needed for the project. During VPC creation you need to provide a CIDR range that the VPC will use.

CIDR ranges must be built using the private address spaces that IETF RFC 1918 dictates. This means that you choose the CIDR range, but it must be in the 192.168.0.0 range for 256 Class C networks, or 172.16.0.0 for 16 Class B networks, or 10.0.0.0 for a Class A network.

The day I was working on these Sagemaker resources, I randomly chose the range 172.17.0.0 for this VPC, and this fateful decision would come to haunt me.

Sagemaker Notebooks

As described briefly above, Sagemaker Notebooks run on an AWS EC2 instance, and provide preconfigured tools for ML use cases. Part of that tooling is a wonderful open source product called Jupyter Notebook, which is a web based Python experimentation environment. Jupyter Notebooks run inside Docker on the Sagemaker Notebook EC2 instance.

When you configure Docker on an EC2 instance or any workstation/laptop/Linux machine, it sets up virtual networking of its own to provide network connectivity to the Docker containers. This local virtual networking sets up a bridge network interface that allows communication between the Docker network and the local machine as well as the internet if allowed. The default Docker bridge network uses the range 172.17.0.0 for the docker0 network interface.

CIDR Range Conflict Causes Connectivity Problem

Now that the background is set, you can see why a connectivity problem was destined to occur on the Sagemaker Notebook instance. Since the VPC CIDR range utilized 172.17.0.0, that meant all EC2 instances or network interfaces created in that VPC would be provided with an IP address within the 172.17.0.0 range.

Because the Jupyter Notebook running in Docker on the Sagemaker Notebook EC2 instance used the Docker bridge network 172.17.0.0 and listened for all traffic being sent to that destination, it superceded the traffic sent to the outside world. Any network packets sent to the default route (0.0.0.0) via the default gateway (172.17.112.1) were actually intercepted by the same Docker bridge network, and not sent outside the bridge network.

This is shown via the route -n command on the Sagemaker Notebook EC2 instance terminal:

sh-4.2$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.17.112.1    0.0.0.0         UG    10002  0        0 eth2
169.254.0.0     0.0.0.0         255.255.255.0   U     0      0        0 veth_def_agent
169.254.169.254 0.0.0.0         255.255.255.255 UH    0      0        0 eth0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.17.112.0    0.0.0.0         255.255.240.0   U     0      0        0 eth2
172.18.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-9bb29e923d05
192.168.0.0     0.0.0.0         255.255.0.0     U     0      0        0 bridge0

Was This Unexpected?

What could have been done to avoid this you ask?

As I mentioned earlier, the default Docker bridge network uses the range 172.17.0.0 for the docker0 network interface. It’s fine that AWS simply used that default without modifying it, but that implementation detail should have been documented in the Sagemaker Notebook documentation.

Ideally my suggested resolution is different. If the AWS end user (aka customer) tries creating a Sagemaker Notebook in a VPC that has a CIDR range the same as the default Docker bridge range (172.17.0.0), they should change the Docker bridge network range to something else to avoid a conflict.

Wrap Up

Ultimately the responsibility was on me to create the VPC and the Sagemaker Notebook and make sure they worked correctly. But this was made infinitely more difficult by AWS making choices themselves. Because AWS Sagemaker did not document the Docker bridge network range they used, and did not programatically configure an alternate Docker bridge network range if the customer selected the same CIDR range for their VPC as the default Docker bridge network, I was left troubleshooting a nasty problem.

Please be aware this is still an open issue. (I say that in a figurative way not literal. There is no AWS Support ticket open, and they have not committed to fixing this in any way.)

If you want to avoid CIDR conflicts on Sagemaker Notebooks in your AWS VPC, make sure you use a CIDR range other than 172.17.0.0.

Photo by fabio on Unsplash

Master your Tech Job Search in 2021

Sun, 06 Jun 2021 08:00:00 -0600

Over the last few years I’ve provided information on how to approach a job search to friends and coworkers. I typically send an email with this information, but I think it’s useful to a broader audience, so I’ll provide it here in a blog post.

This is certainly not meant to be an exhaustive discussion of every aspect of job searching. There are many other resources out there on the internet that have great recommendations. What I’ve written here just happens to be information I thought was useful and not well covered as far as I’ve seen.

Read on to learn how to make your job search more effective.

Get organized

Organizing your job search can be immensely beneificial. I’ve found this method by VM Brasseur to be a good pattern. Using a kanban board like Trello is imperative to keep track of every role you’re considering. I’ve done this in the past and it helps to tabs on the status of each role you’ve applied to. I’ve even kept notes on mutual interests that I discussed with interviewers so that I could remember in case we talked again.

Where to look for remote roles

Since I’ve worked remotely (or as part of a distributed workforce) for 12+ years, I primarily look for roles that allow for remote work. Prior to the pandemic, finding companies that were remote friendly could be challenging. This has changed somewhat, but the new challenge is identifying which companies are willing to continue allowing remote work after life is back to normal. To that end, here are some sites I’ve used to search for roles that are more remote friendly:

If you’re looking to work with a company that doesn’t profit off the ills of society, such as ad tech or surveillance capitalism, there are a few sites that list companies who do good in the world.

If you are interested in the startup world, you can find positions via their venture capital (VC) funding companies:

Finally, there are lists of remote companies all over the place — a cursory search would probably surface even more:

Looking for roles in Twitter posts

The above resources are all sites where you can actively search their list of roles. I’ve found another productive method is to tap your extended network, specifically people you follow on social media. I have experience searching on Twitter, and have gotten many interesting leads.

You can perform an actual Twitter search, but I prefer passively watching my feed for Tweets mentioning that someone has roles available. These Tweets are advertising open roles, so if the job description interests you, contact them via DM (or whatever contact method they describe) to inquire about the role or begin the application process. Even if the specific role isn’t something you are excited about, follow the person and they may post more roles in the future or retweet jobs from people in their network.

Another method is keeping an eye out for someone who Tweets asking for leads. You’ll often see responses from amazing and helpful people that are direct links to job descriptions. Look through these replies to see if there are any roles that interest you, and apply accordingly. I have even seen roles posted to Tweet threads like these days later.

One example of Twitter activity like this could be seen in the 2020 Mozilla layoff, which affected 250 Mozillians. People who wanted to help started using the #MozillaLifeboat hash tag, which had hundreds of tweets per day during that time. Obviously if you can help someone else this way, please do, but it can also be a source of roles that you might not be aware of if you’re doing a job search yourself.

Research to find top companies

There are mediocre companies in every industry, but there are also exceptional companies. If you already know the best companies in your industry, then start looking for roles on their websites. If you feel like you cannot meet their requirements, apply anyway and see if you may be underestimating your abilities. I’m also a fan of finding a really great job description that you don’t yet qualify for and using that to set goals for yourself for 6 months or a year from now. Work on building those skills you are missing and apply to that role or a similar one when you level up and meet your goals.

If you don’t know great workplaces, use your network to find out which ones are held in high regard by people you respect. Then start searching those websites for roles that intrigue you.

Apply to roles just a little bit above your experience level

I recommend that you apply for roles that have requirements that are slightly beyond your experience. Not having a few technologies that a job description lists is not a disqualifier. This should be viewed as a challenge instead. There are three reasons for this.

First, it is important to always be growing and investing in your skills. There is no better way to do this than learning a technology you haven’t used before on the job. Not meeting every bullet point is okay. Be honest during interviews, stating that you haven’t yet been exposed to the particular tool or technology before and would love to have the opportunity. You can also play with it on your own time to learn some of the basics. Then during the interview, you can say something like “I’ve used it for personal projects but haven’t supported it in a production environment.”

Second, it is best in my experience to avoid working in a role doing the exact same things you have done before. While this may feel more comfortable, I find these jobs to quickly become boring. Look for a position that has new technologies that will be interesting to learn. This enables you to add new things to your resume. You want to grow into the position; having skills for every line item listed in the description often ensures you will not love the job in the long term.

Third, when managers write job descriptions, they often list the skills that describe their “dream” candidate. The employee who may have just left the company and created a vacancy may not have ticked all those boxes. There are often few people who meet every requirement, but HR or management wants to list every possible skill that might be needed now or desired for a future project. Think of the job listing as aspirational, both for yourself and the hiring company. Aspire to work in a role that is just above your current capabilities.

Locate free mentors or coaching on Twitter

I have come across very generous people on Twitter who have made offers of free career advice, career coaching, or mentoring. For example, one person occasionally offers 30 minute meetings where you can pick her brain about any of these topics. I’ve seen Tweets asking mentors willing to volunteer their time to respond to her Tweet, and the thread grew to dozens of people offering their time to anyone who needs advice. This is a really special quality that we often forget about when thinking about social media, but it is one of the most selfless things I’ve seen on Twitter and can provide a great opportunity for newer job seekers to talk to seasoned veterans.

I myself have used these mentors to both build my network and get advice on my resume, job search, and more. I won’t share the names of these people because they haven’t given me permission to do so, but to find similar offers, you can search Twitter for the words ‘mentor’ and ‘coaching’ related to the words ‘job searching,’ ‘applying,’ ‘resume,’ ’negotiating,’ and ‘switching.’ You could also search on the technologies or programming languages you use or desire to use in a new role, and you will likely find someone willing to mentor you.

That all being said, don’t pay for this — at least not early in your career or your search. If you are further along in your career you could employ a career coach, but I don’t think it’s necessary for your first job.

Use your network

Your social media network (let’s just say Twitter and LinkedIn for now) is a great way to find roles, mentors, encouragement, and camaraderie. As mentioned above, you can search for roles this way, but you can also ask your ‘whisper network’ about whether a team or a manager or a company is likely to be a good fit for you. Your social media connections might make introductions for you or even offer to setup meetings with someone you need to know to be considered for a position. It’s a good idea to be on the lookout for ways you can leverage your network like this.

But (and this is where we put the social in social media) you should also be offering to do this for anyone you run across that you might be able to help. If you’ve benefitted from this type of relationship, please be willing to give back (or pay it forward, whichever floats your boat.)

Resume building and writing

I won’t go into how to write your resume because there are many blogs and websites that already cover this. Take a look at my resume here as an example if you like.

I will say quickly that you should work hard to create your own resume building experiences. One way to do this is to get involved with open source projects or any professional project to learn how they operate at an organizational level and build skills that you can add to your resume.

Employers look for these team experiences, not just a laundry list of programming languages and frameworks.

Try to do more than just applying to roles on company websites

Related to the networking topic above, you’ll be more effective if you take a more active approach to the application process. Some coaches recommend reaching out to any contacts you have before you apply to find out information about the hiring manager, team culture, or other nuggets of information that can help you achieve your goal. If you don’t know anyone working at a company you are applying to, coaches suggest researching the company and making a new connection by even cold emailing. The goal of this approach is to find a champion that can help surface your application to the top of the pile of applicants.

Other ways to stand out are to work on open source projects and show your value. Or I’ve heard of people doing other work in advance to prove how interested they are in the company and the position they’re applying for. The most extreme case of this I’ve heard about was someone showing off their skills and suggesting that the company hire them for a role that didn’t even exist yet.

This isn’t easy and isn’t a guarantee, but having a contact help you is certainly going to be more effective than just applying by itself.

Interviewing

I’m not going to discuss interviewing in depth at all, but one great suggestion I’ve seen is to find a mock interviewer to practice. Interviewing successfully (or even without those jitters) is a skill that can be improved.

Find someone you trust, or an organization that is willing to act as the interviewer and schedule a mock interview. Approach it as if it were a real interview process. This will make you more confident in real interviews.

Pay scale and salary negotiation resources

Before you go into your first interview, you should know the target salary you are looking for. As @Latesha_Byrd and @techgirl1908 say, “know your worth”!

This is an amazing resource to help you figure out how people in your position are paid, whether remote or in a specific region:

https://www.levels.fyi/

When negotiating your salary, these blog posts are golden (even though a couple are vintage):

Hiring for certain remote jobs may not be possible in every state

Finally, some companies do not hire for remote positions in certain states. There are two reasons for this (that I’m aware of).

First, employers don’t always have a presence in every state. Therefore they often use Professional Employer Organizations (PEOs) that are HR platforms that provide employee services in states where they don’t have an office. This is common practice for remote friendly companies. Still, sometimes they list the state you live in as a state they don’t hire in. While that’s hard to accept, especially if you like the job description, it’s the way things work these days. Not every company hires in every state.

Second, and this is the more underhanded(or shall we say, disappointing) reason, is because they don’t want to share the pay scale for the position. Colorado recently passed a law that went into effect in January 2021 that required pay transparency. This law said that job postings available to Colorado residents needed to share a salary range. In order to avoid sharing salary ranges, some companies are now disqualifying Colorado residents from these roles so that they don’t run afoul of this law.

While I don’t like this response, its entirely within their right to take this action at the present time. C’est la vie.

Wrapping up

I hope you were able to get some valuable information from this post. I found a recent tweet from Sahil Bloom (@SahilBloom) as I was writing this, and it reinforced a lot of what I have been saying for a couple years. Funny how things occur simultaneously like that.

Find me on Twitter (@fshwsprr) to respond if you have something to say; I’d love to hear from you.

Take care!

#awswishlist Series: More granular service IPs in AWS ip-ranges.json list

Sat, 23 Jan 2021 08:00:00 -0600

This will be the first in a series of posts describing Tweets I’ve sent with the hashtag #awswishlist. Tweets to this #awswishlist hashtag come from anyone who uses AWS and is frustrated with the AWS user experience in some way. These Tweets are often responded to by AWS Support staff on Twitter, indicating they’ll be passing the feedback on to the team responsible for the AWS service. But in my tenure using AWS, I’ve only personally seen one wishlist item get resolved within a reasonable time frame.

This blog series will demonstrate why these #awswishlist items are so disappointing, and the contortions that users are required to perform in order to work around these UX deficiencies.

The straw that broke the DevOps back

So I was working on a project to allow traffic through our CDN for SNS notifications. (I know what you’re thinking. Saying “SNS notifications” is redundant. But if I just said “SNS” the subject isn’t clear enough.) It would be great if I could just create a list of CIDRs for the SNS IP ranges. Wait a minute…it appears there’s no way to filter just that service from the full list of 3000 or more CIDRs.

With this realization I shot out the following Tweet:

And this is the response from AWS Support:

What is ip-ranges.json?

The AWS ip-ranges.json file is a huge list of CIDR ranges that are used by AWS services. As far as I know, it lists every AWS IP address so that users can do things like adding these IPs to their firewall allow rule sets. This is just one use case, but the list is provided for users to validate that an IP address is owned by AWS.

Thankfully AWS provides details for the CIDR ranges such as AWS Region (i.e. us-east-1), and the AWS service that these ranges are used for. But the services covered in this list is only a very small subset of all AWS services. This is limiting because users cannot filter every service that may send traffic from the AWS network to the user’s network. For example AWS Simple Notification Service (SNS) sends requests from AWS’s network to the user’s network. Yet this service is not listed in the ip-ranges.json list, so you would have to allow every AWS IP address, not a subset containing only the SNS CIDRs which would be much more reasonable.

The AWS documentation for this ip-ranges.json syntax shows that the user can only perform filtering on the services below, because they are the only services that are broken out separately.

This is a major problem.

Why so few services?

The “Valid Values” list contains 17 service specific items, and the overall AMAZON value is a superset that contains every AWS CIDR (including those in the service specific subsets).

AWS now has over 300 services, obviously not every service has outgoing traffic to a user’s network, but that’s beside the point. If they separate out some services that have outgoing traffic, they should do the same for every service with outgoing traffic.

Also, some of their most important services such as SNS (which by the way, is one of AWS’s first services created, in Apr 2010) are not on this list. This should be corrected immediately so that users can add these SNS CIDRs to their firewall allow lists (I’m not asking for a friend, I’m the customer in this case who’s really interested in doing this).

Mind the gap

While its great that AWS Support responds to these Tweets now (there was a time when they did not, and you didn’t know if your lone voice in the wilderness was reaching anyone,) it seems that these deficiencies are everywhere, meaning in every AWS service. I know there are UX problems, AWS Support knows there are UX problems, Werner Vogels knows there are UX problems, and Jeff Bezos better know there are UX problems. It’s a tall order to get everything right. But AWS keeps saying they have relentless focus on the customer. Fix these rough edges before more services are released.

Why I’m annoyed here is that they started breaking out service subsets in ip-ranges.json, but they didn’t take it to its natural conclusion, which would have been including every (outgoing) service as a valid filter value. It isn’t relentless focus on the customer if the user has to build a list of all 3000 some odd CIDRs from the list in order to allow just the SNS traffic, when it should only require hundreds or even dozens in such an allow list.

AWS Amplify Custom Domain Management

Sun, 24 May 2020 07:34:36 -0600

Recently I moved my personal blog from GitHub Pages to AWS Amplify. While it was fairly easy to setup the Amplify project and CI/CD pieces, the DNS configuration for Custom Domains was less straightforward. The following observations are what I found to be less than ideal while using Amplify for the first time.

AWS Amplify DNS instructions

One problem that I found with AWS Amplify, which was surprising for someone who is familiar with Route53 and DNS, is that the suggested instructions on the Amplify Domain Management page are somewhat cryptic. Amplify is a fairly new AWS service, so I’ll give them some lattitude, but this UI/UX needs to be improved.

These were the problems that I think are the most blatant, and how they can be improved.

The ownership verification CNAME should clearly indicate what the host name is and what the record is. In the above image, it just shows the host name on the left side of the CNAME record type, and the record value on the right side. It was slightly confusing to me, and for someone who is unfamiliar with DNS and AWS Route53 it is likely even more difficult to follow.
The ANAME information is confusing if you’re using AWS Route53, though it does say in very small grey type that this also means ALIAS record. Isn’t this commonly known simply as an A record? I’m no DNS expert but I expect AWS to use common terminology when possible. They do discuss terminology in the Amplify documentation but it should be clearly described in the actual instructions where you’re telling people how to set this up. Documentation should be secondary, as many people don’t read AWS documentation and it is not often easy to locate AWS docs.
The DNS record information (shown in the above image) is hidden in the Actions drop down here.

This should be much more visible. Possibly a big red button saying “Setup Your DNS HERE”.
The CloudFront distribution domain name in the above image is helpful, but it should also include the CloudFront distribution zone ID. I had to pretend to add this CNAME record in the Route53 console before I could figure out what zone this CloudFront distribution was part of. This is necessary if you’re using something like Terraform to proivsion these CNAME or ANAME records.
When a user has a Route53 zone in another AWS organizational account, it should be possible to use that zone, rather than creating ACM certs and zone records in the AWS organization account which houses the Amplify website. This might be possible with IAM cross account roles or something else but I haven’t found documentation to indicate that as of yet.

So there are quite a few concerning UI/UX issues that need to be addressed to make Amplify more welcoming to the average bear.

Integration with other AWS organization accounts

My blog using Amplify is in a separate AWS account than my AWS Route53 and AWS Certificate Manager resources, which might be a common pattern for many organizations. In effect, because the Amplify application is not in the same account as the Route53 and ACM resources for the domain, this becomes the equivalent of configuring a custom domain managed by a third-party DNS provider.

I understand that AWS cannot automatically grant its own services access to another AWS account in the same AWS organization to enable these two services to work together. But there should be the capability to use IAM permissions or roles to grant another account access to the DNS services that Amplify seems to manage under the covers.

Amplify provisions other AWS resources which are not visible

The above observation segues into the next concern, why are Ampify created resources such as Route53 zones and records, CloudFront distributions and ACM certificates hidden when using this service? When Amplify creates your application using a custom domain, it creates ACM certificates, DNS records, and a CloudFront distribution that are all (as far as I can tell) invisible in the console.

I’m familiar with the pattern that when I use the AWS console to create an EC2 instance for example, it creates security groups and possibly VPC resources for me. Anyone who has created an EC2 instance using Infrastructure as Code (IaC) tooling knows there are multiple things that AWS does under the covers in the console for this operation. The difference here though, is that those resources are visible and accounted for in the console.

Oddly, if I create an Amplify app using the console, additional resources that get created on my behalf are not visible in the console. Now I can possibly understand that these are hidden because they might be non-billable resources, and the costs are rolled up into one single Amplify cost line item (although that is not reflected or described in the pricing page.) But these resources should still be shown to me so that I can view them for example to see the Route53 zone ID if I was using it for my own custom domain CNAME record.

Again, this may be because the Amplify team doesn’t want you to attempt to manipulate or configure these resources, but it seems to me an antipattern to generate resources that are not visible, while some of the information from those resources is necessary to use the service properly. I’d much rather that these were visible, but there was a disclaimer that they were associated with an Amplify app and not billed as normal Route53, ACM or CloudFront resources that the user initiated and created themselves.

Conclusion

Overall I’m happy with the complexity that the service is able to abstract for its users. But there are rough edges and I’d ask for AWS to look into working on addressing these concerns. It does appear that in order to make this a very approachable service for some, it has traded off some configurability. That may mean the Amplify service is really only operable in a subset of use cases.

Discourse API Testing

Sun, 26 Apr 2020 23:07:16 -0600

Discourse is used for a number of purposes, but their forum software is quite popular. I’ve been looking at how to us the Discourse API to automate creating posts. This blog post describes how to setup a local Discourse instance and perform API calls against it.

Setup Discourse Container

This step is fairly straightforward thanks to Bitnami’s Discourse Docker container which launches and configures all the components in a Docker Compose environment.

At first I just tried to launch it with docker compose up -d and while it appeared to start and run successfully I couldn’t see anything in my browser. I then read that you have to set the DISCOURSE_HOSTNAME to the IP address you want to run Discourse on. If you’re running on localhost, set it to 127.0.0.1. This issue cleared it up for me.

So after editing the docker-compose.yml file, it now looks like this, and is ready to start Discourse:

git diff docker-compose.yml
diff --git a/docker-compose.yml b/docker-compose.yml
index 1acf76d..51bc761 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -30,7 +30,7 @@ services:
       - DISCOURSE_POSTGRESQL_NAME=bitnami_application
       - DISCOURSE_POSTGRESQL_USERNAME=bn_discourse
       - DISCOURSE_POSTGRESQL_PASSWORD=bitnami1
-      - DISCOURSE_HOSTNAME=www.example.com
+      - DISCOURSE_HOSTNAME=127.0.0.1
   sidekiq:
     image: 'bitnami/discourse:2'
     depends_on:

Now run docker-compose, with the command:

`1`	`docker-compose up -d`

Wait a minute for the application to come up and continue on to the next section. If you want to query the containers to verify they are all up, you’ll see 4 of them, similar to:

docker ps -a
CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS                     PORTS                  NAMES
b4e5a2298efe        bitnami/discourse:2                                   "/app-entrypoint.sh …"   25 hours ago        Up 12 hours                3000/tcp               bitnami-docker-discourse_sidekiq_1
cbe2a7800486        bitnami/discourse:2                                   "/app-entrypoint.sh …"   25 hours ago        Up 12 hours                0.0.0.0:80->3000/tcp   bitnami-docker-discourse_discourse_1
c84794c82b3b        bitnami/postgresql:11                                 "/opt/bitnami/script…"   29 hours ago        Up 12 hours                5432/tcp               bitnami-docker-discourse_postgresql_1
2c00b3c124a2        bitnami/redis:5.0                                     "/opt/bitnami/script…"   29 hours ago        Up 12 hours                6379/tcp               bitnami-docker-discourse_redis_1

Create a Discourse User

For this test, I don’t need to setup SMTP email because I’m really only trying out the Discourse API locally. Unfortunately Discourse relies on SMTP to send emails for every user registration, even the first user signup process. For our purposes, we don’t need email capability, we only need a user to login and create an API token.

This can be accomplished using the following steps.

Connect to the discourse container:

`1`	`[user@host bitnami-docker-discourse]$ docker exec -it bitnami-docker-discourse_discourse_1 /bin/shell`

Using the steps for “Need to log in without receiving a registration email?” in the email troubleshooting documentation as a guide, we’ll run rake to create an admin account:

root@cbe2a7800486:/opt/bitnami/discourse# pwd
/opt/bitnami/discourse
root@cbe2a7800486:/opt/bitnami/discourse# RAILS_ENV=production bundle exec rake admin:create
Email:  myuser@example.com
Password:  
Repeat password:  

Ensuring account is active!

Account created successfully with username myuser
Do you want to grant Admin privileges to this account? (Y/n)  y

Your account now has Admin privileges!

Create API Key

Next we’ll create the API key that we’re going to use to create posts. Follow these steps:

Go to the Admin page (hamburger menu in upper right)
Click on the API tab at the top
Select “New API Key”
Plug in your username, the description, and the user level of “Single User”

Write down the key because you’ll never see it in full again if you forget it. (But of course you can create a new one and delete this one very easily.)

Create New Category

You can create a new category via the GUI, but to do this via the API use a command similar to:

curl -X POST "http://127.0.0.1/categories.json" -H "Content-Type: application/json;" -H "Api-Key: 1234512345" -H "Api-Username: myuser" --data '{"name": "New Category", "color": "b3e0ff", "text_color": "000000"}'

{"category":{"id":6,"name":"New Category","color":"b3e0ff","text_color":"000000","slug":"new-category"
...

The new category ID (as shown in the response,) is 6.

Create a Post in the New Category

To finally perform the post creation using the API, do something like:

curl -X POST "http://127.0.0.1/posts.json" -H "Content-Type: application/json;" -H "Api-Key: 1234512345" -H "Api-Username: myuser" --data '{"title": "This is a new post in the newest category!", "category": 6, "raw": "This is the body for the new post"}'

{"id":24,"name":null,"username":"myuser","avatar_template":"/letter_avatar_proxy/v4/letter/m/9de0a6/{size}.png","created_at":"2020-04-27T03:12:10.295Z","cooked":"\u003cp\u003eThis is the body for the new post
...

And now you’ve created a new post using the API. Mission accomplished! (For real, not like with W.)

Using AWS Lambda@Edge on Cloudfront

Tue, 28 Jan 2020 23:07:16 -0600

I’ve been attempting to learn more about Lambda@Edge and how to use functions at the AWS edge locations, so I wrote up this demo. It isn’t performing a very realistic operation, but it did allow me to understand the Lambda@Edge and Cloudfront cache relationship (aka “event model”) and how to intercept CDN cache requests and modify responses in a Lambda function.

Demo Repository

The repository is at https://github.com/mbacchi/lambda-edge-cloudfront-terraform.

All the instructions for deploying the Terraform and uploading the html/js files to the S3 bucket are in the README.md. Follow those instructions to deploy using Terraform.

NOTE: Deploying can take 5 minutes in order to replicate the CloudFront distribution to all AWS edge locations.

Overview

The below steps are possible only after deploying all the AWS resources via Terraform and placing the html/js files in the S3 bucket.

The basic flow of the actions in this demo are:

The CloudFront distribution sits in front of an S3 bucket which hosts the index.html page
When the Submit butten is pressed, the index.html page passes the querystring to the CloudFront distribution
The viewer_request.py Lambda function associated with the CloudFront distribution intercepts the request before the cache performs the lookup. In the function we perform a 302 redirect to the 2nd html page other.html, with the same query string as passed by the index.html page.
As the response is sent to the viewer, the origin_response.py Lambda function adds the content-security-policy that allows the other.py page to execute scripts in the CloudFront distribution domain, and allows github.com to be used to connect via an API call (also known as the connect source), all on the fly by adding HTTP headers to the response.
When loaded on the viewer’s web browser, the other.html page makes the API call in script.js with the query string, and the async await function loads the response into the DOM when the API query returns.

Lambda@Edge Basics

The below diagram (from the AWS documentation) explains how the request from and response to the viewer can be modified before or after the cache.

In our example we will use the viewer request and origin response points of this diagram to execute our Lambda functions and modify the request/response.

Request and Response

The viewer_request.py Lambda function first looks at the request and identifies if the referrer URI includes the string index.html. If so it redirects to the CloudFront distribution domain name, appending the query string passed to the page.

As that redirect is handled it goes to the cache (and possibly to the origin) to get the 2nd html page, and in the origin_response.py the Content Security Policy (i.e. CSP) is dynamically configured to allow the CloudFront distribution domain name to execute scripts(script-src).

Finally when the other.html page loads, it calls the Javascript async function to query the GitHub API with the querystring username provided to index.html, and fills in the DOM with the response json.

Logging and troubleshooting

The output of the Lambda@Edge function (we have multiple print statements) is directed to AWS Cloudwatch just like any Lambda function. This allows us to take a look at the output and either debug or use for general information. This is going to be in the region that is closest to your user making the request, so in my case I am in Colorado, and the edge location apparently is us-west-2 (Oregon).

You can also use the Chrome (or Brave) developer console to show errors when making requests. Hit Ctrl-Shift-I or use the browser menu to open the Developer Tools and watch while you load the page.

This is all described in the README.md.

Additional Resources

More help about CloudFront, Lambda@Edge and AWS can be found at these links:

Conclusion

This was an interesting project to work on. CloudFront and Lambda@Edge appear quite useful now that I’ve learned the basics. Hope you got something out of this demo too.

Enjoy!

Addendum

In the original version of this post, I indicated that these deployments could take anywhere from 20-30 minutes. I’ve corrected this information because that has now changed.

The AWS CloudFront team has recently shared updated information about their deployment times. They have improved these deployments from 20-30 minutes to typically around 5 minutes.

This is a major improvement and makes it much easier to use.

Running Brave in a Docker Container

Wed, 11 Sep 2019 08:24:14 -0600

Security minded users often run Docker to create an additional sandbox around an untrusted application running on their system. Web Browsers are among the most untrusted applications we run today. Here’s how to setup the Brave Browser to run in Docker.

To paraphrase T.S. Eliot, ‘good coders borrow and great coders steal.’ So I’ll be borrowing from Jess Frazelle’s work on running Chromium in Docker and using her seccomp profile. This specifically provides the list of system calls that Chromium or in our case Brave can run in the Docker container.

Install and Configure Docker

You will first need to install Docker on your machine and make sure the daemon is running. To verify your system is configured to run Docker, try running the hello-world image:

[user@host ~]$ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
1b930d010525: Pull complete 
...

If it doesn’t run there are many guides to configuring Docker available.

Brave in Docker

In order to run Brave in Docker clone the brave-github GitHub repository.

The Dockerfile in this repository uses a basic Fedora image to install Brave from the release channel RPM repository.

Follow these steps to setup additional prereqs:

Change into the brave-docker directory after cloning: cd brave-docker
Next, you’ll need to download the seccomp profile from here: wget https://raw.githubusercontent.com/jfrazelle/dotfiles/master/etc/docker/seccomp/chrome.json -O chrome.json
Add xhost permissions for your user using the command (It’s probably good to come up with a more restrictive method than opening up xhost to all clients.): xhost +

Building the Docker image

To build the image, run:

`1`	`[user@host brave-docker]$ docker build . -t brave-release-fedora`

You can use any tag you like, just replace the -t argument above.

To list your image after building, run the command:

1
2
3

[user@host brave-docker]$ docker images
REPOSITORY             TAG                 IMAGE ID            CREATED              SIZE
brave-release-fedora   latest              729a66f5a7cf        About a minute ago   1.08GB

Running the container

To run Brave in the image you built, use the docker run command:

[user@host brave-docker]$ docker run -it --net host --cpuset-cpus 0 --memory 512mb \
-v $HOME/Downloads:/home/brave/Downloads:z -v /tmp/.X11-unix:/tmp/.X11-unix \
--security-opt seccomp=./chrome.json -e DISPLAY=unix$DISPLAY --device /dev/dri \
-v /dev/shm:/dev/shm --device /dev/snd brave-release-fedora

It will default to using UID/GID 1000 for the brave user in the container. This will allow you to mount your ~/Download directory as a volume in the container, in order to download files as you would using a browser normally in your environment. If you need to change the UID/GID, pass the flag --build-arg UID_GID=YOUR_UID to the docker build command in the previous section.

Have fun!

Setting up an EC2 Instance as an Inlets Exit Node

Wed, 21 Aug 2019 08:41:54 -0600

Inlets is a fairly new project that allows you to setup reverse proxy, websocket tunnels, or other endpoints to the public internet, it is similar to ngrok. The video overview from Alex Ellis shows how simple it makes setting up the environment.

Also, there are scripts in the Inlets repository to provision DigitalOcean droplets. Let’s setup an AWS EC2 instance, though AWS is more expensive than DO, it’s obviously a popular platform. The steps below are required to setup an AWS VPC and related networking before launching our instance.

In this blog post we’ll set up an exit node using the AWS CLI, then make it simpler with Terraform.

Using the AWS CLI

Here is an example of the commands that would need to be run from the AWS CLI if you wanted to use that method for starting up the EC2 instance. (Note: Do not copy all these commands directly, many of them reference objects in my AWS account such as VPC IDs, SG IDs etc. and which were removed after this blog post was written.)

aws ec2 create-vpc --cidr-block 10.0.0.0/16 --region us-east-2

aws ec2 create-security-group --group-name "temp-inlets-sg" --vpc-id vpc-0492e18d8bda513ab --description \
"Inlets security group" --region us-east-2

aws ec2 authorize-security-group-ingress --group-id sg-08754da824fc991bc --protocol tcp --port 22 --cidr \
161.97.199.201/32 --region us-east-2

aws ec2 authorize-security-group-ingress --group-id sg-08754da824fc991bc --protocol tcp --port 8090 --cidr \
161.97.199.201/32 --region us-east-2

aws ec2 create-key-pair --key-name inlets-test-keypair --region us-east-2 | jq -r ".KeyMaterial" \
> inlets-keypair.pem

aws ec2 create-subnet --vpc-id vpc-0492e18d8bda513ab --availability-zone us-east-2a --cidr-block 10.0.5.0/20 \
--region us-east-2

aws ec2 create-internet-gateway --region us-east-2

aws ec2 attach-internet-gateway --vpc-id vpc-0492e18d8bda513ab --internet-gateway-id igw-07411b46a0ce5ae46 \
--region us-east-2

aws ec2 create-route-table --vpc-id vpc-0492e18d8bda513ab --region us-east-2

aws ec2 create-route --route-table-id rtb-0423d1a8bf3f1a226 --destination-cidr-block 0.0.0.0/0 --gateway-id \
igw-07411b46a0ce5ae46 --region us-east-2

aws ec2 associate-route-table  --subnet-id subnet-00de3c90045c569db --route-table-id rtb-0423d1a8bf3f1a226 \
--region us-east-2

aws ec2 run-instances --image-id ami-0eaecb2e7d28d5f33 --count 1 --instance-type t3a.micro --security-group-ids \
sg-08754da824fc991bc --key-name inlets-test-keypair --region us-east-2 --tag-specifications \
'ResourceType=instance,Tags=[{Key="Purpose",Value="TESTING"}]' --associate-public-ip-address

That’s a lot of typing to setup an AWS instance before deploying the inlets project. Lets see if we can do this with less effort using Terraform.

Terraform

The terraform equivalent to setup an EC2 instance with all the plumbing is quite a bit more concise. You can view the Terraform I wrote in this repository.

Note: We don’t create an AWS keypair in Terraform, I consider that bad form. In order to avoid any security concerns we expect that you use a keypair previously generated or uploaded to your AWS account. You can use the aws ec2 create-key-pair command above as an example if you want to use the CLI.

Prerequisites

Terraform
An AWS account
An AWS Keypair

Prepare to run terraform

These steps include:

Install terraform
Clone the inlets-aws-terraform repository (i.e. git clone https://github.com/mbacchi/inlets-aws-terraform)
cd inlets-aws-terraform
Run terraform init
Export environment variables for your AWS_PROFILE and AWS_REGION. This looks something like: export AWS_PROFILE=PROFILE_NAME && export AWS_REGION=us-east-2
Don’t forget to change the PROFILE_NAME in the export command above!
Change the key_name on line 64 of main.tf to the name of your own keypair!

Create the terraform plan

Create the token and run the terraform plan command with the single command line:

`1`	`token=$(head -c 16 /dev/urandom \| sha256sum \| cut -d" " -f1); terraform plan -out=terraform_plan.$(date +%F.%H.%M.%S).out -var "token=$token"`

This will output a file to be used in the next step, named something like: terraform_plan.2019-08-20.23.04.54.out

Note: You might not have sha256sum on your system, it can be replaced with shasum.

Apply the terraform plan

Apply the plan that you created in the previous step after reviewing the plan output.

terraform apply terraform_plan.2019-08-20.23.04.54.out
...
Apply complete! Resources: 25 added, 0 changed, 0 destroyed.

Outputs:

inlets_token = 2395590c8b333cb2cb9f54ac152aac03e0365c1c69ce20348a3124b84c98ec62
public_ip_address = 3.19.218.92

The terraform output that you see in the last few lines provides the Inlets token required by the client to authenticate with the server, and the public IP address of the EC2 instance. Use this info in the below steps.

Connecting to Inlets

Now that you have deployed the AWS infrastructure using Terraform, we will run an application locally and serve it on the public internet from the Inlets exit node.

This is documented in the Inlets README, but basically on your client machine you need an application such as a webserver running. For example my jekyll blog:

cd blog
make serve 
JEKYLL_ENV=development jekyll serve
Configuration file: _config.yml
            Source: .
       Destination: _site
 Incremental build: disabled. Enable with --incremental
      Generating... 
                    done in 1.149 seconds.
 Auto-regeneration: enabled for 'myblog'
    Server address: http://127.0.0.1:4000
  Server running... press ctrl-c to stop.

In another terminal session I can start the Inlets client:

export TOKEN="2395590c8b333cb2cb9f54ac152aac03e0365c1c69ce20348a3124b84c98ec62"

export REMOTE="3.19.218.92:8090"

inlets client  --remote=$REMOTE --upstream=http://127.0.0.1:4000 --token $TOKEN

In a web browser, I can open the url 3.19.218.92:8090 and see my application that was running locally only.

Remove the Terraformed infrastructure

When you want to remove this infrastructure, run terraform destroy from the cloned inlets-aws-terraform directory where you ran terraform apply etc.

Why Did Official Python Docker Images Disappear for an Afternoon?

Mon, 29 Jul 2019 09:19:13 -0600

We have all become accustomed to services on the internet being reliable and available approximately 100% of the time. Many services obviously have outages, recently GitHub, Cloudflare, Twitter, Facebook have all had widespread service disruptions. Some apps affect downstream processes and services that rely on them. This is a story about how I found the official Python images on Docker Hub were missing for the linux/amd64 architecture for an entire afternoon(potentially longer).

The Stage is Set

Unfortunately, the official Python Docker Hub images don’t explicitly indicate which architecture they support. You would expect images for popular distributions such as Debian Buster would be available for both amd64 and ARM architectures.

On Saturday, July 13 I was working on some Dockerfiles that I use to test my Python library, and I couldn’t download the Buster image I had used in the past. The error I encountered was:

`1`	`no matching manifest for linux/amd64 in the manifest list entries`

This support page indicated I could list all images for my architecture using a command similar to:

DOCKER_CLI_EXPERIMENTAL=enabled docker manifest inspect -v python:3.8-rc-buster | jq '.[].Descriptor.platform'
{
  "architecture": "amd64",
  "os": "linux"
}
{
  "architecture": "arm",
  "os": "linux",
  "variant": "v6"
}
...

The only architecure returned for this image at the time was ARM. I worked around this problem by using the Alpine Linux image instead because I needed to get this testing completed. But that limitation struck me as strange. It kept gnawing at me, I couldn’t shake the feeling I must have been using incorrect command. A couple weeks later I decided to write a blog post about my experience. Funny enough as I was writing the very blog post you’re reading now, I found that the Buster images were in fact available once again.

Act Two

Today the results returned from Docker Hub include amd64, ARM and others:

DOCKER_CLI_EXPERIMENTAL=enabled docker manifest inspect -v python:3.8-rc-buster | jq '.[].Descriptor.platform'
{
  "architecture": "amd64",
  "os": "linux"
}
{
  "architecture": "arm",
  "os": "linux",
  "variant": "v5"
}
{
  "architecture": "arm",
  "os": "linux",
  "variant": "v7"
}
{
  "architecture": "arm64",
  "os": "linux",
  "variant": "v8"
}
{
  "architecture": "386",
  "os": "linux"
}
{
  "architecture": "ppc64le",
  "os": "linux"
}
{
  "architecture": "s390x",
  "os": "linux"
}

This is expected. But not what I found 2 weeks ago.

There was no outage reported on the Docker Hub status page for that day. It stands to reason then that the Python official Docker images were somehow not built properly for amd64 and other platforms during this window of time.

I would love to know how this occurred. I guess this reinforces how the folks who say everything is broken really have a point. At least I wasn’t using this in an automated process as part of a CI pipeline or something. Although I also see that type of transient service outage in my day job an just hit the Restart Job button in Jenkins.

Returning HTTP Errors from API Gateway in AWS Lambda Functions using Python and Serverless Framework

Tue, 21 May 2019 09:22:07 -0600

There are many examples available for how to return an error from an AWS Lambda function through API Gateway to a client in Node.js, but relatively few for how to do so using the Python runtime. Here we will try to give some basic info using Python with a POST action.

Prereqs

This assumes you already have an AWS account, have configured your AWS access credentials or profile and set the AWS_REGION environment variable to your region of choice. Otherwise it will use the us-west-2 region by default.

You will also need the Serverless Framework to be installed.

Lambda-Proxy Integration

In this blog post we are going to cover the Lambda-Proxy integration method versus the Lambda method. This is the simplest and from what I’ve found least documented because it just seems obvious to AWS technical writers, Serverless framework developers and blog authors. Most examples are written in Node.js because of that because that’s what both AWS and Serverless framework documents discuss. Our example hopes to make it a little more clear.

Create Serverless project

Let’s start by creating a new Serverless framework project. We will be using the aws-python3 template for this, which populates a basic serverless.yml configuration file and a basic handler function. To do so we use the serverless create command (abbreviated as sls create for brevity) to generate the skeleton to our project.

[user@linux serverless]$ sls create --template aws-python3 --path test-lambda-exception-handling
Serverless: Generating boilerplate...
Serverless: Generating boilerplate in "/serverless/test-lambda-exception-handling"
 _______                             __
|   _   .-----.----.--.--.-----.----|  .-----.-----.-----.
|   |___|  -__|   _|  |  |  -__|   _|  |  -__|__ --|__ --|
|____   |_____|__|  \___/|_____|__| |__|_____|_____|_____|
|   |   |             The Serverless Application Framework
|       |                           serverless.com, v1.38.0
 -------'

Serverless: Successfully generated boilerplate for template: "aws-python3"

Once we have this template in place, you can modify the files yourself, or copy the below into your project. It is also available in my GitHub repository if you’d rather look at that.

The handler function and YAML are:

import json

def throw_error(event, context):
    data = json.loads(event['body'])
    print(data) # this can be seen in CloudWatch Logs

    response = {
        "statusCode": 400,
        "headers": {
            "Content-Type": 'application/json',
            "context.aws_request_id": context.aws_request_id # for fun show the Lambda requestID
        },
        "body": "Bad Request",
        "isBase64Encoded": False
    }

    return response

service: test-lambda-exception-handling

provider:
  name: aws
  runtime: python3.7
  region: ${AWS_REGION, 'us-east-2'}

  stage: dev

functions:
  throw_error:
    handler: handler.throw_error
    events:
      - http:
          path: throw
          method: post
          cors: true

What this does

The serverless.yml file defines the FaaS or Cloud provider where you are going to run the function, as well as the function itself. This is a very basic single function for the purposes of this demo. The documentation is here. As you can see we are running in AWS Lambda, and using the function named throw_error.

The handler can be named anything, but must be referenced in the .yml file. We’re using the name that the Serverless template provided for us. In a Lambda runtime environment, you’re passed an event object and a context object. You are provided event details including any data passed to the function via the POST request. The context object has information about the execution environment, invocation and other things, described here.

Our handler simply loads the event data, and then prints it so that it can be viewed in Cloudwatch Logs. This isn’t returned to the calling client. Because we’re using API Gateway, the response must conform to a specific format.

In our function we simply return a 400 Bad Request response immediately after decoding the data passed in.

Deploying the project

If we’re cautios we can look at what will be deployed by first running the sls package command. Then we can look in the .zip file that it will upload to AWS Lambda.

[user@linux test-lambda-exception-handling]$ sls package
Serverless: Packaging service...
Serverless: Excluding development dependencies...
[user@linux test-lambda-exception-handling]$ ls -a
.  ..  .gitignore  handler.py  .serverless  serverless.yml
[user@linux test-lambda-exception-handling]$ ls .serverless/
cloudformation-template-create-stack.json  cloudformation-template-update-stack.json  serverless-state.json  test-lambda-exception-handling.zip
[user@linux test-lambda-exception-handling]$ zipinfo -l .serverless/test-lambda-exception-handling.zip 
Archive:  .serverless/test-lambda-exception-handling.zip
Zip file size: 407 bytes, number of entries: 1
-rw-rw-r--  4.5 unx      451 bl      273 defN 80-Jan-01 00:00 handler.py
1 file, 451 bytes uncompressed, 273 bytes compressed:  39.5%

To deploy we use the sls deploy command.

[user@linux test-lambda-exception-handling]$ sls deploy
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Creating Stack...
Serverless: Checking Stack create progress...
.....
Serverless: Stack create finished...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service test-lambda-exception-handling.zip file to S3 (407 B)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
.................................
Serverless: Stack update finished...
Service Information
service: test-lambda-exception-handling
stage: dev
region: us-east-2
stack: test-lambda-exception-handling-dev
resources: 11
api keys:
  None
endpoints:
  POST - https://7vt4w7lb4l.execute-api.us-east-2.amazonaws.com/dev/throw
functions:
  throw_error: test-lambda-exception-handling-dev-throw_error
layers:
  None

This returns the URL for POST actions, which we will use in our next step.

Invoking the Function

Now we can simply run a POST against our new function URL and see how it will respond.

[user@linux test-lambda-exception-handling]$ curl -i -X POST https://7vt4w7lb4l.execute-api.us-east-2.amazonaws.com/dev/throw --data '{"blah": "blah1"}'
HTTP/2 400 
content-type: application/json
content-length: 11
date: Wed, 22 May 2019 01:34:50 GMT
x-amzn-requestid: ca8164e7-7c31-11e9-b114-d91ebd2032b3
context.aws_request_id: 87f10bcd-7c85-46d5-bcd7-493aca45e1f0
x-amz-apigw-id: aD8RIG3MCYcFk2w=
x-amzn-trace-id: Root=1-5ce4a73a-af05ec85fe34062d036b396f;Sampled=0
x-cache: Error from cloudfront
via: 1.1 c4fb40b7909e4dd897bba2e297b284e7.cloudfront.net (CloudFront)
x-amz-cf-id: Sovagss83OsF-hv8b7QHcf9GkmV2r1A_XEF8KzI18GMrhWzGvfaGSQ==

Bad Request[user@linux test-lambda-exception-handling]$

This shows that we receive the Bad Request response, with HTTP status code 400 as expected.

In a future post I’ll describe how to return errors in Python with the Lambda integration method.

Using Git hook templates to avoid committing secrets to public repositories

Fri, 11 May 2018 12:39:43 -0600

Git doesn’t have the concept of a per user global hook. It would be nice if you could create hooks in your home directory that could be executed in all repositories that you work with. Instead, it does allow you to write hooks that reside in a user specific template directory to then be copied into any repositories that you clone or create from scratch.

After following this guide to configure template hooks, when you clone or create a new repository using the git init command, your hooks will be copied into your new repositories.

Why would I want hooks in every new repository?

There could be many reasons why you want to setup a hook in every new repository. My current use case has to do with preventing AWS credentials from accidentally being committed to any of my GitHub repositories. This prevents you from having someone get into your AWS account and potentially racking up a large bill you would be responsible for. Automatically scanning your commit for any patterns that look like AWS secrets will prevent this costly mistake.

Template configuration

The git init man page talks about the Template Directory which gives us the ability to copy hooks into new repositories. It must first be configured using the git config command, specfying the init.templatedir which is the directory where you will store your hooks. You will run a command such as:

`1`	`git config --global init.templatedir '~/.git_template'`

This config directive is added to the [init] section in your .gitconfig file, which looks like the following:

1
2

[init]
	templatedir = ~/.git_template

Create the hook

Now we can create our hook and place it in the template hooks directory (which is actually templatedir/hooks). This hook employs git-secrets to prevent me from accidentally committing my AWS credentials to GitHub:

#!/bin/bash
#
# A git hook to run git-secrets (locally 'installed') against a repository.
# This should be extended to do much more verification and error handling,
# but is presented as a demonstration.
#
# This assumes you've already setup git-secrets patterns (i.e. "git-secrets --register-aws").

# Verify that the git-secrets file is in our path
secretspath=$(which git-secrets)
if [[ -z "${secretspath}" || ! -x "${secretspath}" ]]; then
  echo "git-secrets not found!"
  exit 1
fi

RESPONSE=`git-secrets --scan -r .`
RC=$?

if [[ $RC -eq 1 ]]; then
    # return code 1 from git-secrets indicates scan found a verboten string
    echo
    echo
    echo "ERROR: [pre-commit hook] Aborting commit because git-secrets found a prohibited pattern in this repository"
    echo "Please remove the file with the pattern in the output above from this repository!"
    echo
    exit 1
else
    # no secrets found in repo
    exit 0
fi

Looking in that directory we see the new hook, but we need to make it executable so I will run the command chmod 744 on the file:

[user1@host hooks]$ chmod 744 ~/.git_template/hooks/pre-commit
[user1@host hooks]$ ls -l ~/.git_template/hooks
total 4
-rwxrw-r--. 1 user1 user1 931 May 11 20:34 pre-commit

Prepare a file containing a pattern that will fail the git-secrets scan

At this point we will create a new directory to test our hook. Start by creating a directory to work in:

`1`	`[user1@host repos]$ mkdir template-dir-demo && cd template-dir-demo`

Then create a file with a string that looks like an AWS secret(hint, the AWS secret access key is a 40 character string containing lower, upper and digit characters):

[user1@host template-dir-demo]$ ipython
Python 3.6.4 (default, Mar 13 2018, 18:18:20)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.2.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import random

In [2]: from string import ascii_uppercase, ascii_lowercase, digits

In [3]: chars = ascii_uppercase + ascii_lowercase + digits

In [4]: key = ''.join(random.choice(chars) for _ in range(40))

In [5]: key
Out[5]: 'dbMi8e8YYNnTUnlTEEyyYbK3KfQExuaRXF8roC9d'

[user1@host template-dir-demo]$ echo 'aws_secret_access_key = dbMi8e8YYNnTUnlTEEyyYbK3KfQExuaRXF8roC9d' > blah1.txt
[user1@host template-dir-demo]$ cat blah1.txt
aws_secret_access_key = dbMi8e8YYNnTUnlTEEyyYbK3KfQExuaRXF8roC9d

Now we will run git init . to initialize a git repository in the current directory, and we see that the pre-commit hook is copied from the template directory:

[user1@host template-dir-demo]$ git init .
Initialized empty Git repository in /home/user1/repos/template-dir-demo/.git/
[user1@host template-dir-demo]$ ls -altr .git/hooks/
total 12
-rwxrwxr-x. 1 user1 user1  931 May 11 20:39 pre-commit

Perform a commit

Finally, lets stage and commit the file and see if the pre-commit hook allows the file to be staged and committed:

[user1@host template-dir-demo]$ git add blah1.txt
[user1@host template-dir-demo]$ git commit -m "blah1 file with a secret"
./blah1.txt:1:aws_secret_access_key = dbMi8e8YYNnTUnlTEEyyYbK3KfQExuaRXF8roC9d

[ERROR] Matched one or more prohibited patterns

Possible mitigations:
- Mark false positives as allowed using: git config --add secrets.allowed ...
- Mark false positives as allowed by adding regular expressions to .gitallowed at repository's root directory
- List your configured patterns: git config --get-all secrets.patterns
- List your configured allowed patterns: git config --get-all secrets.allowed
- List your configured allowed patterns in .gitallowed at repository's root directory
- Use --no-verify if this is a one-time false positive


ERROR: [pre-commit hook] Aborting commit because git-secrets found a prohibited pattern in this repository
Please remove the file with the pattern in the output above from this repository!

As you can see using a Git template for hooks will help us automatically setup hooks to use in any new repository whether it is cloned or created with the git init command.

I hope this was helpful to you.

AWS CodeCommit SSH Key ID

Mon, 22 Jan 2018 14:28:31 -0600

Working on AWS CodeCommit today, I setup SSH access to the repository. During the initial configuration I provided the SSH public key in the AWS Console, but then couldn’t connect to my repository. In the brief instructions on the IAM Console page they tell you how to update your ~/.ssh/config file, but the example doesn’t explicitly say what the IdentityFile is supposed to be set to. In the more detailed instructions they do specify the IdentityFile should reference your private key, not your public key.

For others who are confused about this, and my future self, here’s the AWS documentation for how to configure SSH Key ID for CodeCommit.

3rd Party Github Credential Scanning

Sat, 20 Jan 2018 14:34:36 -0600

While writing a Python library that performs scanning of Git repositories similar to AWS Labs' git-secrets, I was surprised by some 3rd party scanning services randomly scanning my repository for AWS credentials. I had included deactivated AWS credentials in my repository so that I could test my library. My plan was to replace these credentials with a randomly generated string later on but at first I was satisfied to commit the actual (but not active) credentials to Github themselves.

A couple weeks after I created the repository on Github, I got an email from a random address named ‘githubbot’. In the email they indicated that they had found a credential in my repository. As a reward for notifying me of this potentially costly situation, the email indicated I could send them bitcoin, and they included the wallet address. Below is the email itself:

Later that day, in order to remove the credential file used in testing, I pushed a commit that moved the credential strings in the test driver itself. Within an hour I got an email from another unrelated service GitGuardian, indicating I exposed an API key in my repository. Who knew so many good samaritan services existed to help people out. Here’s this email:

What I realized from this encounter is that there are all kinds of unknown actors in the Github universe. Some will be benevolent like these, but there very well could be black hat types who are scanning every public repository for ways to get ahold of your AWS or other keys and will take advantage of them.

Watch what you do with your keys!!!! If you want some pointers on how to prevent releasing credentials accidentally to Github, take a look at my recent [blog post on preventing secret leaks]({{ site.baseurl }}{% post_url 2017-12-22-3-ways-prevent-secret-leaks-github %}).

Writing to the AWS Lambda SAM Local container /tmp filesystem

Sun, 07 Jan 2018 14:55:57 -0600

While using AWS Lambda SAM Local to test Lambda functions locally, I encountered an error writing to the current directory where the function was running in the container (/var/task/). I’m not claiming a best practice of writing to the filesystem while running a Lambda function, but that’s part of my learning process for the moment, and I will investigate other workflows shortly. But what I was able to get working successfuly was writing to the /tmp filesystem instead of the current working directory.

Here is the example code where it writes to the current directory with an the result that we encounter the error that the directory is read only:

[mbacchi@nuc2 sample-sam-local-write]$ cat samlocalwrite.py
import os
import sys


def handler(event, context):
    print("Starting samlocalwrite lambda handler function")
    print("Directory provided from event was: {}".format(event['directory']))

    remote_repo = event['directory']

    #local_repo = os.path.join(os.path.sep, "tmp", os.path.basename(event['directory']))
    local_repo = os.path.basename(event['directory'])

    if not os.path.exists(local_repo):
        os.makedirs(local_repo)

    with open(local_repo + "blah.txt", 'w') as f:
        f.write('blah\n')

    return "Hello"

And the output of the sam local invoke command:

[mbacchi@nuc2 sample-sam-local-write]$ sam local invoke "SamLocalWriteFunction" -e event.json
2018/01/07 15:50:24 Successfully parsed template.yaml
2018/01/07 15:50:24 Connected to Docker 1.35
2018/01/07 15:50:24 Fetching lambci/lambda:python3.6 image for python3.6 runtime...
python3.6: Pulling from lambci/lambda
f338a32fa56c: Pull complete
4926b20b634f: Pull complete
298eae5902d7: Pull complete
e58d162628c7: Pull complete
e02bc73f2e71: Pull complete
Digest: sha256:0682e157b34e18cf182b2aaffb501971c7a0c08c785f337629122b7de34e3945
Status: Downloaded newer image for lambci/lambda:python3.6
2018/01/07 15:51:25 Invoking samlocalwrite.handler (python3.6)
2018/01/07 15:51:25 Decompressing /home/mbacchi/data/repos/mbacchi/sample-sam-local-write/samlocalwrite.zip
2018/01/07 15:51:46 WARNING: No AWS credentials found. Missing credentials may lead to slow startup times as detailed in https://github.com/awslabs/aws-sam-local/issues/134
2018/01/07 15:51:46 Mounting /tmp/aws-sam-local-1515358285999917851 as /var/task:ro inside runtime container
START RequestId: 9371dabf-9678-4fc4-83d8-5d8c8cf8c9fa Version: $LATEST
Starting samlocalwrite lambda handler function
Directory provided from event was: whatever
[Errno 30] Read-only file system: 'whatever': OSError
Traceback (most recent call last):
  File "/var/task/samlocalwrite.py", line 18, in handler
    os.makedirs(local_repo)
  File "/var/lang/lib/python3.6/os.py", line 220, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: 'whatever'

END RequestId: 9371dabf-9678-4fc4-83d8-5d8c8cf8c9fa
REPORT RequestId: 9371dabf-9678-4fc4-83d8-5d8c8cf8c9fa Duration: 161 ms Billed Duration: 0 ms Memory Size: 0 MB Max Memory Used: 18 MB

{"errorMessage": "[Errno 30] Read-only file system: 'whatever'", "errorType": "OSError", "stackTrace": [["/var/task/samlocalwrite.py", 18, "handler", "os.makedirs(local_repo)"], ["/var/lang/lib/python3.6/os.py", 220, "makedirs", "mkdir(name, mode)"]]}

If we make a slight modification to write to the /tmp filesystem:

[mbacchi@nuc2 sample-sam-local-write]$ diff samlocalwrite.py samlocalwrite-new.py
14,15c14,15
<     #local_repo = os.path.join(os.path.sep, "tmp", os.path.basename(event['directory']))
<     local_repo = os.path.basename(event['directory'])
---
>     local_repo = os.path.join(os.path.sep, "tmp", os.path.basename(event['directory']))
>     #local_repo = os.path.basename(event['directory'])

We get a successful run:

[mbacchi@nuc2 sample-sam-local-write]$ sam local invoke "SamLocalWriteFunction" -e event.json
2018/01/07 15:55:54 Successfully parsed template.yaml
2018/01/07 15:55:54 Connected to Docker 1.35
2018/01/07 15:55:54 Fetching lambci/lambda:python3.6 image for python3.6 runtime...
python3.6: Pulling from lambci/lambda
Digest: sha256:0682e157b34e18cf182b2aaffb501971c7a0c08c785f337629122b7de34e3945
Status: Image is up to date for lambci/lambda:python3.6
2018/01/07 15:55:54 Invoking samlocalwrite.handler (python3.6)
2018/01/07 15:55:54 Decompressing /home/mbacchi/data/repos/mbacchi/sample-sam-local-write/samlocalwrite.zip
2018/01/07 15:56:15 WARNING: No AWS credentials found. Missing credentials may lead to slow startup times as detailed in https://github.com/awslabs/aws-sam-local/issues/134
2018/01/07 15:56:15 Mounting /tmp/aws-sam-local-1515358554848977835 as /var/task:ro inside runtime container
START RequestId: 0af0a302-49db-4ca2-a0fa-db31acb0182a Version: $LATEST
Starting samlocalwrite lambda handler function
Directory provided from event was: whatever
END RequestId: 0af0a302-49db-4ca2-a0fa-db31acb0182a
REPORT RequestId: 0af0a302-49db-4ca2-a0fa-db31acb0182a Duration: 160 ms Billed Duration: 0 ms Memory Size: 0 MB Max Memory Used: 18 MB

"Hello"

Again, this may not be the perfect solution but it is a way to write to the image filesystem if necessary in your Lambda function.

3 Ways to Prevent .pypirc Credentials or Other Secrets from Leaking onto Github

Fri, 22 Dec 2017 15:02:42 -0600

Even if you’re not involved in the Python community, you might have heard about this security incident a while back. This is a not uncommon scenario where developers who may not be Github or distribution tooling (or security) experts make a mistake and breed mistrust in their project as well as the distribution medium itself.

But setting up your environment to prevent these accidental credential disclosures is easy to do, and will enhance your security posture.

Here are a few ways to prevent these exposures with git and other tools.

Add the .pypirc file to the .gitignore list

This is the most straightforward, and will force git to ignore the .pypirc when you’re adding/committing changes to the repository.

I typically use my global .gitignore file, but you can also add it to the repository specific .gitignore file. Here’s how to do both.

Create a ~/.gitignore and make this your personal git core.excludesfile:

[mbacchi@hostname test-pypi]$ git config --global core.excludesfile=~/.gitignore
[mbacchi@hostname test-pypi]$ git config -l --global | grep core.excludesfile
core.excludesfile=~/.gitignore
[mbacchi@hostname test-pypi]$ echo ".pypirc">> ~/.gitignore
[mbacchi@hostname test-pypi]$ cat ~/.gitignore
.idea
.pypirc

Now you can test whether this will work using the git check-ignore command:

[mbacchi@hostname test-pypi]$ git check-ignore -v .pypirc
/home/mbacchi/.gitignore:4:.pypirc	.pypirc
[mbacchi@hostname test-pypi]$ echo $?
0

If you want to add it to the project repository instead, simply perform the same operation on the .gitignore file in the repository. But be aware that unless you also commit the .gitignore file into the repository this ignore will lost upon removing and recreating this repository.

Add a git pre-commit hook to your git repository that will analyze the staged files for .pypirc

In order to use a git pre-commit hook, first add the file pre-commit to the .git/hooks directory in your repository:

[mbacchi@hostname test-pypi]$ cat .git/hooks/pre-commit
#!/bin/sh
#
# A git hook to check whether .pypirc files exist in the staged files list

PYPIRC=$(git status -s| grep .pypirc)

if [[ "$PYPIRC" == *".pypirc"* ]]; then
    # .pypirc is included in the staged files
    echo "ERROR: [pre-commit hook] Aborting commit because you have the file .pypirc staged to be committed."
    echo "Remove it from the repository using \"git rm --cached .pypirc\""
    echo
    echo "You should also move your .pypirc file to your home directory with the command \"mv .pypirc ~\""
    exit 1
else
    # not found
    exit 0
fi

Make sure it is executable, if it can’t execute it won’t function correctly in the git workflow:

1
2
3

[mbacchi@hostname test-pypi]$ chmod 744 .git/hooks/pre-commit
[mbacchi@hostname test-pypi]$ ls -l .git/hooks/pre-commit
-rwxr--r--. 1 mbacchi mbacchi 533 Dec 22 15:17 .git/hooks/pre-commit

This will do a git status and grep the output for the .pypirc file, if found it will return 1, which tells git commit to abort.

Now, test that it works as expected:

[mbacchi@hostname test-pypi]$ git add .pypirc
[mbacchi@hostname test-pypi]$ git status
On branch master

No commits yet

Changes to be committed:
  (use "git rm --cached <file>..." to unstage)

	new file:   .pypirc
Untracked files:
  (use "git add <file>..." to include in what will be committed)

	LICENSE.txt
	README.rst
	blah/
	kerplunk/
	setup.cfg
	setup.py

[mbacchi@hostname test-pypi]$ git commit
ERROR: [pre-commit hook] Aborting commit because you have the file .pypirc staged to be committed.
Remove it from the repository using "git rm --cached .pypirc"

You should also move your .pypirc file to your home directory with the command "mv .pypirc ~"
[mbacchi@hostname test-pypi]$ echo $?
1

NOTE: This does only work on the repository level, it can’t be added to your personal git configuration to be used for all repos you work in.

Use git-secrets to perform analysis of your current staged files, as well as the repository’s commit history for secret keys

Not specific to .pypirc credentials, we have also heard stories of people accidentally leaking AWS or other private keys (think SSH id_rsa vs. id_rsa.pub files for example) on Github. Not excited about doing this yourself? Me either. So here’s one way I’ve been analyzing both my current git repository commits, as well as my git history.

Using the AWS Labs developed git-secrets we can setup our global git config to prevent strings that look like AWS Access Key ID or AWS Secret Access Key strings to be committed to git repositories and Github.

Clone and install the git-secrets bash script in your personal bin tree:

[mbacchi@hostname repos]$ git clone git@github.com:awslabs/git-secrets.git
Cloning into 'git-secrets'...
remote: Counting objects: 250, done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 250 (delta 2), reused 2 (delta 0), pack-reused 243
Receiving objects: 100% (250/250), 80.42 KiB | 1.36 MiB/s, done.
Resolving deltas: 100% (141/141), done.
[mbacchi@hostname repos]$ cd git-secrets/
[mbacchi@hostname git-secrets]$ cp git-secrets ~/bin
[mbacchi@hostname git-secrets]$ cd ../test-pypi/
[mbacchi@hostname test-pypi]$ git-secrets --help
usage: git secrets --scan [-r|--recursive] [--cached] [--no-index] [--untracked] [<files>...]
   or: git secrets --scan-history
   or: git secrets --install [-f|--force] [<target-directory>]
   or: git secrets --list [--global]
   or: git secrets --add [-a|--allowed] [-l|--literal] [--global] <pattern>
   or: git secrets --add-provider [--global] <command> [arguments...]
   or: git secrets --register-aws [--global]
   or: git secrets --aws-provider [<credentials-file>]

    --scan                Scans <files> for prohibited patterns
    --scan-history        Scans repo for prohibited patterns
    --install             Installs git hooks for Git repository or Git template directory
    --list                Lists secret patterns
    --add                 Adds a prohibited or allowed pattern, ensuring to de-dupe with existing patterns
    --add-provider        Adds a secret provider that when called outputs secret patterns on new lines
    --aws-provider        Secret provider that outputs credentials found in an ini file
    --register-aws        Adds common AWS patterns to the git config and scans for ~/.aws/credentials
    -r, --recursive       --scan scans directories recursively
    --cached              --scan scans searches blobs registered in the index file
    --no-index            --scan searches files in the current directory that is not managed by Git
    --untracked           In addition to searching in the tracked files in the working tree, --scan also in untracked files
    -f, --force           --install overwrites hooks if the hook already exists
    -l, --literal         --add and --add-allowed patterns are escaped so that they are literal
    -a, --allowed         --add adds an allowed pattern instead of a prohibited pattern
    --global              Uses the --global git config

We’re going to register the AWS secret patterns in the test-pypi repository:

[mbacchi@hostname test-pypi]$ git-secrets --register-aws
[mbacchi@hostname test-pypi]$ git config -l | grep secrets
secrets.providers=git secrets --aws-provider
secrets.patterns=[A-Z0-9]{20}
secrets.patterns=("|')?(AWS|aws|Aws)?_?(SECRET|secret|Secret)?_?(ACCESS|access|Access)?_?(KEY|key|Key)("|')?\s*(:|=>|=)\s*("|')?[A-Za-z0-9/\+=]{40}("|')?
secrets.patterns=("|')?(AWS|aws|Aws)?_?(ACCOUNT|account|Account)_?(ID|id|Id)?("|')?\s*(:|=>|=)\s*("|')?[0-9]{4}\-?[0-9]{4}\-?[0-9]{4}("|')?
secrets.allowed=AKIAIOSFODNN7EXAMPLE
secrets.allowed=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Now using an AWS credential file that I made invalid immediately after creating(seriously, this won’t work if you try using it), I will scan the repository for secrets:

[mbacchi@hostname test-pypi]$ cat aws-credentials
[default]
aws_access_key_id=AKIAISITMFSJISKWFAJX
aws_secret_access_key=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

[mbacchi@hostname test-pypi]$ git-secrets --scan -r .
./aws-credentials:2:aws_access_key_id=AKIAISITMFSJISKWFAJX
./aws-credentials:3:aws_secret_access_key=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

[ERROR] Matched one or more prohibited patterns

Possible mitigations:
- Mark false positives as allowed using: git config --add secrets.allowed ...
- Mark false positives as allowed by adding regular expressions to .gitallowed at repository's root directory
- List your configured patterns: git config --get-all secrets.patterns
- List your configured allowed patterns: git config --get-all secrets.allowed
- List your configured allowed patterns in .gitallowed at repository's root directory
- Use --no-verify if this is a one-time false positive

This could be setup for other patterns of course but AWS is the most straightforward and frequently committed to Github repositories. You can also scan the commit history for previously committed secrets.

Using the steps above, you could create a pre-commit hook for your git repository to prevent secrets from being committed to git.

Hope these suggestions make their way into your personal or team’s workflow to make you more secure!

Using dnsmasq on Asuswrt-merlin to ignore MAC addresses

Wed, 13 Dec 2017 15:06:30 -0600

Here’s another ‘for posterity’ type post.

I forgot I had configured my WiFi router to ignore certain MAC addresses and when I tried to put that machine on the Network today I had a rude awakening. Using the asuswrt-merlin firmware on my RT-AC66U router, I had configured things to allow 2 machines to be served DHCP from another PC so that I could run Cobbler on that machine and boot/kickstart these two hosts just by hitting the power button. But when I disconnected this Cobbler server at the beginning of my renovation, I forgot the MAC addresses were being ignored by the router.

In order to enable these MAC addresses on the WiFi router I first tried to comment the lines out, but then instead moved the config file to root’s home dir. Also I had to reboot the router:

admin@RT-AC66U:/tmp/home/root# mv /jffs/configs/dnsmasq.conf.add .
admin@RT-AC66U:/tmp/home/root# ls
dnsmasq.conf.add
admin@RT-AC66U:/tmp/home/root# more dnsmasq.conf.add
# 12/13/2017 comment these out while other dhcp server isn't available
#dhcp-host=ec:a8:6b:f4:ce:01,ignore
#dhcp-host=ec:a8:6b:f5:a4:ed,ignore
admin@RT-AC66U:/tmp/home/root# ls -ltr /jffs/configs/
admin@RT-AC66U:/tmp/home/root# reboot

Ajax/jQuery on AWS Lambda

Sun, 03 Dec 2017 15:10:12 -0600

I’ve been trying to learn more about AWS Lambda, at the same time learning some web development. For a newcomer to web application development, there is the question of whether you work on leveling up with traditional server based apps or “serverless”, aka FaaS, aka Lambda. I wanted to try converting a simple Flask app to Lambda, this is how I did it.

What project should I work on?

Borrowing inspiration from Katelyn Lemay’s tweet I started with a straightforward application challenge from freeCodeCamp which displays quotes when the user clicks a button. I figured this would exercise the muscles required to do some client side scripting to experiment with how that worked on Lambda. I also wanted to use Python vs. straight Javascript for the backend. I found out that there are some interesting differences between running a Flask application on a webserver vs. running it as a function on Lambda.

Here’s the basic application I created (again, the idea and some css borrowed from @klemay): https://github.com/mbacchi/flask-quotes

I’m using Zappa to deploy the Python app to AWS Lambda. Zappa makes it easy to upload an app without having to do a lot of preparation on AWS and packaging the bits manually. This is probably not useful to everyone but I wanted to avoid the overhead of running Lambda in order to get the app functioning before I learned it in detail, and Zappa helped with that.

What’s so interesting about that?

This rudimentary application introduces two things to me. First I haven’t done Ajax/jQuery in a Python application before. That’s not exciting to many folks but when the concept is new to you its hard to work out the mechanics of the process. Second I knew the term Document Object Model but didn’t know how to manipulate it nor how to initiate the Ajax call.

jQuery is intimidating too, but approachable if you start with the fundamentals. Now I know $.getJSON() means roughly the same thing as $.ajax() (give or take, its shorthand for the ajax call.) But wow is it confusing, for example what are the mechanics of not providing the optional second parameter (data), yet allowing the third optional handler (success) to work? I use that here.

Local webserver vs. Lambda

So after getting this to work using a local Flask server, how do we transition to serving it from AWS? I initially tried to just upload the same app with Zappa to Lambda, but while the inital load of the page worked, any time I clicked the “New Quote” button nothing happened. This had me furiously searching google for whether or not I needed a full REST API to work on AWS Lambda or some other changes. I did find out I probably needed to enable CORS on the API Gateway.

There were other potential answers for why it wasn’t working that I found out about. Did I need to use lambda.invoke or some similar mechanism to call the new_quote function via the onClick action in the browser? Did I need to setup a full REST API with Cognito, or some other API Gateway configuration that I was missing?

In the end I realized I was calling the new_quote function using a relative path, but I needed to call the full URL to the API Gateway that I already knew from performing “zappa deploy dev” earlier, so when I ran “zappa update” it used the same gateway. That change looked like:

(venv) [user1@hostname flask-quotes]$ git diff 3d9eccd static/js/quote.js
diff --git a/static/js/quote.js b/static/js/quote.js
index aa51353..5df7a88 100644
--- a/static/js/quote.js
+++ b/static/js/quote.js
@@ -1,6 +1,6 @@
 function newQuote() {
     $(document).ready(function() {
-        $.getJSON('/new_quote',
+        $.getJSON('https://qlyaou0kc3.execute-api.us-east-1.amazonaws.com/dev/new_quote',
             function (data) {
                 $("#quote-text").text(data);
             });
(venv) [user1@hostname flask-quotes]$

For reference the Zappa settings for the project were:

(venv) [user1@hostname flask-quotes]$ cat zappa_settings.json
{
    "dev": {
        "app_function": "app.app",
        "aws_region": "us-east-1",
        "profile_name": "default",
        "project_name": "flask-quotes",
        "runtime": "python3.6",
        "s3_bucket": "zappa-p1kcu59iy",
	"cors": true
    }
}

Overall this was a fun challenge and although I banged my head on a few things I learned a ton about Lambda, Javascript, Ajax, jQuery, client side scripting and Zappa.

Using Zappa as an AWS Lambda Python Framework

Fri, 01 Dec 2017 15:15:15 -0600

I spent a little time with Zappa today which is an AWS Lambda (aka “serverless”) framework for Python. Its not hard to create a very basic Flask application, then invoke Zappa to perform the many manual steps of creating a Lambda function, resulting in a URL where your application is running.

First, I verified I was using Python 3.6, then I created a Python Virtual environment and installed Zappa and then Flask.

[user1@hostname hello]$ python3 --version
Python 3.6.2
[user1@hostname hello]$ mkvenv
[user1@hostname hello]$
[user1@hostname hello]$ . venv/bin/activate
(venv) [user1@hostname hello]$ pip install zappa
Collecting zappa
  Using cached zappa-0.45.1-py3-none-any.whl
...
Collecting cfn_flip>=0.2.5 (from troposphere>=1.9.0->zappa)
  Using cached cfn_flip-0.2.5.tar.gz
Installing collected packages: PyYAML, six, python-dateutil, docutils, jmespath, botocore, s3transfer,
boto3, click, placebo, kappa, durationpy, wheel, wsgi-request-logger, toml, urllib3, chardet, certifi,
idna, requests, argcomplete, Unidecode, python-slugify, future, lambda-packages, tqdm, hjson, Werkzeug,
base58, cfn-flip, troposphere, zappa
  Running setup.py install for PyYAML ... done
  Running setup.py install for placebo ... done
  Running setup.py install for kappa ... done
  Running setup.py install for durationpy ... done
  Running setup.py install for wsgi-request-logger ... done
  Running setup.py install for toml ... done
  Running setup.py install for future ... done
  Running setup.py install for lambda-packages ... done
  Running setup.py install for hjson ... done
  Running setup.py install for cfn-flip ... done
  Running setup.py install for troposphere ... done
Successfully installed PyYAML-3.12 Unidecode-0.4.21 Werkzeug-0.12 argcomplete-1.9.2 base58-0.2.4 boto3-1.4.8
botocore-1.8.6 certifi-2017.11.5 cfn-flip-0.2.5 chardet-3.0.4 click-6.7 docutils-0.14 durationpy-0.5
future-0.16.0 hjson-3.0.1 idna-2.6 jmespath-0.9.3 kappa-0.6.0 lambda-packages-0.19.0 placebo-0.8.1
python-dateutil-2.6.1 python-slugify-1.2.4 requests-2.18.4 s3transfer-0.1.12 six-1.11.0 toml-0.9.3
tqdm-4.19.1 troposphere-2.1.1 urllib3-1.22 wheel-0.30.0 wsgi-request-logger-0.4.6 zappa-0.45.1

(venv) [user1@hostname hello]$ pip install Flask
Collecting Flask
  Using cached Flask-0.12.2-py2.py3-none-any.whl
Requirement already satisfied: click>=2.0 in ./venv/lib/python3.6/site-packages (from Flask)
Collecting itsdangerous>=0.21 (from Flask)
  Using cached itsdangerous-0.24.tar.gz
Collecting Jinja2>=2.4 (from Flask)
  Downloading Jinja2-2.10-py2.py3-none-any.whl (126kB)
    100% |████████████████████████████████| 133kB 2.6MB/s
Requirement already satisfied: Werkzeug>=0.7 in ./venv/lib/python3.6/site-packages (from Flask)
Collecting MarkupSafe>=0.23 (from Jinja2>=2.4->Flask)
  Using cached MarkupSafe-1.0.tar.gz
Building wheels for collected packages: itsdangerous, MarkupSafe
  Running setup.py bdist_wheel for itsdangerous ... done
  Stored in directory: /home/user1/.cache/pip/wheels/fc/a8/66/24d655233c757e178d45dea2de22a04c6d92766abfb741129a
  Running setup.py bdist_wheel for MarkupSafe ... done
  Stored in directory: /home/user1/.cache/pip/wheels/88/a7/30/e39a54a87bcbe25308fa3ca64e8ddc75d9b3e5afa21ee32d57
Successfully built itsdangerous MarkupSafe
Installing collected packages: itsdangerous, MarkupSafe, Jinja2, Flask
Successfully installed Flask-0.12.2 Jinja2-2.10 MarkupSafe-1.0 itsdangerous-0.24

I created a very simple Hello World app in Flask(borrowed from the Flask quickstart here):

(venv) [user1@hostname hello]$ cat app.py
from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello, Lambda users!'

With the Flask “application” in my current directory, I was able to run “zappa init” to get everything built. It asks some questions:

(venv) [user1@hostname hello]$ zappa init

███████╗ █████╗ ██████╗ ██████╗  █████╗
╚══███╔╝██╔══██╗██╔══██╗██╔══██╗██╔══██╗
  ███╔╝ ███████║██████╔╝██████╔╝███████║
 ███╔╝  ██╔══██║██╔═══╝ ██╔═══╝ ██╔══██║
███████╗██║  ██║██║     ██║     ██║  ██║
╚══════╝╚═╝  ╚═╝╚═╝     ╚═╝     ╚═╝  ╚═╝

Welcome to Zappa!

Zappa is a system for running server-less Python web applications on AWS Lambda and AWS API Gateway.
This `init` command will help you create and configure your new Zappa deployment.
Let's get started!

Your Zappa configuration can support multiple production stages, like 'dev', 'staging', and 'production'.
What do you want to call this environment (default 'dev'):

AWS Lambda and API Gateway are only available in certain regions. Let's check to make sure you have a profile
set up in one that will work.
Okay, using profile default!

Your Zappa deployments will need to be uploaded to a private S3 bucket.
If you don't have a bucket yet, we'll create one for you too.
What do you want call your bucket? (default 'zappa-fysh7qlsu'):

It looks like this is a Flask application.
What's the modular path to your app's function?
This will likely be something like 'your_module.app'.
We discovered: app.app
Where is your app's function? (default 'app.app'):

You can optionally deploy to all available regions in order to provide fast global service.
If you are using Zappa for the first time, you probably don't want to do this!
Would you like to deploy this application globally? (default 'n') [y/n/(p)rimary]: n

Okay, here's your zappa_settings.json:

{
    "dev": {
        "app_function": "app.app",
        "aws_region": "us-east-1",
        "profile_name": "default",
        "project_name": "hello",
        "runtime": "python3.6",
        "s3_bucket": "zappa-fysh7qlsu"
    }
}

Does this look okay? (default 'y') [y/n]: y

Done! Now you can deploy your Zappa application by executing:

	$ zappa deploy dev

After that, you can update your application code with:

	$ zappa update dev

To learn more, check out our project page on GitHub here: https://github.com/Miserlou/Zappa
and stop by our Slack channel here: https://slack.zappa.io

Enjoy!,
 ~ Team Zappa!

Simple, right? After that I deployed the application:

(venv) [user1@hostname hello]$ zappa deploy dev

Calling deploy for stage dev..
Creating hello-dev-ZappaLambdaExecutionRole IAM Role..
Creating zappa-permissions policy on hello-dev-ZappaLambdaExecutionRole IAM Role.
Downloading and installing dependencies..
 - sqlite==python36: Using precompiled lambda package
Packaging project as zip.
Uploading hello-dev-1512173647.zip (5.7MiB)..
100%|██████████████████████████████████████████████████████████████████████████████████████████| 6.02M/6.02M [00:01<00:00, 3.63MB/s]
Scheduling..
Scheduled hello-dev-zappa-keep-warm-handler.keep_warm_callback with expression rate(4 minutes)!
Uploading hello-dev-template-1512173657.json (1.6KiB)..
100%|██████████████████████████████████████████████████████████████████████████████████████████| 1.59K/1.59K [00:00<00:00, 10.7KB/s]
Waiting for stack hello-dev to create (this can take a bit)..
100%|████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:15<00:00,  5.89s/res]
Deploying API Gateway..
Deployment complete!: https://te5p0mowh3.execute-api.us-east-1.amazonaws.com/dev

After uploading, you can use curl to connect:

1
2

(venv) [user1@hostname hello]$ curl https://te5p0mowh3.execute-api.us-east-1.amazonaws.com/dev
Hello, Lambda users!

And here’s the AWS console showing the invocation count of the lambda function:

To remove the test Lambda I created, use “zappa undeploy”:

(venv) [user1@hostname hello]$ zappa undeploy dev
Calling undeploy for stage dev..
Are you sure you want to undeploy? [y/n] y
Deleting API Gateway..
Waiting for stack hello-dev to be deleted..
Unscheduling..
Unscheduled hello-dev-zappa-keep-warm-handler.keep_warm_callback.
Deleting Lambda function..
Done!

Docker error IPv4 forwarding is disabled

Fri, 29 Sep 2017 15:29:29 -0600

Another common error is that the docker daemon cannot connect to the outside world to download anything during build time. This can be corrected in a number of ways, but I have done it thusly. The error is commonly encountered as you are trying to build a docker image, the warning "[Warning] IPv4 forwarding is disabled. Networking will not work." tells you that you need to enable IPv4 forwarding.

Step 10 : RUN set -x && pip install -U pip setuptools && pip install -r /tmp/requirements/dev.txt ...
 ---> [Warning] IPv4 forwarding is disabled. Networking will not work.
 ---> Running in b4b9fb134518
+ pip install -U pip setuptools
Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection
broken by NewConnectionError(Failed to establish a new connection: [Errno -2]
Name or service not known,): /simple/pip/

First setup the docker daemon config file to have a DNS server or two available to make requests of:

[mbacchi@centos7 ~]$ cat /etc/docker/daemon.json
{
    "dns": ["10.1.1.254","8.8.8.8","10.115.1.220"],
    "live-restore": true
}

You can see above that I have multiple local(intranet vpn) nameservers as well as the Google DNS server 8.8.8.8. This would require a docker daemon restart to be enabled(systemctl restart docker), but we will wait for the next step to be completed before performing that daemon restart.

Now we must enable forwarding between interfaces in the kernel. This will be done using the sysctl parameter net.ipv4.ip_forward. It can be set once from the command line but upon system restart will not be retained, so I set it in the docker sysctl file:

1
2
3

[mbacchi@centos7 ~]$ cat  /usr/lib/sysctl.d/99-docker.conf
fs.may_detach_mounts=1
net.ipv4.ip_forward=1

This will then set the net.ipv4.ip_forward variable to true every time the docker daemon is started or restarted.

Now we can perform that restart and make sure both of these changes are enabled.

1
2

[mbacchi@centos7 ~]$ sudo systemctl restart docker
[mbacchi@centos7 ~]$

Docker socket group permissions

Thu, 28 Sep 2017 15:32:12 -0600

I always forget this when trying to run docker as a non-root user, so documenting it for posterity.

If you get an error connecting to the docker daemon as a standard user, such as:

1
2

[mbacchi@centos7 ~]$ docker ps
Cannot connect to the Docker daemon. Is the docker daemon running on this host?

And you’re sure your docker daemon is actually up:

[mbacchi@centos7 ~]$ sudo systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2017-09-28 17:08:49 EDT; 8min ago
     Docs: http://docs.docker.com
 Main PID: 3375 (dockerd-current)
   CGroup: /system.slice/docker.service
           ├─3375 /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --defaul...
           └─3380 /usr/bin/docker-containerd-current -l unix:///var/run/docker/libcontainerd/docker-containerd.sock -...

Sep 28 17:08:48 centos7 dockerd-current[3375]: time="2017-09-28T17:08:48.265161468-04:00" level=info msg="Graph ...onds"
Sep 28 17:08:48 centos7 dockerd-current[3375]: time="2017-09-28T17:08:48.265978330-04:00" level=warning msg="mou...ound"
Sep 28 17:08:48 centos7 dockerd-current[3375]: time="2017-09-28T17:08:48.266206907-04:00" level=info msg="Loadin...art."
Sep 28 17:08:48 centos7 dockerd-current[3375]: ......time="2017-09-28T17:08:48.473174045-04:00" level=info msg="...true"
Sep 28 17:08:48 centos7 dockerd-current[3375]: time="2017-09-28T17:08:48.840829045-04:00" level=info msg="Defaul...ress"
Sep 28 17:08:49 centos7 dockerd-current[3375]: time="2017-09-28T17:08:48.998079496-04:00" level=info msg="Loadin...one."
Sep 28 17:08:49 centos7 dockerd-current[3375]: time="2017-09-28T17:08:48.998264761-04:00" level=info msg="Daemon...tion"
Sep 28 17:08:49 centos7 dockerd-current[3375]: time="2017-09-28T17:08:48.998296244-04:00" level=info msg="Docker....12.6
Sep 28 17:08:49 centos7 systemd[1]: Started Docker Application Container Engine.
Sep 28 17:08:49 centos7 dockerd-current[3375]: time="2017-09-28T17:08:49.028663143-04:00" level=info msg="API li...sock"
Hint: Some lines were ellipsized, use -l to show in full.

The solution I’ve found most straightforward (but rather apathetic on the security front) is to change the permissions of the docker.sock file, but your userid must be in the dockerroot group:

[mbacchi@centos7 ~]$ grep docker /etc/group
dockerroot:x:990:mbacchi
[mbacchi@centos7 ~]$  ls -ltr /var/run/docker.sock
srw-rw----. 1 root root 0 Sep 28 17:08 /var/run/docker.sock
[mbacchi@centos7 ~]$  sudo chown root:dockerroot /var/run/docker.sock
[mbacchi@centos7 ~]$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

Empty greenhouse.io job posts

Thu, 24 Aug 2017 15:34:16 -0600

I was interested in a job description a company posted via Twitter, but I couldn’t view the text and didn’t understand why. This wasn’t the first time this happened so decided to dig into the problem and determine if it was an issue with my environment or the website. My initial assumption was there was some problem with the company’s website. But they must test that these postings are visible occasionally right? Looking at the page source there was no job description content, but rather this URL:

https://app.greenhouse.io/embed/job_board/js?for=fastly

Well this was a little unexpected but not totally surprising. They use a service to host all their job openings and then embed it in an iframe on their site. Could the iframe be a problem? I did a google search and found no hits whatsoever. How could I be the first person to encounter this? I’m not an experienced web developer but have been teaching myself some Python Pyramid stuff lately. Lets see what the developer console says about the page. It indicated the error:

`1`	`GET https://app.greenhouse.io/embed/job_board/js?for=fastly net::ERR_BLOCKED_BY_CLIENT`

This told me that the server wasn’t the problem but instead my client was blocking the content. What now? Oh, ya, I have extensions such as Fair Adblocker and Privacy Badger installed to prevent sites from tracking me. I looked to see if those were blocking any greenhouse.io sites and sure enough, there were 4 URLs being blocked:

app.greenhouse.io
boards.greenhouse.io
boards-use1-cdn.greenhouse.io
boards-api.greenhouse.io

Once I disabled full blocking on those sites, and only enabled blocking of cookies, I was able to view the iframe with the job description.

Hope this helps somebody (maybe even me) sometime in the future.

Using Docker to create an ad hoc Yum repository

Fri, 27 Jan 2017 15:37:47 -0600

Docker can be used to quickly create and serve many services, one such example is serving RPMs via Yum in an ad hoc manner. Have you ever wanted to create a Yum repository consisting of some RPMs very quickly to be used for testing purposes? I did this week. I could have created the repo and installed a web server on any machine. But what if we had the RPMs and the Yum repository both dynamically hosted on the servers where the Chef cookbooks were being executed? That was my goal for the docker-yumrepo project, enabling the creation of an ad hoc Yum repository.

This tool has prerequisites that createrepo_c and docker to be installed prior. You will place the RPMs you want in the repo in the src directory(after running ‘make src’), then it will build and runs a docker image. This will serve the repository on port 80 (so make sure you don’t have another webserver on port 80.) Voila, you now have a yum repository accessible on the URL:

http://localhost/docker-yumrepo

To use this you can install a Yum repo config file such as:

[user@centos docker-yumrepo]$ cat /etc/yum.repos.d/docker-yumrepo.repo
[docker-yumrepo]
name=docker-yumrepo
baseurl=http://localhost/docker-yumrepo
enabled=1
gpgcheck=0

Instructions are also available in the README.

Enjoy!

Vagrant NFS synced_folders

Sun, 15 Jan 2017 15:40:23 -0600

After upgrading to Fedora 25 yesterday, Vagrant 1.8.5 in the updates repo was unusable with VirtualBox for 2 reasons:

The newest centos/7 box didn’t use vbox guest additions for shared folders, forcing the use of NFS
Vagrant issue 8138 which is not fixed in the newest RPM from Vagrant means I had to install from source.

This required two workarounds in my Vagrantfile:

The setting config.vm.synced_folder requires the ’type: "nfs"’ parameter.
The config.vm.provision section required the ‘chef.synced_folder_type = "nfs"’ parameter.

This looks like the following now:

...
  config.vm.network "private_network", ip: "10.10.10.12"
  config.vm.synced_folder "django3", "/django3", type: "nfs"

  config.berkshelf.enabled = true

  config.vm.provision :chef_solo do |chef|
    chef.synced_folder_type = "nfs"
    chef.cookbooks_path = "cookbooks"
    chef.roles_path = "roles"
...

Matt Bacchi

Using the VSCode Claude Code Extension with Bedrock and Claude Sonnet 4.5

Overview

Unfortunate Missing Features and a Bug/Regression with the Claude Code Extension

Create IAM User and Bedrock Permissions

Setup AWS Profile

Enable Bedrock Claude Sonnet 4.5 and Validate Access

Install and Configure VSCode

Did it work?

Conclusion

Resources

Firecracker Virtualization Overview

Firecracker High Level Overview

Configuration

Initialization

Enhanced Isolation using Jailer

Demonstration

Firecracker Ecosystem Discrepancies Compared to Lambda

Synchronous

Asynchronous

The Cold Truth

Conclusion

Installing Nvidia Datacenter GPU Manager on Amazon Linux 2023

What is the Datacenter GPU Manager

Amazon Linux 2 End Of Life

Installing DCGM on AL2023

Conclusion

Switching to the Terraform S3 Backend with Native State File Locks

What is the Terraform S3 Backend

What is State File Locking

AWS S3 Conditional Writes

S3 Conditional Write Support Added in Terraform v1.10

Configuring the S3 Backend to Use Native State File Locking

Terraform Configuration Specifics

Version Constraints

Sample Configuration

State File Locking in Action

Error from DynamoDB State File Locking

Error from S3 Backend Native State File Locking

Dealing with Stale State File Locks

Delete Your Old DynamoDB Tables

Summary

A Survey of Serverless Sustainability Trends

What is Sustainability

Sustainable Workloads

Region Selection

Over-provisioned Capacity and Time Shifting

Time Shifting on Kubernetes

Summary

Limitations of AWS EC2 Image Builder Lifecycle Policies

Standard use case

Why so many AMIs?

EC2 Image Builder Lifecycle Policies

Limitations of the current Lifecycle Policies service

Recommended Solutions (Feature Requests)

Conclusion

S3 Bucket Takeover Neutralization

What is Hijacking?

Preventing S3 Bucket Takeover

Decommissioning Process

Conclusion

Event Driven Processing of ip-ranges.json

What is Event Driven Architecture?

Why use EDA for data processing?

SNS notification for ip-ranges.json

Data organization of the ip-ranges.json file

DynamoDB Single Table Design

Lambda Function Overview

AWS CDK configuration

Wrapping Up

Resources

New and Improved Job Searching Resources in 2024

Ventureloop

Otta

Climatebase

StillHiring Today

Bundling Go Lambda Functions with the AWS CDK

AWS CDK Deployment Using Typescript

Source Organization

Bundling Go Functions