This contains various commands and information that I find useful for AWS work.

Install latest version of AWS CLI on linux

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

Resource: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html

UI

Backup instance manually

Go to your instance
Right click and select Image from the dropdown
Click Create Image
Give your backup a name and description
Click No reboot if you want your instance to stay in a running state
Click Create Image
At this point you should be able to find the AMI that is associated with your backup under AMIs. Give the AMI a more descriptive name if you'd like.

Resource: https://n2ws.com/blog/how-to-guides/automate-amazon-ec2-instance-backup

Parameter Store location

Login
Search for Systems Manager
Click on Parameter Store in the menu on the left-hand side

EC2

Use env vars

Run the following with the proper values:

export AWS_ACCESS_KEY_ID=AKIAI44QH8DHBEXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Set up credentials file

You can run aws configure if you want a guided setup. Alternatively, you can add the following to ~/.aws/credentials:

[default]
aws_access_key_id = <YOUR_ACCESS_KEY_ID>
aws_secret_access_key = <YOUR_SECRET_ACCESS_KEY>

If you don't opt for the guided setup, don't forget to set the region in ~/.aws/config:

[default]
region = <YOUR_REGION>

Populate env vars with credentials file

export AWS_ACCESS_KEY_ID=$(aws configure get aws_access_key_id)
export AWS_SECRET_ACCESS_KEY=$(aws configure get aws_secret_access_key)

Resource: https://github.com/kubernetes/kops/blob/master/docs/getting_started/aws.md#setup-iam-user

Populate config file with env vars

aws configure set region $AWS_REGION \
--profile default

Multiple profiles

Your credentials file will probably look something like this:

[default]
aws_access_key_id = <YOUR_ACCESS_KEY_ID>
aws_secret_access_key = <YOUR_SECRET_ACCESS_KEY>

[notdefault]
aws_access_key_id = <YOUR_OTHER_ACCESS_KEY_ID>
aws_secret_access_key = <YOUR_OTHER_SECRET_ACCESS_KEY>

To use the notdefault profile, run the following command:

export AWS_PROFILE=notdefault

Use temp credentials

Add the following to your credentials file:

[temp]
aws_access_key_id = <YOUR_TEMP_ACCESS_KEY_ID>
aws_secret_access_key = <YOUR_TEMP_SECRET_ACCESS_KEY>
aws_session_token = <YOUR_SESSION_TOKEN>

Then run this command:

export AWS_PROFILE=temp

Resource: https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html

Use env vars for temp credentials

Run the following with the proper values:

export AWS_ACCESS_KEY_ID=AKIAI44QH8DHBEXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export AWS_SESSION_TOKEN=AQoDYXdzEJr...<remainder of security token>

Resource: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html

List instances

aws ec2 describe-instances

Get number of instances

aws ec2 describe-instances --query 'Reservations[*].Instances[*].[InstanceId]'
--output text | wc -l

Resource: https://stackoverflow.com/questions/40164786/determine-how-many-aws-instances-are-in-a-zone

Get running instances

aws ec2 describe-instances --filters Name=instance-state-name,Values=running

Resource: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Filtering.html

Reboot all instances in a region

aws ec2 reboot-instances --instance-ids $(aws ec2 describe-instances --query "Reservations[*].Instances[*].InstanceId" | jq '.[]|.[0]' -r)

Assign an elastic IP to an instance

aws ec2 associate-address --allocation-id eipalloc-<eip id> --instance-id <the instance id>

Create instance with a tag

aws ec2 run-instances --image-id ami-xxxxxxx --count 1 --instance-type t2.medium --key-name MyKeyPair --security-group-ids sg-xxxxxx --subnet-id subnet-xxxxxx --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=my-test-instance}]'

Resource: https://serverfault.com/questions/724501/how-to-add-a-tag-when-launching-an-ec2-instance-using-aws-clis

Create instance using security group name

aws ec2 run-instances --image-id ami-xxxxxxxx --count 1 --instance-type t1.micro --key-name MyKeyPair --security-groups MySecurityGroup

Get security group id from group name

sg_name=sg-bla
aws ec2 describe-security-groups --filters Name=group-name,Values=$sg_name --query "SecurityGroups[*].[GroupId]" --output text

Resource: https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-security-groups.html

Add new ingress rule to security group

Using the value from the example above, we can get the id of a security group and update it to do things such as allowing a codebuild instance temporary SSH access to an instance:

aws ec2 authorize-security-group-ingress \
    --group-id $sg_id \
    --protocol tcp \
    --port 22 \
    --cidr $(curl ifconfig.me)/32

Resource: https://fossies.org/linux/aws-cli/awscli/examples/ec2/authorize-security-group-ingress.rst

List instances with filtering

This example in particular will get you all your m1.micro instances.

aws ec2 describe-instances --filters "Name=instance-type,Values=m1.micro"

List instance by instance id

aws ec2 describe-instances --instance-ids i-xxxxx

Destroy instance

aws ec2 terminate-instances --instance-ids <instance id(s)>

If you want to terminate multiple instances, be sure to use this format:

id1 id2 id3

Get info about a specific AMI by name and output to JSON

aws ec2 describe-images --filters "Name=name,Values=<AMI Name>" --output json

Get AMI id from AMI name

AMI_ID=$(aws ec2 describe-images --filters "Name=name,Values=THEAMINAME" --query 'sort_by(Images, &CreationDate)[-1].[ImageId]' --output text)
echo $AMI_ID

Resource: https://stackoverflow.com/questions/40835953/how-to-find-ami-id-of-centos-7-image-in-aws-marketplace

Get AMI id with some python

This uses the run_cmd() function found in /python-notes/.

import json

def get_ami_id(ec2_output):
    json.loads(ec2_output.decode('utf-8'))['Images'][0]['ImageId']

ec2_output = run_cmd('aws ec2 describe-images --filters "Name=name,Values=<AMI Name>" --output json')
ami_id = get_ami_id(ec2_output)
print(ami_id)

Deregister an AMI

aws ec2 deregister-image --image-id <ami id>

Get list of all instances with the state terminated

aws ec2 describe-instances --filters "Name=instance-state-name,Values=terminated"

List all instances that match a tag name and are running

aws ec2 describe-instances --filters "Name=tag:Name,Values=*somename*" "Name=instance-state-name,Values=running" | jq

Resources:
https://stackoverflow.com/questions/23936216/how-can-i-get-list-of-only-running-instances-when-using-ec2-describe-tags - filter by running instances
https://serverfault.com/questions/560337/search-ec2-instance-by-its-name-from-aws-command-line-tool - filter by instance name

Alternatively, if you want running instances, change Values=terminated to Values=running.

Get info about an AMI by product-code

aws --region <region> ec2 describe-images --owners aws-marketplace --filters Name=product-code,Values=<product code>

This is useful if you have the product code, and want more information (like the image ID). For CentOS, you can get the product code here. I started down this path when I was messing around with the code in this gist for automatically creating encrypted AMI's.

Resize ec2 instance

https://medium.com/@kenichishibata/resize-aws-ebs-4d6e2bf00feb

Show available subnets

aws ec2 describe-subnets

Attach volume at root

aws ec2 attach-volume --volume-id vol-xxxx --instance-id i-xxxx --device /dev/sda1

Resource: https://stackoverflow.com/questions/37142357/error-instance-does-not-have-a-volume-attached-at-root-dev-sda1

List snapshots

aws ec2 describe-snapshots --output json --query 'Snapshots[*].SnapshotId' --max-items 10 | head

Add custom tag for cloud-init finishing

aws ec2 create-tags --resources `ec2metadata --instance-id` --tags Key=BootstrapStatus,Value=complete

Resource: https://stackoverflow.com/questions/33019093/how-do-detect-that-cloud-init-completed-initialization

CodeCommit

CodeCommit is a service in AWS that provides an option for private git repos. Access can be dictated by IAM which is nice.

CodeBuild

Pretty decent, relatively up-to-date tutorial on using CodeBuild and CodeCommit to autobuild AMI's: https://aws.amazon.com/blogs/devops/how-to-create-an-ami-builder-with-aws-codebuild-and-hashicorp-packer/

If you want to use the gist to create encrypted AMI's mentioned above, be sure to specify aws_region, aws_vpc, aws_subnet, and ssh_username in the variables section.

Build the proper IAM role

Login to the UI
Click on IAM
Click Roles
Click Create role
Click EC2, then click Next: Permissions
Search for CodeCommit, check the box next to AWSCodeCommitReadOnly
Click Next: Tags
Give it some tags if you'd like, click Next: Review
Specify a Role name, like CodeCommit-Read
Click Create role

Now we're cooking. Let's test it out by building an instance and not forgetting to assign it the CodeCommit-Read IAM role. You can figure this part out.

Cloning into a repo

Once you've got a working instance:

SSH into it
Escalate privileges:

sudo su

Install the awscli with pip:

pip install awscli

Run these commands and be sure to change the region to match the one you're using:

git config --system credential.https://git-codecommit.us-west-2.amazonaws.com.helper '!aws codecommit credential-helper $@'
git config --system credential.https://git-codecommit.us-west-2.amazonaws.com.UseHttpPath true

Run this command and be sure to change the region to match the one you're working with:

aws configure set region us-west-2

Clone your repo:

git clone https://git-codecommit.us-west-2.amazonaws.com/v1/repos/GREATREPONAME

Resources:
https://jameswing.net/aws/codecommit-with-ec2-role-credentials.html
https://stackoverflow.com/questions/46164223/aws-pull-latest-code-from-codecommit-on-ec2-instance-startup - This site got me to the above site, but had incomplete information for their proposed solution.
https://jameswing.net/aws/codecommit-with-ec2-role-credentials.html - git config commands to authenticate

Integrating this in with CodeBuild

To get this to work with CodeBuild for automated and repeatable builds, I needed to do a few other things. Primarily, take advantage of the Parameter Store. When I was trying to build initially, my buildspec.yml looked something like this (basically emulating the one found in here):

---
version: 0.2

phases:
  pre_build:
    commands:
      - echo "Installing HashiCorp Packer..."
      - curl -qL -o packer.zip https://releases.hashicorp.com/packer/1.1.1/packer_1.1.1_linux_amd64.zip && unzip packer.zip
      - echo "Installing jq..."
      - curl -qL -o jq https://stedolan.github.io/jq/download/linux64/jq && chmod +x ./jq
      - echo "Validating kali.json"
      - ./packer validate kali.json
  build:
    commands:
      ### HashiCorp Packer cannot currently obtain the AWS CodeBuild-assigned role and its credentials
      ### Manually capture and configure the AWS CLI to provide HashiCorp Packer with AWS credentials
      ### More info here: https://github.com/mitchellh/packer/issues/4279
      - echo "Configuring AWS credentials"
      - curl -qL -o aws_credentials.json http://169.254.170.2/$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI > aws_credentials.json
      - aws configure set region $AWS_REGION
      - aws configure set aws_access_key_id `./jq -r '.AccessKeyId' aws_credentials.json`
      - aws configure set aws_secret_access_key `./jq -r '.SecretAccessKey' aws_credentials.json`
      - aws configure set aws_session_token `./jq -r '.Token' aws_credentials.json`
      - echo "Building HashiCorp Packer template, kali.json"
      - ./packer build kali.json
  post_build:
    commands:
      - echo "HashiCorp Packer build completed on `date`"

However, I was getting this obscure error message about authentication, and spent several hours messing around with IAM roles, but didn't have any luck. At some point, I eventually decided to try throwing a "parameter" in for the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. This worked great, but I noticed that whenever I tried the build again, I would run into the same issue as before. To fix it, I had to modify the buildspec.yml to look like this (obviously the values you have for your parameter store may vary depending on what you set for them):

---
version: 0.2

env:
  parameter-store:
    AWS_ACCESS_KEY_ID: "/CodeBuild/AWS_ACCESS_KEY_ID"
    AWS_SECRET_ACCESS_KEY: "/CodeBuild/AWS_SECRET_ACCESS_KEY"

phases:
  pre_build:
    commands:
      - echo "Installing HashiCorp Packer..."
      - curl -qL -o packer.zip https://releases.hashicorp.com/packer/1.1.1/packer_1.1.1_linux_amd64.zip && unzip packer.zip
      - echo "Installing jq..."
      - curl -qL -o jq https://stedolan.github.io/jq/download/linux64/jq && chmod +x ./jq
      - echo "Validating kali.json"
      - ./packer validate kali.json
  build:
    commands:
      ### HashiCorp Packer cannot currently obtain the AWS CodeBuild-assigned role and its credentials
      ### Manually capture and configure the AWS CLI to provide HashiCorp Packer with AWS credentials
      ### More info here: https://github.com/mitchellh/packer/issues/4279
      - echo "Configuring AWS credentials"
      - curl -qL -o aws_credentials.json http://169.254.170.2/$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI > aws_credentials.json
      - aws configure set region $AWS_REGION
      - aws configure set aws_access_key_id `./jq -r '.AccessKeyId' aws_credentials.json`
      - aws configure set aws_secret_access_key `./jq -r '.SecretAccessKey' aws_credentials.json`
      - aws configure set aws_session_token `./jq -r '.Token' aws_credentials.json`
      - echo "Building HashiCorp Packer template, kali.json"
      - ./packer build kali.json
  post_build:
    commands:
      - echo "HashiCorp Packer build completed on `date`"

At this point, everything is working consistently with the IAM role mentioned previously being specified in the packer file (this is a snippet):

"variables": {
    "iam_role": "CodeCommit-Read"
  },

  "builders": [{
    "iam_instance_profile": "{{user `iam_role` }}",
  }],

Validate buildspec

python3 -c 'import yaml, sys; yaml.safe_load(sys.stdin)' < buildspec.yml

Resource: https://howchoo.com/python/how-to-validate-yaml-from-the-command-line

Debug Codebuild

You can get a shell to your codebuild system, which is incredibly helpful when it comes to debugging build problems.

Add the AmazonSSMFullAccess policy to your codebuild service role
Add a breakpoint to buildspec.yml:
Click Start build with overrides -> Advanced build overrides
Under environment, click the checkbox next to Enable session connection
Click Start build
Click the AWS Session Manager link that appears under build status to access the system

Once you're done debugging, type in codebuild-resume

Resource: https://dev.to/glnds/how-to-debug-and-trace-problems-in-aws-codebuild-1cgl

SSM

Get the information for a particular parameter (this will give you the encrypted value if the parameter is a SecureString): aws ssm get-parameter --name <nameofparam>

List parameters

aws ssm describe-parameters

Access a parameter

aws ssm get-parameter --name /path/to/parameter

Miscellaneous

Encrypt your pem file:

openssl rsa -des3 -in key.pem -out encrypted-key.pem
# Enter the pass phrase you've selected
mv encrypted-key.pem key.pem
chmod 400 key.pem

Remove the encryption:

openssl rsa -in key.pem -out key.open.pem
# Enter the pass phrase you've selected
mv key.open.pem key.pem

Resource: https://security.stackexchange.com/questions/59136/can-i-add-a-password-to-an-existing-private-key

Set up aws cli with pipenv on OSX

https://duseev.com/articles/perfect-aws-cli-setup/

S3

Create bucket

$BUCKET_NAME=my-bucket-is-neat
aws s3 mb s3://$BUCKET_NAME

Resource: https://linuxacademy.com/blog/amazon-web-services-2/aws-s3-cheat-sheet/

List buckets

aws s3 ls

List files in a bucket

aws s3 ls s3://target/

Download bucket

aws s3 sync s3://mybucket .

Resource: https://stackoverflow.com/questions/8659382/downloading-an-entire-s3-bucket/55061863

Copy file down

aws s3 cp s3://target/file.html file.html

Copy file up

aws s3 cp TEST s3://target

Copy folder up

aws s3 cp foldertocopy s3://bucket/foldertocopy --recursive

Resource: https://coderwall.com/p/rckamw/copy-all-files-in-a-folder-from-google-drive-to-aws-s3

Read buckets into an array

buckets=($(aws s3 ls |grep tf | awk '{print $3}' | tr " " "\n"))
# Print first element
echo ${buckets[1]}

Iterate over buckets

for b in "${buckets[@]}"; do echo "Bucket: $b"; done

Resource: https://stackoverflow.com/questions/15685736/how-to-extract-a-particular-element-from-an-array-in-bash

Empty bucket

Delete objects in the bucket:

bucket=bucketname
aws s3api delete-objects --bucket ${bucket} --delete "$(aws s3api list-object-versions --bucket ${bucket} --query='{Objects: Versions[].{Key:Key,VersionId:VersionId}}')"

Delete markers in the bucket:

bucket=bucketname
aws s3api delete-objects --bucket ${bucket} --delete "$(aws s3api list-object-versions --bucket ${bucket} --query='{Objects: DeleteMarkers[].{Key:Key,VersionId:VersionId}}')"

Resource: https://stackoverflow.com/questions/29809105/how-do-i-delete-a-versioned-bucket-in-aws-s3-using-the-cli

Delete bucket

aws s3 rb s3://bucketname --force

Resource: https://docs.aws.amazon.com/AmazonS3/latest/userguide/delete-bucket.html

Copy multiple folders to bucket

aws s3 cp /path/to/dir/with/folders/to/copy s3://bucket/ --recursive --exclude ".git/*"

Resource: https://superuser.com/questions/1497268/selectively-uploading-multiple-folders-to-aws-s3-using-cli

Set up S3 IAM for backup/restore

Storing aws credentials on an instance to access an S3 bucket can be a bad idea. Let's talk about what we need to do in order to backup/restore stuff from an S3 bucket safely:

Create Policy

Go to IAM
Policies
Create Policy
Policy Generator, or copy and paste JSON from here (YOLO) into Create Your Own Policy. This is the one I used:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::techvomit"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket name>/*"
            ]
        }
    ]
}

Create a Role

Go to Roles in IAM
Click Create role
Select EC2
Select EC2 again and click Next: Permissions
Find the policy you created previously
Click Next: Review
Give the Role a name and a description, click Create role

Assign the role to your instance

This will be the instance that houses the service that requires a backup and restore service (your S3 bucket).

In EC2, if the instance is already created, right click it, Instance Settings, Attach/Replace IAM Role
Specify the IAM role you created previously, click Apply.

Set up automated expiration of objects

This will ensure that backups don't stick around longer than they need to. You can also set up rules to transfer them to long term storage during this process, but we're not going to cover that here.
From the bucket overview screen:

Click Management
Click Add lifecycle rule
Specify a name, click Next
Click Next
Check Current version and Previous versions
Specify a desired number of days to expiration for both the current version and the previous versions, click Next
Click Save

Mount bucket as local directory

Warning, this is painfully slow once you have it set up.

Follow the instructions found on this site.

Then, run this script:

#!/bin/bash
folder="/tmp/folder"
if [ ! -d $folder ]; then
    mkdir $folder
fi

s3fs bucket_name $folder -o passwd_file=${HOME}/.passwd-s3fs -o volname="S3-Bucket"

Create IAM role to grant read access to an s3 bucket

If accessing from an ec2 instance, find your ec2 instance in the web UI, right click it -> Security -> Modify IAM Role. Otherwise, just open the IAM console
Click Roles -> Create role
Click EC2
Click Next: Permissions
Click Create policy
Click JSON
Copy the json from here:

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action":[
            "s3:ListBucket"
         ],
         "Resource":"arn:aws:s3:::awsexamplebucket"
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:GetObject"
         ],
         "Resource":"arn:aws:s3:::awsexamplebucket/*"
      }
   ]
}

Change awsexamplebucket to the name of your bucket and click Review policy
Specify a Name for the policy and click Create policy

Get id of KMS key associated with a bucket

aws s3api get-bucket-encryption --bucket $(aws s3 ls | grep -i bucketname | awk '{print $3}') | jq '.ServerSideEncryptionConfiguration.Rules[0].ApplyServerSideEncryptionByDefault.KMSMasterKeyID' | awk -F '/' '{print $2}' | tr -d '"'

Resources:
https://stackoverflow.com/questions/59203164/aws-s3-bucket-bulk-encryption-on-all-s3-buckets - solved most of the problem for jq parsing
https://gist.github.com/olih/f7437fb6962fb3ee9fe95bda8d2c8fa4 - jq cheatsheet

Boto

Create session

from boto3.session import Session

def create_session():
  session = Session(aws_access_key_id=access_key,aws_secret_access_key=secret_key,aws_session_token=session_token)
  return session

Resource: https://stackoverflow.com/questions/30249069/listing-contents-of-a-bucket-with-boto3

List buckets with boto3

def get_s3_buckets(session):
  s3 = session.resource('s3')
  print("Bucket List:")
  for bucket in s3.buckets.all():
    print(bucket.name)

Resource: https://stackoverflow.com/questions/36042968/get-all-s3-buckets-given-a-prefix

Show items in an s3 bucket

def list_s3_bucket_items(session, bucket):
  s3 = session.resource('s3')
  my_bucket = s3.Bucket(bucket)

  for file in my_bucket.objects.all():
    print(file.key)

List Users

def get_users(session):
  client = boto3.client('iam', aws_access_key_id=access_key, aws_secret_access_key=secret_key,aws_session_token=session_token)
  users = client.list_users()
  for key in users['Users']:
    print(key['UserName'])

Resource: https://stackoverflow.com/questions/46073435/how-can-we-fetch-iam-users-their-groups-and-policies

Get account id

def sts(session):
  sts_client = boto3.client('sts',aws_access_key_id=access_key,aws_secret_access_key=secret_key, aws_session_token=session_token)
  print(sts_client.get_caller_identity()['Account'])

Create ec2 instance with name

EC2_RESOURCE = boto3.resource('ec2')

def create_ec2_instance():
    instance = EC2_RESOURCE.create_instances(
        ImageId='ami-ID_GOES_HERE',
        MinCount=1,
        MaxCount=1,
        InstanceType='t2.micro',
        SecurityGroupIds = ["sg-ID_GOES_HERE"]
        KeyName='KEY_NAME_GOES_HERE',
        TagSpecifications=[
            {
                'ResourceType': 'instance',
                'Tags': [
                    {
                        'Key': 'Name',
                        'Value': 'INSTANCE_NAME_HERE'
                    }
                ]
            }
        ]
    )
    return instance[0]

Resources:
https://blog.ipswitch.com/how-to-create-an-ec2-instance-with-python
https://stackoverflow.com/questions/52436835/how-to-set-tags-for-aws-ec2-instance-in-boto3
http://blog.conygre.com/2017/03/27/boto-script-to-launch-an-ec2-instance-with-an-elastic-ip-and-a-route53-entry/

Allocate and associate an elastic IP

import boto3
from botocore.exceptions import ClientError

# Wait for instance to finish lanuching before assigning the elastic IP address
print('Waiting for instance to get to a running state, please wait...')
instance.wait_until_running();

EC2_CLIENT = boto3.client('ec2')

try:
    # Allocate an elastic IP
    eip = EC2_CLIENT.allocate_address(Domain='vpc')
    
    # Associate the elastic IP address with an instance launched previously
    response = EC2_CLIENT.associate_address(
                   AllocationId=eip['AllocationId'],
                   InstanceId='INSTANCE_ID_GOES_HERE'
               )
    print(response)
except ClientError as e:
    print(e)

Allocate existing elastic IP

EC2_CLIENT.associate_address(
    AllocationId='eipalloc-EXISTING_EIP_ID_GOES_HERE',
    InstanceId=INSTANCE_ID_GOES_HERE
)

Resource:
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/ec2-example-elastic-ip-addresses.html
http://blog.conygre.com/2017/03/27/boto-script-to-launch-an-ec2-instance-with-an-elastic-ip-and-a-route53-entry/

Wait for instance to finish starting

retries = 10
retry_delay = 10
retry_count = 0
instance[0].wait_until_running()
instance[0].reload()
while retry_count <= retries:
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    result = sock.connect_ex((instance[0].public_ip_address,22))
    if result == 0:
        print(f"The instance is up and accessible on port 22 at {instance[0].public_ip_address}")
        break
    else:
        print("Instance is still coming up, retrying . . . ")
        time.sleep(retry_delay)

Resource:
https://stackoverflow.com/questions/46379043/boto3-wait-until-running-doesnt-work-as-desired

Metadata

Get Credentials

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/

Resource:
https://gist.github.com/quiver/87f93bc7df6da7049d41

Get region

curl --silent 169.254.169.254/latest/dynamic/instance-identity/document | jq -r .region

Resource: https://gist.github.com/quiver/87f93bc7df6da7049d41

Get role-name

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/

The role name will be listed here.

Resource: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html

Get Account ID

curl http://169.254.169.254/latest/meta-data/identity-credentials/ec2/info/

Get public hostname

curl 169.254.169.254/latest/meta-data/public-hostname

Show configuration

aws configure list

Get account id

aws sts get-caller-identity | jq '.Account'

Resource: https://shapeshed.com/jq-json/#how-to-find-a-key-and-value

Go SDK

Stand up EC2 Instance

This accounts for the exceptionally annoying message: An error occurred (VPCIdNotSpecified) when calling the RunInstances operation: No default VPC for this user that does not have any solutions in sight unless you do some deep diving. Essentially this means that a default VPC isn't defined and subsequently you need to provide a subnet id:

package main

import (
    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/ec2"

    "fmt"
    "log"
)

func main() {

    // Get credentials from ~/.aws/credentials
    sess, err := session.NewSession(&aws.Config{
        Region: aws.String("us-west-2")},
    )

    // Create EC2 service client
    svc := ec2.New(sess)

    // Specify the details of the instance that you want to create.
    runResult, err := svc.RunInstances(&ec2.RunInstancesInput{
        ImageId:      aws.String("ami-id-here"),
        InstanceType: aws.String("t2.small"),
        MinCount:     aws.Int64(1),
        MaxCount:     aws.Int64(1),
        SecurityGroupIds: aws.StringSlice([]string{"sg-id-here"}),
        KeyName: aws.String("keypairname-here"),
        SubnetId:     aws.String("subnet-id-here"),
    })

    if err != nil {
        fmt.Println("Could not create instance", err)
        return
    }

    fmt.Println("Created instance", *runResult.Instances[0].InstanceId)

    // Add tags to the created instance
    _, errtag := svc.CreateTags(&ec2.CreateTagsInput{
        Resources: []*string{runResult.Instances[0].InstanceId},
        Tags: []*ec2.Tag{
            {
                Key:   aws.String("Name"),
                Value: aws.String("GoInstance"),
            },
        },
    })
    if errtag != nil {
        log.Println("Could not create tags for instance", runResult.Instances[0].InstanceId, errtag)
        return
    }

    fmt.Println("Successfully tagged instance")
}

Resources:
https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/ec2-example-create-images.html - starting point
https://gist.github.com/stephen-mw/9f289d724c4cfd3c88f2
https://stackoverflow.com/questions/50289221/unable-to-create-ec2-instance-using-boto3 - where I found some of the solution (for boto, which translated to this fortunately)
https://docs.aws.amazon.com/sdk-for-go/api/aws/#StringSlice

Stand up EC2 Instance with lambda

Modify the previous code to make it into a lambda function - find it here.

Next you'll need to get the binary for the function:

env GOOS=linux GOARCH=amd64 go build -o /tmp/main

With that, you'll need to zip it up:

zip -j /tmp/main.zip /tmp/main

At this point, you need to create the iam role:

Navigate to https://console.aws.amazon.com/iam/home#/roles
Click Create role
Click Lambda
Click Next: Permissions
Add the following policies:

AmazonEc2FullAccess
AWSLambdaBasicExecutionRole
AWSLambdaVPCAccessExecutionRole

Click Next: Tags
Give it a Name tag and click Next: Review
Give it a Role name such as "LambdaCreateEc2Instance"
Click Create role
Once it's completed, click the role and copy the Role ARN

Now, you'll need to run the following command to create the lambda function:

aws lambda create-function --function-name createEc2Instance --runtime go1.x \
  --zip-file fileb:///tmp/main.zip --handler main \
  --role <Role ARN copied previously>

Lastly, you'll need to populate all of the environment variables. To do this, you can use this script:

aws lambda update-function-configuration --function-name createEc2Instance \
    --environment "Variables={AMI=ami-id-here, INSTANCE_TYPE=t2.small, SECURITY_GROUP=sg-id-here, KEYNAME=keypairname-here, SUBNET_ID=subnet-id-here}"

Alternatively, you can set the values in the lambda UI by clicking Manage environment variables:

Screen-Shot-2020-10-02-at-5.07.57-PM

but this gets very tedious very quickly.

If you want to throw all of this into a Makefile to streamline testing, you could do something like this:

build:
	env GOOS=linux GOARCH=amd64 go build -o /tmp/main

deploy:
	zip -j /tmp/main.zip /tmp/main
	bash scripts/create_function.sh
	bash scripts/create_env_vars.sh

run:
	aws lambda invoke --function-name createEc2Instance /tmp/output.json

Run the whole thing with this command:

make build && make deploy && make run

At this point, you can go ahead and invoke the lambda function to see if everything is working as expected:

aws lambda invoke --function-name createEc2Instance /tmp/output.json

This can of course be determined by looking at your running EC2 instances and seeing if there's a new one that's spinning up from your invoking the lambda function.

Resources:
https://www.alexedwards.net/blog/serverless-api-with-go-and-aws-lambda#setting-up-the-https-api - doing all the lambda cli stuff and making a lambda function with golang
https://medium.com/appgambit/aws-lambda-launch-ec2-instances-40d32d93fb58 -doing the web UI stuff
https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars.html - setting the env vars programatically
https://www.softkraft.co/aws-lambda-in-golang/ - fantastic in-depth guide for using Go with Lambda

CORS with lambda and API Gateway

Want to do AJAX stuff with your lambda function(s)? Cool, you're in the right place.

Open your gateway
Click Actions -> Enable CORS
Check the boxes for POST, GET, and OPTIONS
Input the following for Access-Control-Allow-Headers:

'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token'

Input the following for Access-Control-Allow-Origin:

'*'

Click Enable CORS and replace existing CORS headers

For Options Method

Open the Method Response and click the arrow next to 200. Add the following headers:

Screen-Shot-2020-10-12-at-3.12.12-PM

For GET Method

Be sure to add the appropriate headers to your APIGatewayProxyResponse:

Headers: map[string]string{
				"Access-Control-Allow-Origin":      "*",
				"Access-Control-Allow-Credentials": "true",
			},

Next, open the Method Response and click the arrow next to 200. Add the following headers:

Screen-Shot-2020-10-12-at-3.11.12-PM

For POST Method

Open the Method Response and click the arrow next to 200. Add the following header:
Screen-Shot-2020-10-12-at-3.12.47-PM

Finishing touches

Finally, be sure to click Actions and Deploy API when you're done

Resource: https://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-cors-console.html

Return Response for API Gateway

You have two options here:

return events.APIGatewayProxyResponse{
			StatusCode: http.StatusBadGateway,
			Headers: map[string]string{
				"Access-Control-Allow-Origin":      "*",
				"Access-Control-Allow-Credentials": "true",
			},
			Body: string("Method not Allowed"),
		}, nil

or alternatively:

resp := events.APIGatewayProxyResponse{Headers: make(map[string]string)}
resp.Headers["Access-Control-Allow-Origin"] = "*"
resp.Headers["Access-Control-Allow-Credentials"] = "true"
resp.StatusCode = http.StatusOK
resp.Body = string(publicInstanceIp)
return resp, nil

Resources:
https://github.com/serverless/examples/blob/master/aws-golang-simple-http-endpoint/hello/main.go - used to figure out the first option

Update function via CLI

This is useful to run after updating your code. This will grab main.zip in the current directory:

env GOOS=linux GOARCH=amd64 go build -o main
zip -j main.zip main
aws lambda update-function-code --function-name <lambda function name> --zip-file fileb:///${PWD}/main.zip

Resource: https://stackoverflow.com/questions/49611739/aws-lambda-update-function-code-with-jar-package-via-aws-cli

Use serverless framework

This framework makes it easier to develop and deploy serverless resources, such as AWS Lambda Functions.

To start we'll need to install the Serverless Framework:

npm install -g serverless

Then we will need to create the project with a boilerplate template. A couple of examples:

# Nodejs
serverless create --template aws-nodejs --path myservice
# Golang
cd $GOPATH/src && serverless create -t aws-go-dep -p myservice

From here, you need to populate the serverless.yml template. This will use the lambda code from above that deploys ec2 instances:

service: lambdainstancedeployer

frameworkVersion: '2'

provider:
  name: aws
  runtime: go1.x
  stage: ${opt:stage, 'dev'}
  region: ${opt:region, 'us-west-2'}
  environment:
    DYNAMO_TABLE: ${self:service}-${opt:stage, self:provider.stage}
  memorySize: 3008
  timeout: 30 # API Gateway max timeout
  iamRoleStatements:
    - Effect: Allow
      Action:
        - dynamodb:Query
        - dynamodb:Scan
        - dynamodb:GetItem
        - dynamodb:PutItem
        - dynamodb:UpdateItem
        - dynamodb:DeleteItem
      Resource: "arn:aws:dynamodb:${opt:region, self:provider.region}:*:table/${self:provider.environment.DYNAMO_TABLE}"
    - Effect: Allow
      Action:
        - ec2:RunInstances
        - ec2:DescribeInstances
        - ec2:DescribeInstanceStatus
        - ec2:TerminateInstances
        - ec2:StopInstances
        - ec2:StartInstances
        - ec2:CreateTags
        - ec2:DeleteTags
      Resource: "*"

package:
  exclude:
    - ./**
  include:
    - ./bin/**

functions:
  lambdaMop:
    handler: bin/lambdaMop
    events:
      - http:
          path: /deployer
          method: post
          cors: true
      - http:
          path: /deployer
          method: get
          cors: true
    environment:
      AMI: ami-xxxxxx
      INSTANCE_TYPE: t2.small
      REGION: us-west-2

resources:
  Resources:
    InstanceDeployerDynamoDbTable:
      Type: 'AWS::DynamoDB::Table'
      # Uncomment if you want to want to ensure the table isn't deleted
      # DeletionPolicy: Retain
      DeletionPolicy: Delete
      Properties:
        AttributeDefinitions:
          - AttributeName: email
            AttributeType: S
        KeySchema:
          - AttributeName: email
            KeyType: HASH
        BillingMode: PAY_PER_REQUEST
        TableName: ${self:provider.environment.DYNAMO_TABLE}

it will also create the API gateway, IAM role and DynamoDB table.

Modify the Makefile if you'd like. The one I like to use can be found right below.

Next, compile the function and build it:

cd myservice && make build

Resources:
https://www.serverless.com/blog/framework-example-golang-lambda-support - Lambda + Golang + Serverless walkthrough
https://marcelog.github.io/articles/aws_lambda_start_stop_ec2_instance.html - useful information for IAM actions needed for ec2 operations
https://forum.serverless.com/t/missing-required-key-tablename-in-params-error/4492/5 - how to set the dynamodb iam permissions
https://forum.serverless.com/t/deleting-table-from-dynamodb/3837 - how to delete a database or retain it

More useful Makefile

Move your functions into a functions folder in the repo for the serverless work.

Next, change the Makefile to the following:

functions := $(shell find functions -name \*main.go | awk -F'/' '{print $$2}')

build: ## Build golang binary
	@for function in $(functions) ; do \
		cd functions/$$function ; \
		env GOOS=linux go build -ldflags="-s -w" -o ../../bin/$$function ; \
		cd .. ; \
	done
	serverless deploy

destroy:
	serverless remove

This will output all function binaries into the bin/ directory at the top level of your project.

Resources:
https://github.com/serverless/examples/blob/master/aws-golang-auth-examples/Makefile - super useful Makefile example

Decode Error Message from CloudWatch Logs

msg="themessage"
aws sts decode-authorization-message --encoded-message $msg --query DecodedMessage --output text | jq '.'

Resource: https://aws.amazon.com/premiumsupport/knowledge-center/aws-backup-encoded-authorization-failure/

Secrets Manager

Create IAM role to grant read access to a secret

If accessing from an ec2 instance, find your ec2 instance in the web UI, right click it -> Security -> Modify IAM Role. Otherwise, just open the IAM console
Click Roles -> Create role
Click EC2
Click Next: Permissions
Click Create policy
Click JSON
Copy the json from here:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetResourcePolicy",
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret",
                "secretsmanager:ListSecretVersionIds"
            ],
            "Resource": "<your secret ARN>"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "secretsmanager:ListSecrets",
            "Resource": "*"
        }
    ]
}

Change <your secret ARN> to the proper value of your secret, which you can find in the Secrets Manager UI and click Review policy
Specify a Name for the policy and click Create policy

Resource: https://docs.aws.amazon.com/mediaconnect/latest/ug/iam-policy-examples-asm-secrets.html

Get secret from secrets manager and output to file

aws secretsmanager get-secret-value --secret-id <your secrets id> --query SecretString --output text | tee <file to output secret to>

Resource: https://stackoverflow.com/questions/50911540/parsing-secrets-from-aws-secrets-manager-using-aws-cli

Get several secrets

users=(user1 user2 user3)
environment='prod'
for user in "${users[@]}"; do
  sec=$(aws secretsmanager get-secret-value --secret-id $environment-$user \
    --query SecretString \
    --output text)
  echo "Secret for $environment-$user is $sec"
done

Create new secret from a file

aws secretsmanager create-secret --name MyTestDatabaseSecret \
    --description "My test database secret created with the CLI" \
    --secret-string file://mycreds.json
    --output text

Resource: https://docs.aws.amazon.com/cli/latest/reference/secretsmanager/create-secret.html

Add access key and secret access key as secrets

aws secretsmanager create-secret --name "prod/someuser_aws_access_key_id" \
    --description "someuser prod aws_access_key_id" \
    --secret-string "$(sed '2q;d' ~/.aws/credentials | awk '{print $3}')" \
    --output text
    
aws secretsmanager create-secret --name "prod/someuser_aws_secret_access_key" \
    --description "someuser prod aws_secret_access_key" \
    --secret-string "$(sed '3q;d' ~/.aws/credentials | awk '{print $3}')" \
    --output text

List secrets

aws secretsmanager list-secrets --output text

Update secret from a file

aws secretsmanager update-secret --secret-id <name of secret or ARN> \
--description "<what the secret is>" \
--secret-string "file://somesecret" \
--output text

Delete secret without waiting period

aws secretsmanager delete-secret --secret-id <name of secret or ARN> --force-delete-without-recovery

Resources:
https://docs.aws.amazon.com/cli/latest/reference/secretsmanager/delete-secret.html - gave me the information to piece together the command
https://stackoverflow.com/questions/55931838/aws-secret-manager-secret-deleted-without-mandatory-retention-period - gave me idea for the solution

One liner for ssh secret

If you have an SSH key in Secrets Manager, you can run the following to grab it and put it into a file on your local system:

aws secretsmanager get-secret-value --secret-id ssh_key | jq '.SecretString' | sed 's/\\n/\n/g' | sed 's/"//g' | tee ~/.ssh/ssh_key && chmod 400 ~/.ssh/ssh_key

Resource: https://stackoverflow.com/questions/32715008/jq-how-to-print-newline-and-not-newline-character-from-json-value - clean up the JSON so the file works

CloudTrail

Find when an ec2 instance was terminated

This will require you to have the instance id of the terminated instance and a rough sense of the day that it was terminated.

Open the CloudTrail service
Click Event history
Select Event name from the dropdown
Input TerminateInstances
Search for the terminated instance id under the Resource name column

Resource: https://aws.amazon.com/premiumsupport/knowledge-center/cloudtrail-search-api-calls/

IAM

Create user

NAME='kops'
aws iam create-user --user-name $NAME

Delete user

NAME='kops'
aws iam delete-user --user-name $NAME

Create access keys for a user

NAME='kops'
aws iam create-access-key --user-name $NAME

Resource: https://github.com/kubernetes/kops/blob/master/docs/getting_started/aws.md#setup-iam-user

List policies

aws iam list-policies

Delete policy

aws iam delete-policy --policy-arn $ARN

Delete policies with word terraform in them

aws iam list-policies |grep terraform |grep arn | awk '{print $2}' | tr -d '"' | tr -d ',' | xargs -I{} aws iam delete-policy --policy-arn {}

List instance profiles

aws iam list-instance-profiles

Delete instance profile

aws iam delete-instance-profile --instance-profile-name $PROFILE_NAME

Attach IAM instance profile to ec2 instance

Open the Amazon EC2 console
Click Instances
Click the instance you want to access the s3 bucket from
Click Actions in the upper right-hand side of the screen
CLick Security -> Modify IAM role
Enter the name of the IAM role created previously
Click Save

To download files from the S3 bucket, follow the steps at the top of the page under INSTALL LATEST VERSION OF AWS CLI ON LINUX to get the AWS cli utils in order to grab stuff from the bucket.

Resources:
https://aws.amazon.com/premiumsupport/knowledge-center/ec2-instance-access-s3-bucket/ - Set up IAM and attach it to ec2 instance
https://stackoverflow.com/questions/6615168/is-there-an-s3-policy-for-limiting-access-to-only-see-access-one-bucket - IAM policy used

Cloud-init

Validate cloud-init

cloud-init devel schema --config-file bob.yaml

Resource: https://stackoverflow.com/questions/54427198/cloud-init-validator

Delete cloud-init logs

cloud-init clean --logs

Log locations for cloud-init

/var/log/cloud-init.log
/var/log/cloud-init-output.log
/run/cloud-init
/var/lib/cloud/instance/user-data.txt

cloud-init package version
dmesg output
journalctl output

Resource: https://cloudinit.readthedocs.io/en/latest/topics/cli.html

Location of userdata script

View what's provided as userdata by running this command:

cat /var/lib/cloud/instance/cloud-config.txt

Resource: https://stackoverflow.com/questions/59505762/what-is-the-linux-file-location-where-the-cloud-init-user-data-is-passed-to

DynamoDB

List Tables

aws dynamodb list-tables

Resource: https://docs.aws.amazon.com/cli/latest/reference/dynamodb/list-tables.html

Delete Table

TABLE=yourtable
aws dynamodb delete-table --table-name $TABLE

AWS Cheatsheet

Install latest version of AWS CLI on linux

UI

Backup instance manually

Parameter Store location

EC2

Use env vars

Set up credentials file

Populate env vars with credentials file

Populate config file with env vars

Multiple profiles

Use temp credentials

Use env vars for temp credentials

List instances

Get number of instances

Get running instances

Reboot all instances in a region

Assign an elastic IP to an instance

Create instance with a tag

Create instance using security group name

Get security group id from group name

Add new ingress rule to security group

List instances with filtering

List instance by instance id

Destroy instance

Get info about a specific AMI by name and output to JSON

Get AMI id from AMI name

Get AMI id with some python

Deregister an AMI

Get list of all instances with the state terminated

List all instances that match a tag name and are running

Get info about an AMI by product-code

Resize ec2 instance

Show available subnets

Attach volume at root

List snapshots

Add custom tag for cloud-init finishing

CodeCommit

CodeBuild

Build the proper IAM role

Cloning into a repo

Integrating this in with CodeBuild

Validate buildspec

Debug Codebuild

SSM

List parameters

Access a parameter

Miscellaneous

Encrypt your pem file:

Remove the encryption:

Set up aws cli with pipenv on OSX

S3

Create bucket

List buckets

List files in a bucket

Download bucket

Copy file down

Copy file up

Copy folder up

Read buckets into an array

Iterate over buckets

Empty bucket

Delete bucket

Copy multiple folders to bucket

Set up S3 IAM for backup/restore

Create Policy

Create a Role

Assign the role to your instance

Set up automated expiration of objects

Mount bucket as local directory

Create IAM role to grant read access to an s3 bucket

Get id of KMS key associated with a bucket

Boto

Create session

List buckets with boto3

Show items in an s3 bucket

List Users

Get account id

Create ec2 instance with name

Allocate and associate an elastic IP