Check CPU/Volume credits on AWS T micro/nano/small instances

T instances and Gentoo

AWS t2/t3/t3a/t4g nano/micro/small instances is not the best choice for compilation of heavy applications, especially on Gentoo where every package is compiled from source code. However, these small T instances are very cost-effective and could be used as playground or for small projects.

Before upgrading any heavy package like gcc, glibc, kernel it is good to check if your instance has enough CPU credits to compile package without falling back to baseline performance.

You could read more about burstable credit balance concept here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-credits-baseline-concepts.html

These days you can enable unlimited CPU credits for T instances to avoid performance drop: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-performance-instances-unlimited-mode.html

Also, if T instance is low on memory (like 512MB) then it could be complicated to build heavy packages like gcc, glibc, binutils etc. If you are experimenting with Gentoo on cheapest T instances, then I would recommend to limit compilation to 1 thread, ensure that you have swap, ensure that kernel avoids using swap where possible, and ensure that swap did not exceed EBS volume IOPS burst balance. Keep in mind, that compilation is causing a lot of IO and your EBS volume can run out of IOPS burst balance and fallback to very slow IO causing compilations to take hours.

Recommended configuration for T nano/micro/small instances

Set minimal usage of swap to avoid spending volume IOPS burst:

# set in runtime
sysctl vm.swappiness=1

# persist on reboot
nano /etc/sysctl.conf 
vm.swappiness=1

Setup swap on the same disk or separate volume:

# create swap file
fallocate -l 1G /swap
chmod 600 /swap

# format swap file
mkswap /swap
swaplabel -L swap /swap

# enable swap in runtime
swapon /swap

# persist on reboot
nano /etc/fstab
LABEL="swap"            none            swap            sw              0 0

# check swap status
swapon --show

Limit number of parallel compilation processes. Each process will eat RAM that is limited on nano/micro instances. Set MAKEOPTS="-j1" in /etc/portage/make.conf.

Run portage with lower CPU priority to make system more responsive when something is merging. Set PORTAGE_NICENESS="-20" in /etc/portage/make.conf.

Script to check cpu/volume credits

The script below could be used on t instances to check cpu credit balance (if unlimited balance is not enabled) and also to test volume burst balance. Can be useful to check before running intensive CPU or/and IOPS compilation. Feel free to modify to your use case.

File check-ec2-credits:

#!/bin/bash

CPU_CREDITS_WARN=50 # in percents

# https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-credits-baseline-concepts.html
instance_type_max_cpu_credits_t2_nano=72
instance_type_max_cpu_credits_t2_micro=144
instance_type_max_cpu_credits_t2_small=288
instance_type_max_cpu_credits_t2_medium=576
instance_type_max_cpu_credits_t2_large=864
instance_type_max_cpu_credits_t2_xlarge=1296
instance_type_max_cpu_credits_t2_2xlarge=1944
instance_type_max_cpu_credits_t3_nano=144
instance_type_max_cpu_credits_t3_micro=288
instance_type_max_cpu_credits_t3_small=576
instance_type_max_cpu_credits_t3_medium=576
instance_type_max_cpu_credits_t3_large=864
instance_type_max_cpu_credits_t3_xlarge=2304
instance_type_max_cpu_credits_t3_2xlarge=4608
instance_type_max_cpu_credits_t3a_nano=144
instance_type_max_cpu_credits_t3a_micro=288
instance_type_max_cpu_credits_t3a_small=576
instance_type_max_cpu_credits_t3a_medium=576
instance_type_max_cpu_credits_t3a_large=864
instance_type_max_cpu_credits_t3a_xlarge=2304
instance_type_max_cpu_credits_t3a_2xlarge=4608
instance_type_max_cpu_credits_t4g_nano=144
instance_type_max_cpu_credits_t4g_micro=288
instance_type_max_cpu_credits_t4g_small=576
instance_type_max_cpu_credits_t4g_medium=576
instance_type_max_cpu_credits_t4g_large=864
instance_type_max_cpu_credits_t4g_xlarge=2304
instance_type_max_cpu_credits_t4g_2xlarge=4608

metadata="$(curl -sL http://169.254.169.254/latest/dynamic/instance-identity/document)" 
region=$(echo "$metadata" | jq -r .region)
instance_id=$(echo "$metadata" | jq -r .instanceId)
instance_type=$(echo "$metadata" | jq -r .instanceType)

# metrics could delay, so check for lat 10 minutes
start_time=$(date -u -d "10 minutes ago" +%Y-%m-%dT%H:%M:%S)
end_time=$(date -u +%Y-%m-%dT%H:%M:%S)

cpu_credits_value=$(
   aws cloudwatch get-metric-statistics \
        --namespace "AWS/EC2" \
        --region "${region}" \
        --metric-name "CPUCreditBalance" \
        --start-time "${start_time}" \
        --end-time "${end_time}" \
        --period 60 \
        --statistics Average \
        --dimensions Name=InstanceId,Value="${instance_id}" \
        --query 'Datapoints | reverse(sort_by(@, &Timestamp))[0].Average' \
        --output text
)

cpu_credits_max_var=instance_type_max_cpu_credits_${instance_type/./_}
cpu_credits_max=${!cpu_credits_max_var}
cpu_credits_ratio=$(echo "$cpu_credits_value / $cpu_credits_max * 100" | bc -l)
cpu_credits_warn=$(echo "$cpu_credits_ratio <= $CPU_CREDITS_WARN" | bc -l)

echo "Instance ID: $instance_id"
echo "Instance Type: $instance_type"
echo "Region ID: $region"
echo "CPU Credits: $(printf %.0f $cpu_credits_value)/$(printf %.0f $cpu_credits_max) -> $(printf %.0f $cpu_credits_ratio)%"

instance_volumes=$(
    aws ec2 describe-instances \
        --instance-ids i-016138d31fdd842aa \
        --query 'Reservations[].Instances[].BlockDeviceMappings[].Ebs.VolumeId' \
        --output text
)

for volume_id in $instance_volumes; do
    volume_burst_balance=$(
        aws cloudwatch get-metric-statistics \
            --namespace "AWS/EBS" \
            --region "${region}" \
            --metric-name "BurstBalance" \
            --start-time "${start_time}" \
            --end-time "${end_time}" \
            --period 60 \
            --statistics Average \
            --dimensions Name=VolumeId,Value="${volume_id}" \
            --query 'Datapoints | reverse(sort_by(@, &Timestamp))[0].Average'
    )

    echo "Volume burst balance: $volume_id -> $(echo "$volume_burst_balance / 1" | bc)%" 
done

exit $cpu_credits_warn

Install dependencies

Please note, this script depends on bash, jq and aws:

• bash is preinstalled
• jq could be installed with emerge jq
• aws could be installed with emerge awscli

Configure AWS credentials

aws requires configuration on instance where you will use it. You could follow official article: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-quick-configuration

You will also need to grant some permission to be able to read metadata.

Your policy could look like (where A.B.C.D is ip address of your instance):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
               "cloudwatch:GetMetricStatistics",
               "ec2:DescribeInstances"
            ],
            "Resource": "*",
            "Condition": {
                "IpAddress": {
                    "aws:SourceIp": "A.B.C.D/32"
                }
            }
        }
    ]
}

The policy behind is a primitive example. The better approach is to not hardcode any credentials on instance, and attach IAM policy to instance instead, plus limit ec2:DescribeInstances to be runnable only for specific instance.

Checklist:

• log into aws console
• create user
• remember Access Key ID and Secret Access Key
• add inline policy
• log into instance and configure aws

Testing

Run script and you should be able to see something like below:

# check-ec2-credits 
Instance ID: i-01234567890123456
Instance Type: t2.nano
Region ID: us-east-1
CPU Credits: 25/72 -> 34%
Volume burst balance: vol-00112233445566778 -> 89%