Auto Scale Fargate Service Based On SQS ApproximateNumberOfMessagesVisible


Answer :

Yes you can do this. You have to use a step scaling policy, and you need to have an alarm created already for your SQS queue depth (ApproximateNumberOfMessagesVisible).

Go to CloudWatch, create a new alarm. We'll call this alarm sqs-queue-depth-high, and have it trigger when the approximate number of messages visible is 1000.

With that done, go to ECS to the service you want to autoscale. Click Update for the service. Add a scaling policy and choose the Step Tracking variety. You'll see there's an option to create a new alarm (which only lets you choose between CPU or MemoryUtilization), or use an existing alarm.

Type sqs-queue-depth-high in the "Use existing alarm" field and press enter, you should see a green checkmark that lets you know the name is valid (i.e. the alarm exists). You'll see new dropdowns where you can adjust the step policy now.

This works for any metric alarm and ECS services. If you're going to be trying to scale this setup out, for multiple environments for example, or making it any more sophisticated than 2 steps, do yourself a favor and jump in with CloudFormation or Terraform to help manage it. Nothing is worse than having to adjust a 5-step alarm across 10 services.


AWS provides a solution for scaling based on SQS queue: https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-using-sqs-queue.html

Main idea

  1. Create a CloudWatch Custom Metric sqs-backlog-per-task using formula: sqs-backlog-per-task = sqs-messages-number / running-task-number.
  2. Create a Target Tracking Scaling Policy based on the backlogPerInstance metric.

Implementation details

Custom Metric

In my case all the infrastructure (Fargate, SQS, and other resources) is described in CloudFormation stack. So for calculating and logging the custom metric I decided to use AWS Lambda function which is also described in CloudFormation stack and deployed together with the entire infrastructure.

Below you can find code snippets for the AWS Lambda function for logging the following custom metrics:

  • sqs-backlog-per-task - used for scaling
  • running-task-number - used for scaling optimization and debugging

AWS Lambda function described in AWS SAM syntax in CloudFormation stack (infrastructure.yml):

CustomMetricLoggerFunction:     Type: AWS::Serverless::Function     Properties:       FunctionName: custom-metric-logger       Handler: custom-metric-logger.handler       Runtime: nodejs8.10       MemorySize: 128       Timeout: 3       Role: !GetAtt CustomMetricLoggerFunctionRole.Arn       Environment:         Variables:           ECS_CLUSTER_NAME: !Ref Cluster           ECS_SERVICE_NAME: !GetAtt Service.Name           SQS_URL: !Ref Queue       Events:         Schedule:           Type: Schedule           Properties:             Schedule: 'cron(0/1 * * * ? *)' # every one minute

AWS Lambda Javascript code for calculating and logging (custom-metric-logger.js):

var AWS = require('aws-sdk');  exports.handler = async () => {   try {     var sqsMessagesNumber = await getSqsMessagesNumber();     var runningContainersNumber = await getRunningContainersNumber();      var backlogPerInstance = sqsMessagesNumber;     if (runningContainersNumber > 0) {       backlogPerInstance = parseInt(sqsMessagesNumber / runningContainersNumber);     }      await putRunningTaskNumberMetricData(runningContainersNumber);     await putSqsBacklogPerTaskMetricData(backlogPerInstance);      return {       statusCode: 200     };   } catch (err) {     console.log(err);      return {       statusCode: 500     };   } };  function getSqsMessagesNumber() {   return new Promise((resolve, reject) => {     var data = {       QueueUrl: process.env.SQS_URL,       AttributeNames: ['ApproximateNumberOfMessages']     };      var sqs = new AWS.SQS();     sqs.getQueueAttributes(data, (err, data) => {       if (err) {         reject(err);       } else {         resolve(parseInt(data.Attributes.ApproximateNumberOfMessages));       }     });   }); }  function getRunningContainersNumber() {   return new Promise((resolve, reject) => {     var data = {       services: [         process.env.ECS_SERVICE_NAME       ],       cluster: process.env.ECS_CLUSTER_NAME     };      var ecs = new AWS.ECS();     ecs.describeServices(data, (err, data) => {       if (err) {         reject(err);       } else {         resolve(data.services[0].runningCount);       }     });   }); }  function putRunningTaskNumberMetricData(value) {   return new Promise((resolve, reject) => {     var data = {       MetricData: [{         MetricName: 'running-task-number',         Value: value,         Unit: 'Count',         Timestamp: new Date()       }],       Namespace: 'fargate-sqs-service'     };      var cloudwatch = new AWS.CloudWatch();     cloudwatch.putMetricData(data, (err, data) => {       if (err) {         reject(err);       } else {         resolve(data);       }     });   }); }  function putSqsBacklogPerTaskMetricData(value) {   return new Promise((resolve, reject) => {     var data = {       MetricData: [{         MetricName: 'sqs-backlog-per-task',         Value: value,         Unit: 'Count',         Timestamp: new Date()       }],       Namespace: 'fargate-sqs-service'     };      var cloudwatch = new AWS.CloudWatch();     cloudwatch.putMetricData(data, (err, data) => {       if (err) {         reject(err);       } else {         resolve(data);       }     });   }); }

Target Tracking Scaling Policy

Then based on the sqs-backlog-per-task metric I created Target Tracking Scaling Policy in my Cloud Formation template.

Target Tracking Scaling Policy based on the sqs-backlog-per-task metric (infrastructure.yml):

ServiceScalingPolicy:     Type: AWS::ApplicationAutoScaling::ScalingPolicy     Properties:       PolicyName: service-scaling-policy       PolicyType: TargetTrackingScaling       ScalingTargetId: !Ref ServiceScalableTarget       TargetTrackingScalingPolicyConfiguration:         ScaleInCooldown: 60         ScaleOutCooldown: 60         CustomizedMetricSpecification:           Namespace: fargate-sqs-service           MetricName: sqs-backlog-per-task           Statistic: Average           Unit: Count         TargetValue: 2000

As a result AWS Application Auto Scaling creates and manages the CloudWatch alarms that trigger the scaling policy and calculates the scaling adjustment based on the metric and the target value. The scaling policy adds or removes capacity as required to keep the metric at, or close to, the specified target value. In addition to keeping the metric close to the target value, a target tracking scaling policy also adjusts to changes in the metric due to a changing load pattern.


I wrote a blog article about exactly this topic including a docker container to run it. The article can be found at: https://allaboutaws.com/how-to-auto-scale-aws-ecs-containers-sqs-queue-metrics

The prebuild container is available at DockerHub: https://hub.docker.com/r/sh39sxn/ecs-autoscaling-sqs-metrics

The files are available at GitHub: https://github.com/sh39sxn/ecs-autoscaling-sqs-metrics

I hope it helps you.


Comments

Post a Comment

Popular posts from this blog

Chemistry - Bond Angles In NH3 And NCl3

Answer : Solution 1: As Tan Yong Boon stated, it is Bent’s Rule with which we can explain the lower bond angle of \ce N F 3 \ce{NF3} \ce NF 3 when compared to \ce N H 3 \ce{NH3} \ce N H 3 . The rule as stated by Henry Bent: Atomic s character concentrates in orbitals directed towards electropositive substituents. Fluorine is more electronegative that hydrogen, and the \ce N − F \ce{N−F} \ce N − F bond would have greater p character than the \ce N − H \ce{N−H} \ce N − H bond. And more s character leads to large bond angles. Thus, the bond angle is greater in \ce N H 3 \ce{NH3} \ce N H 3 than in \ce N F 3 \ce{NF3} \ce NF 3 . Now, consider \ce N C l 3 \ce{NCl3} \ce NCl 3 . Clearly, \ce C l \ce{Cl} \ce Cl atom islarger in size than the central atom, nitrogen. Hence the higher bond angle here is due to the steric crowding caused by \ce C l \ce{Cl} \ce Cl atoms.(More pronounced than the electrongativity of \ce C l \ce{Cl} \ce Cl atom). They repel eachother and hence b...
Read more

Are Regular VACUUM ANALYZE Still Recommended Under 9.1?

Answer : VACUUM is only needed on updated or deleted rows in non-temporary tables. Obviously you're doing lots of INSERTs but it's not obvious from the description that you're also doing lots of UPDATEs or DELETEs. These operations can be tracked with the pg_stat_all_tables view, specifically the n_tup_upd and n_tup_del columns. Also, even more to the point, there is a n_dead_tup column that tells, per table, how much rows need to be vacuumed. (see Monitoring statistics in the doc for functions and views related to statistics gathering). A possible strategy in your case would be to suppress the scheduled VACUUM, keeping an eye on this view and checking on which tables the n_dead_tup is going up significantly. Then apply the aggressive VACUUM to these tables only. This will be a win if there are large tables whose rows never get deleted nor updated and the aggressive VACUUM is really necessary only on smaller tables. But keep running the ANALYZE for the optimiz...
Read more

Change The Font Size Of Visual Studio Solution Explorer

Answer : Go to Tools->Options->Environment->Fonts and Colors In "Show Settings for" chose "Environment Font" In "Font" replace Automatic to for example, Arial and change size. Change the font size of that setting, all Font size in the Explorers (Solution Explorer, Server Explorer and Toolbox) and menus will change. You can set that in Tools, Options.
Read more