ECS

Q:How often are docker images pulled by the ECS Agent?

As part of the AWS Batch creation process, Tower Forge will set ECS Agent parameters in the EC2 Launch Template that is created for your cluster's EC2 instances.

• For clients using Tower Enterprise v22.01 or later:
  • Any AWS Batch environment created by Tower Forge will set the ECS Agent's ECS_IMAGE_PULL_BEHAVIOUR set to once.
• For clients using Tower Enterprise v21.12 or earlier:
  • Any AWS Batch environment created by Tower Forge will set the ECS Agent's ECS_IMAGE_PULL_BEHAVIOUR set to default.

Please see the AWS ECS documentation for an in-depth explanation of this difference.

Note:This behaviour cannot be changed within the Tower Application.

Q: We encountered an error saying unable to parse HTTP 429 response body.

CannotPullContainerError: Error response from daemon: error parsing HTTP 429 response body: invalid character 'T' looking for beginning of value: "Too Many Requests (HAP429)"

This is because of the dockerhub rate limit of 100 anonymous pulls per 6 hours. We suggest to use the following on your launch template in order to avoid this issue:

echo ECS_IMAGE_PULL_BEHAVIOR=once >> /etc/ecs/ecs.config

Q: Help! My job failed due to a CannotInspectContainerError error.

There are multiple reasons why your pipeline could fail with an Essential container in task exited - CannotInspectContainerError: Could not transition to inspecting; timed out after waiting 30s error.

Please try the following:

1. Upgrade your ECS Agent to 1.54.1 or newer (instructions for checking your ECS Agent version);
2. Provision more storage space for your EC2 instance (preferrably via ebs-autoscaling to ensure scalability).
3. If the error is accompanied by command exit status: 123 and a permissions denied error tied to a system command, please ensure that the binary is set to be executable (i.e. chmod u+x).