› ›

AWS Fargate Integration (for ECS clusters)

Version	1.3.0 (View all)
Compatible Kibana version(s)	8.13.0 or higher 9.0.0 or higher
Supported Serverless project types What’s this?	Security Observability
Subscription level What’s this?	Basic
Level of support What’s this?	Elastic

Overview

edit

The AWS Fargate integration helps to retrieve metadata, network metrics, and Docker stats about your containers and the tasks that are part of an Amazon Elastic Container Service (Amazon ECS) cluster.

The AWS Fargate integration currently supports ECS clusters only. It does not support EKS clusters.

Credentials

edit

This integration does not require AWS credentials. The ECS task metadata endpoint is accessible only inside the cluster.

Setup

edit

To start collecting AWS Fargate metrics, you must run the Elastic Agent as a sidecar container alongside your application container in the same task definition.

Each task definition must run an Agent because task metadata information is only available to containers running in the task.

Here’s an example of an Elastic Agent running as a sidecar with an application container:

TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Family: !Ref TaskName
      Cpu: 256
      Memory: 512
      NetworkMode: awsvpc
      ExecutionRoleArn: !Ref ExecutionRole
      ContainerDefinitions:
        - Name: <application-container>              << ===== Application container
          Image: <application-container-image>
          <application-container-settings>
        - Name: elastic-agent-container              << ===== Elastic Agent container
          Image: docker.elastic.co/beats/elastic-agent:8.12.0

The Elastic Agent collects metrics using the Amazon ECS task metadata endpoint.

The Amazon ECS task metadata endpoint is an HTTP endpoint available to each container and enabled by default on AWS Fargate platform version 1.4.0 and later. The Elastic Agent uses Task metadata endpoint version 4.

Getting started using the AWS Management Console

edit

This section shows you how to run the Elastic Agent in a ECS cluster, start collecting Fargate on ECS metrics, and send them to an Elastic Stack.

To quickly deploy on your existing ECS cluster, follow these steps.

Task Definition

edit

Open the AWS Management Console and visit the Amazon ECS page. Here you can select "Task Definitions" and then "Create new Task Definition" to start the wizard.

Step 1:

Select "Fargate" from the list of available launch types.

Step 2:

Add your preferred name for the "Task definition name", for example "elastic-agent-fargate-deployment".
For the "Task role", select "ecsFargateTaskExecutionRole".
For the "Operating system family", select "Linux".
Pick a value for "Task memory (GB)" and "Task CPU (vCPU)"; the lowest values are fine for testing purposes.
Click on "Add container".

As for the container, you can use the following values:

Container name: elastic-agent-container
Image: docker.elastic.co/beats/elastic-agent:8.12.0
Environment variables:
- FLEET_ENROLL: yes
- FLEET_ENROLLMENT_TOKEN: <enrollment-token>
- FLEET_URL: <fleet-server-url>

use the AWS Secrets Manager to store the Fleet Server enrollment token.

Service

edit

Select an existing ECS cluster and create a new service with launch type "FARGATE". Use the task definition we just created.

As soon as the Elastic Agent is started, open the dashboard "[AWS Fargate] Fargate Overview" and you will see the metrics show up in few minutes.

Getting started using the AWS CLI

edit

In this example, we will use the AWS CLI and a CloudFormation template to set up the following resources:

an ECS cluster,
a task definition for the Elastic Agent,
a service to execute the agent task on the cluster.

Setup

edit

Prepare you terminal and AWS environment to create the ECS cluster for the testing.

Pick a region

edit

Set default AWS region for this session:

export AWS_DEFAULT_REGION="us-east-1"

Secrets management

edit

Store the enrollment token and the Fleet Server URL in the AWS Secrets Manager:

aws secretsmanager create-secret \
    --name FLEET_ENROLLMENT_TOKEN \
    --secret-string <your-fleet-enrollment-token-goes-here>

aws secretsmanager create-secret \
    --name FLEET_URL \
    --secret-string <your-fleet-url>

Take note of the Amazon Resource Name (ARN) of both secrets, we’ll use them in a moment.

if you need to update them during your tests, use the following put-secret-value to do it:

aws secretsmanager put-secret-value \
    --secret-id FLEET_ENROLLMENT_TOKEN \
    --secret-string <fleet-enrollment-token>

Networking

edit

One more thing. You need to pick one subnet where your ECS cluster will be created in. Take note of the subnet ID for the very next step.

Deploy the stack

edit

Copy the following CloudFormation template and save it on you computer with the name cloudformation.yml:

AWSTemplateFormatVersion: "2010-09-09"
Parameters:
  SubnetID:
    Type: String
    Description: Enter the ID of the subnet you want to create the cluster in.
  FleetEnrollmentTokenSecretArn:
    Type: String
    Description: Enter the Amazon Resource Name (ARN) of the secret holding the enrollment token for the Elastic Agent.
  FleetUrlSecretArn:
    Type: String
    Description: Enter the Amazon Resource Name (ARN) of the secret holding the Fleet Server URL.
  ClusterName:
    Type: String
    Default: elastic-agent-fargate
    Description: Enter the name of the Fargate cluster to create.
  RoleName:
    Type: String
    Default: ecsFargateTaskExecutionRole
    Description: Enter the Amazon Resource Name (ARN) of the task execution role that grants the Amazon ECS container agent permission to make AWS API calls on your behalf.
  TaskName:
    Type: String
    Default: elastic-agent-fargate-task
    Description: Enter the name of the task definition to create.
  ServiceName:
    Type: String
    Default: elastic-agent-fargate-service
    Description: Enter the name of the service to create.
  LogGroupName:
    Type: String
    Default: elastic-agent-fargate-log-group
    Description: Enter the name of the log group to create.
Resources:
  Cluster:
    Type: AWS::ECS::Cluster
    Properties:
      ClusterName: !Ref ClusterName
      ClusterSettings:
        - Name: containerInsights
          Value: disabled
  LogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Ref LogGroupName
  ExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Ref RoleName
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: ecs-tasks.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
      Policies:
        - PolicyName: !Sub 'EcsTaskExecutionRole-${AWS::StackName}'
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - secretsmanager:GetSecretValue
                Resource:
                  - !Ref FleetEnrollmentTokenSecretArn
                  - !Ref FleetUrlSecretArn
  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Family: !Ref TaskName
      Cpu: 256
      Memory: 512
      NetworkMode: awsvpc
      ExecutionRoleArn: !Ref ExecutionRole
      ContainerDefinitions:
        - Name: elastic-agent-container
          Image: docker.elastic.co/beats/elastic-agent:8.12.0
          Secrets:
            - Name: FLEET_ENROLLMENT_TOKEN
              ValueFrom: !Ref FleetEnrollmentTokenSecretArn
            - Name: FLEET_URL
              ValueFrom: !Ref FleetUrlSecretArn
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-region: !Ref AWS::Region
              awslogs-group: !Ref LogGroup
              awslogs-stream-prefix: ecs
          Environment:
            - Name: FLEET_ENROLL
              Value: true
              # You migh need to set FLEET_INSECURE to true
              # if you're connecting to a development
              # environment. Use it responsibly.
              # - Name: FLEET_INSECURE
              #   Value: true
      RequiresCompatibilities:
        - EC2
        - FARGATE
  Service:
    Type: AWS::ECS::Service
    Properties:
      ServiceName: !Ref ServiceName
      Cluster: !Ref Cluster
      TaskDefinition: !Ref TaskDefinition
      DesiredCount: 1
      LaunchType: FARGATE
      NetworkConfiguration:
        AwsvpcConfiguration:
          AssignPublicIp: ENABLED
          Subnets:
            - !Ref SubnetID

We are now finally ready to deploy the ECS cluster with the Elastic Agent running in its own task.

aws cloudformation create-stack \
    --stack-name elastic-agent-fargate-deployment \
    --template-body file://./cloudformation.yml \
    --capabilities CAPABILITY_NAMED_IAM \
    --parameters \
        ParameterKey=SubnetID,ParameterValue=<subnet-id> \
        ParameterKey=FleetEnrollmentTokenSecretArn,ParameterValue=arn:aws:secretsmanager:eu-west-1:000123456789:secret:FLEET_ENROLLMENT_TOKEN-ZxsJGw \
        ParameterKey=FleetUrlSecretArn,ParameterValue=arn:aws:secretsmanager:eu-west-1:000123456789:secret:FLEET_URL-mvjF3a \
        ParameterKey=ClusterName,ParameterValue=elastic-agent-fargate \
        ParameterKey=RoleName,ParameterValue=ecsFargateTaskExecutionRole \
        ParameterKey=TaskName,ParameterValue=elastic-agent-fargate-task \
        ParameterKey=ServiceName,ParameterValue=elastic-agent-fargate-service \
        ParameterKey=LogGroupName,ParameterValue=elastic-agent-fargate-log-group

The AWS CLI will return a StackId:

{
    "StackId": "arn:aws:cloudformation:eu-west-1:000123456789:stack/elastic-agent-deployment/fc324160-b0f9-11ec-9c45-0643aa7239c3"
}

Check the stack status until it has reached the CREATE_COMPLETE status. Use the AWS Management Console or the AWS CLI (requires the tool jq):

$ aws cloudformation list-stacks | jq '.StackSummaries[] | .StackName + " " + .StackStatus'

"elastic-agent-fargate-deployment CREATE_COMPLETE"

That’s it!

Clean up

edit

Once you’re done with experimenting, you can remove all the resources (ECS cluster, task, service, etc) with the following command:

aws cloudformation delete-stack --stack-name elastic-agent-fargate-deployment

Metrics

edit

Task Stats

edit

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields

Field	Description	Type	Metric Type
@timestamp	Event timestamp.	date
agent.id	Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id.	keyword
awsfargate.task_stats.cluster_name	Cluster name	keyword
awsfargate.task_stats.cpu.core.*.norm.pct	Percentage of time per CPU core normalized by the number of CPU cores.	scaled_float	gauge
awsfargate.task_stats.cpu.core.*.pct	Percentage of time per CPU core.	scaled_float	gauge
awsfargate.task_stats.cpu.core.*.ticks	CPU ticks per CPU core.	long	counter
awsfargate.task_stats.cpu.kernel.norm.pct	Percentage of time in kernel space normalized by the number of CPU cores.	scaled_float	gauge
awsfargate.task_stats.cpu.kernel.pct	Percentage of time in kernel space.	scaled_float	gauge
awsfargate.task_stats.cpu.kernel.ticks	CPU ticks in kernel space.	long	counter
awsfargate.task_stats.cpu.system.norm.pct	Percentage of total CPU time in the system normalized by the number of CPU cores.	scaled_float	gauge
awsfargate.task_stats.cpu.system.pct	Percentage of total CPU time in the system.	scaled_float	gauge
awsfargate.task_stats.cpu.system.ticks	CPU system ticks.	long	counter
awsfargate.task_stats.cpu.total.norm.pct	Total CPU usage normalized by the number of CPU cores.	scaled_float	gauge
awsfargate.task_stats.cpu.total.pct	Total CPU usage.	scaled_float	gauge
awsfargate.task_stats.cpu.user.norm.pct	Percentage of time in user space normalized by the number of CPU cores.	scaled_float	gauge
awsfargate.task_stats.cpu.user.pct	Percentage of time in user space.	scaled_float	gauge
awsfargate.task_stats.cpu.user.ticks	CPU ticks in user space.	long	counter
awsfargate.task_stats.diskio.read.bytes	Bytes read during the life of the container	long	counter
awsfargate.task_stats.diskio.read.ops	Number of reads during the life of the container	long	counter
awsfargate.task_stats.diskio.read.queued	Total number of queued requests	long	counter
awsfargate.task_stats.diskio.read.rate	Number of current reads per second	long	gauge
awsfargate.task_stats.diskio.read.service_time	Total time to service IO requests, in nanoseconds	long	counter
awsfargate.task_stats.diskio.read.wait_time	Total time requests spent waiting in queues for service, in nanoseconds	long	counter
awsfargate.task_stats.diskio.reads	Number of current reads per second	scaled_float	gauge
awsfargate.task_stats.diskio.summary.bytes	Bytes read and written during the life of the container	long	counter
awsfargate.task_stats.diskio.summary.ops	Number of I/O operations during the life of the container	long	counter
awsfargate.task_stats.diskio.summary.queued	Total number of queued requests	long	counter
awsfargate.task_stats.diskio.summary.rate	Number of current operations per second	long	gauge
awsfargate.task_stats.diskio.summary.service_time	Total time to service IO requests, in nanoseconds	long	counter
awsfargate.task_stats.diskio.summary.wait_time	Total time requests spent waiting in queues for service, in nanoseconds	long	counter
awsfargate.task_stats.diskio.total	Number of reads and writes per second	scaled_float	gauge
awsfargate.task_stats.diskio.write.bytes	Bytes written during the life of the container	long	counter
awsfargate.task_stats.diskio.write.ops	Number of writes during the life of the container	long	counter
awsfargate.task_stats.diskio.write.queued	Total number of queued requests	long	counter
awsfargate.task_stats.diskio.write.rate	Number of current writes per second	long	gauge
awsfargate.task_stats.diskio.write.service_time	Total time to service IO requests, in nanoseconds	long	counter
awsfargate.task_stats.diskio.write.wait_time	Total time requests spent waiting in queues for service, in nanoseconds	long	counter
awsfargate.task_stats.diskio.writes	Number of current writes per second	scaled_float	gauge
awsfargate.task_stats.identifier	Container identifier across tasks and clusters, which equals to container.name + / + container.id.	keyword
awsfargate.task_stats.memory.commit.peak	Peak committed bytes on Windows	long	counter
awsfargate.task_stats.memory.commit.total	Total bytes	long	counter
awsfargate.task_stats.memory.fail.count	Fail counter.	scaled_float	counter
awsfargate.task_stats.memory.limit	Memory limit.	long	gauge
awsfargate.task_stats.memory.private_working_set.total	Private working sets on Windows	long	gauge
awsfargate.task_stats.memory.rss.pct	Memory resident set size percentage.	scaled_float	gauge
awsfargate.task_stats.memory.rss.total	Total memory resident set size.	long	gauge
awsfargate.task_stats.memory.rss.usage.max	Max memory usage.	long	counter
awsfargate.task_stats.memory.rss.usage.pct	Memory usage percentage.	scaled_float	gauge
awsfargate.task_stats.memory.rss.usage.total	Total memory usage.	long	gauge
awsfargate.task_stats.memory.stats.*	Raw memory stats from the cgroups memory.stat interface	unsigned_long
awsfargate.task_stats.memory.usage.max	Max memory usage.	long	counter
awsfargate.task_stats.memory.usage.total	Total memory usage.	long	gauge
awsfargate.task_stats.network.*.inbound.bytes	Total number of incoming bytes.	long	counter
awsfargate.task_stats.network.*.inbound.dropped	Total number of dropped incoming packets.	long	counter
awsfargate.task_stats.network.*.inbound.errors	Total errors on incoming packets.	long	counter
awsfargate.task_stats.network.*.inbound.packets	Total number of incoming packets.	long	counter
awsfargate.task_stats.network.*.outbound.bytes	Total number of incoming bytes.	long	counter
awsfargate.task_stats.network.*.outbound.dropped	Total number of dropped incoming packets.	long	counter
awsfargate.task_stats.network.*.outbound.errors	Total errors on incoming packets.	long	counter
awsfargate.task_stats.network.*.outbound.packets	Total number of incoming packets.	long	counter
awsfargate.task_stats.task_desired_status	The desired status for the task from Amazon ECS.	keyword
awsfargate.task_stats.task_known_status	The known status for the task from Amazon ECS.	keyword
awsfargate.task_stats.task_name	ECS task name	keyword
container.labels.com_amazonaws_ecs_cluster	ECS Cluster name	keyword
container.labels.com_amazonaws_ecs_container-name	ECS container name	keyword
container.labels.com_amazonaws_ecs_task-arn	ECS task ARN	keyword
container.labels.com_amazonaws_ecs_task-definition-family	ECS task definition family	keyword
container.labels.com_amazonaws_ecs_task-definition-version	ECS task definition version	keyword
container.name	Container name.	keyword
data_stream.dataset	Data stream dataset.	constant_keyword
data_stream.namespace	Data stream namespace.	constant_keyword
data_stream.type	Data stream type.	constant_keyword

Example

An example event for task_stats looks as following:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "awsfargate": {
        "task_stats": {
            "cluster_name": "default",
            "task_known_status": "RUNNING",
            "task_desired_status": "RUNNING",
            "cpu": {
                "core": {
                    "1": {
                        "pct": 0,
                        "norm": {
                            "pct": 0
                        },
                        "ticks": 1520000000
                    },
                    "2": {
                        "pct": 0,
                        "norm": {
                            "pct": 0
                        },
                        "ticks": 1420180000000
                    }
                },
                "kernel": {
                    "norm": {
                        "pct": 0
                    },
                    "pct": 0,
                    "ticks": 1520000000
                },
                "system": {
                    "norm": {
                        "pct": 1
                    },
                    "pct": 2,
                    "ticks": 1420180000000
                },
                "total": {
                    "norm": {
                        "pct": 0.2
                    },
                    "pct": 0.4
                },
                "user": {
                    "norm": {
                        "pct": 0
                    },
                    "pct": 0,
                    "ticks": 490000000
                }
            },
            "diskio": {
                "read": {
                    "bytes": 3452928,
                    "ops": 118,
                    "queued": 0,
                    "rate": 0,
                    "service_time": 0,
                    "wait_time": 0
                },
                "reads": 0,
                "summary": {
                    "bytes": 3452928,
                    "ops": 118,
                    "queued": 0,
                    "rate": 0,
                    "service_time": 0,
                    "wait_time": 0
                },
                "total": 0,
                "write": {
                    "bytes": 0,
                    "ops": 0,
                    "queued": 0,
                    "rate": 0,
                    "service_time": 0,
                    "wait_time": 0
                },
                "writes": 0
            },
            "identifier": "query-metadata/1234",
            "memory": {
                "fail": {
                    "count": 0
                },
                "limit": 0,
                "rss": {
                    "pct": 0.0010557805807105247,
                    "total": 4157440
                },
                "stats": {
                    "active_anon": 4157440,
                    "active_file": 4497408,
                    "cache": 6000640,
                    "dirty": 16384,
                    "hierarchical_memory_limit": 2147483648,
                    "hierarchical_memsw_limit": 9223372036854772000,
                    "inactive_anon": 0,
                    "inactive_file": 1503232,
                    "mapped_file": 2183168,
                    "pgfault": 6668,
                    "pgmajfault": 52,
                    "pgpgin": 5925,
                    "pgpgout": 3445,
                    "rss": 4157440,
                    "rss_huge": 0,
                    "total_active_anon": 4157440,
                    "total_active_file": 4497408,
                    "total_cache": 600064,
                    "total_dirty": 16384,
                    "total_inactive_anon": 0,
                    "total_inactive_file": 4497408,
                    "total_mapped_file": 2183168,
                    "total_pgfault": 6668,
                    "total_pgmajfault": 52,
                    "total_pgpgin": 5925,
                    "total_pgpgout": 3445,
                    "total_rss": 4157440,
                    "total_rss_huge": 0,
                    "total_unevictable": 0,
                    "total_writeback": 0,
                    "unevictable": 0,
                    "writeback": 0
                },
                "usage": {
                    "max": 15294464,
                    "total": 12349440
                }
            },
            "network": {
                "eth0": {
                    "inbound": {
                        "bytes": 137315578,
                        "dropped": 0,
                        "errors": 0,
                        "packets": 94338
                    },
                    "outbound": {
                        "bytes": 1086811,
                        "dropped": 0,
                        "errors": 0,
                        "packets": 25857
                    }
                }
            },
            "task_name": "query-metadata"
        }
    },
    "cloud": {
        "region": "us-west-2"
    },
    "container": {
        "id": "1234",
        "image": {
            "name": "mreferre/eksutils"
        },
        "labels": {
            "com_amazonaws_ecs_cluster": "arn:aws:ecs:us-west-2:111122223333:cluster/default",
            "com_amazonaws_ecs_container-name": "query-metadata",
            "com_amazonaws_ecs_task-arn": "arn:aws:ecs:us-west-2:111122223333:task/default/febee046097849aba589d4435207c04a",
            "com_amazonaws_ecs_task-definition-family": "query-metadata",
            "com_amazonaws_ecs_task-definition-version": "7"
        },
        "name": "query-metadata"
    },
    "service": {
        "type": "awsfargate"
    }
}

Changelog

edit

Changelog

Version	Details	Kibana version(s)
1.3.0	Enhancement (View pull request) Add support for Kibana `9.0.0`.	8.13.0 or higher 9.0.0 or higher
1.2.2	Enhancement (View pull request) Add missing category.	8.13.0 or higher
1.2.1	Enhancement (View pull request) Clarify that the integration supports ECS clusters only.	8.13.0 or higher
1.2.0	Enhancement (View pull request) Add processor support for task_stats data stream.	8.13.0 or higher
1.1.0	Enhancement (View pull request) ECS version updated to 8.11.0. Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.	8.13.0 or higher
1.0.0	Enhancement (View pull request) Make AWS Fargate GA	8.12.0 or higher
0.5.1	Enhancement (View pull request) Improve documentation	—
0.5.0	Bug fix (View pull request) Remove memory.usage.pct field and use memory.usage.total instead for plain memory usage.	—
0.4.0	Enhancement (View pull request) Update the package format_version to 3.0.0.	—
0.3.0	Enhancement (View pull request) Enable TSDB for task stats data stream. This improves storage usage and query performance. For more details, see https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html.	—
0.2.5	Enhancement (View pull request) Update DiskIO Write and Read visualizations to use last_value instead of average.	—
0.2.4	Enhancement (View pull request) Migrate AWS Fargate input control to new control panel.	—
0.2.3	Enhancement (View pull request) Set dimension fields and add `agent.id`.	—
0.2.2	Enhancement (View pull request) Add metric type to fields.	—
0.2.1	Enhancement (View pull request) Added categories and/or subcategories.	—
0.2.0	Enhancement (View pull request) Improve dashboards by removing individual visualizations from library	—
0.1.3	Enhancement (View pull request) Clarify how to run the `awsfargate` integration as a sidecar container.	—
0.1.2	Enhancement (View pull request) Add DesiredStatus and KnownStatus for Fargate Tasks among the collected fields	—
0.1.1	Enhancement (View pull request) Improve description and screenshots	—
0.1.0	Enhancement (View pull request) initial release	—

« Amazon ELB Inspector »

On this page

Overview
Credentials
Setup
Getting started using the AWS Management Console
Task Definition
Service
Getting started using the AWS CLI
Setup
Pick a region
Secrets management
Networking
Deploy the stack
Clean up
Further Readings
Metrics
Task Stats
Changelog

Was this helpful?

Feedback

The Search AI Company

Generative AI

Search

Security

Observability

By solution

Industries

AWS Fargate Integration (for ECS clusters)