Amazon EC2

Collect logs and metrics for Amazon Elastic Compute Cloud service with Elastic Agent


Version	2.10.0 (View all)
Compatible Kibana version(s)	8.10.2 or higher
Subscription level What's this?	Basic

The Amazon EC2 integration allows you to monitor Amazon Elastic Compute Cloud (Amazon EC2)—a cloud compute platform.

Use the Amazon EC2 integration to collect logs and metrics related to your EC2 instances. Then visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference the logs and metrics when troubleshooting an issue.

For example, you could use this data to track Amazon EC2 CPU utilization. Then you can alert when utilization for an instance crosses a predefined threshold.

IMPORTANT: Extra AWS charges on AWS API requests will be generated by this integration. Please refer to the AWS integration for more details.

Data streams

The Amazon EC2 integration collects two types of data: logs and metrics.

Logs help you keep a record of events happening in Amazon EC2. Logs collected by the Amazon EC2 integration include the region in which an instance is running, the operating system architecture, container information, and more. See more details in the Logs reference.

Metrics give you insight into the state of your Amazon EC2 instances. Metrics collected by the Amazon EC2 integration include the Amazon EC2 instance ID, the number of earned CPU credits that an instance has accrued since it was launched or started, and more. See more details in the Metrics reference.

Requirements

You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.

Before using any AWS integration you will need:

AWS Credentials to connect with your AWS account.
AWS Permissions to make sure the user you're using to connect has permission to share the relevant data.

For more details about these requirements, please take a look at the AWS integration documentation.

Setup

Use this integration if you only need to collect data from the Amazon EC2 service.

If you want to collect data from two or more AWS services, consider using the AWS integration. When you configure the AWS integration, you can collect data from as many AWS services as you'd like.

For step-by-step instructions on how to set up an integration, see the Getting started guide.

Advanced options

CloudWatch

The CloudWatch logs input has several advanced options to fit specific use cases.

Latency

AWS CloudWatch Logs sometimes takes extra time to make the latest logs available to clients like the Agent.

The CloudWatch integration offers the latency setting to address this scenario. Latency translates the query's time range to consider the CloudWatch Logs latency. For example, a 5m latency means the integration will query CloudWatch for logs available 5 minutes ago.

Number of workers

If you are collecting log events from multiple log groups using log_group_name_prefix, you should review the value of the number_of_workers.

The number_of_workers setting defines the number of workers assigned to reading from log groups. Each log group matching the log_group_name_prefix requires a worker to keep log ingestion as close to real-time as possible. For example, if log_group_name_prefix matches five log groups, then number_of_workers should be set to 5. The default value is 1.

Logs reference

The ec2 data stream supports both EC2 logs stored in AWS CloudWatch and EC2 logs stored in Amazon S3. For logs stored in S3, you must export logs from log groups to an Amazon S3 bucket which has SQS notification setup already.

With this data stream, EC2 logs will be parsed into fields like ip_address and process.name. For logs from other services, please use the AWS CloudWatch integration.

Exported fields

Field	Description	Type
@timestamp	Event timestamp.	date
aws.ec2.ip_address	The internet address of the requester.	keyword
aws.s3.bucket.arn	The AWS S3 bucket ARN.	keyword
aws.s3.bucket.name	The AWS S3 bucket name.	keyword
aws.s3.object.key	The AWS S3 Object key.	keyword
cloud.account.id	The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.	keyword
cloud.availability_zone	Availability zone in which this host, resource, or service is located.	keyword
cloud.image.id	Image ID for the cloud instance.	keyword
cloud.instance.id	Instance ID of the host machine.	keyword
cloud.instance.name	Instance name of the host machine.	keyword
cloud.machine.type	Machine type of the host machine.	keyword
cloud.project.id	The cloud project identifier. Examples: Google Cloud Project id, Azure Project id.	keyword
cloud.provider	Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.	keyword
cloud.region	Region in which this host, resource, or service is located.	keyword
container.id	Unique container id.	keyword
container.image.name	Name of the image the container was built on.	keyword
container.labels	Image labels.	object
container.name	Container name.	keyword
data_stream.dataset	Data stream dataset.	constant_keyword
data_stream.namespace	Data stream namespace.	constant_keyword
data_stream.type	Data stream type.	constant_keyword
ecs.version	ECS version this event conforms to. `ecs.version` is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events.	keyword
error.message	Error message.	match_only_text
event.dataset	Event dataset	constant_keyword
event.module	Event module	constant_keyword
host.architecture	Operating system architecture.	keyword
host.containerized	If the host is a container.	boolean
host.domain	Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider.	keyword
host.hostname	Hostname of the host. It normally contains what the `hostname` command returns on the host machine.	keyword
host.id	Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`.	keyword
host.ip	Host ip addresses.	ip
host.mac	Host MAC addresses. The notation format from RFC 7042 is suggested: Each octet (that is, 8-bit byte) is represented by two [uppercase] hexadecimal digits giving the value of the octet as an unsigned integer. Successive octets are separated by a hyphen.	keyword
host.name	Name of the host. It can contain what hostname returns on Unix systems, the fully qualified domain name (FQDN), or a name specified by the user. The recommended value is the lowercase FQDN of the host.	keyword
host.os.build	OS build information.	keyword
host.os.codename	OS codename, if any.	keyword
host.os.family	OS family (such as redhat, debian, freebsd, windows).	keyword
host.os.kernel	Operating system kernel version as a raw string.	keyword
host.os.name	Operating system name, without the version.	keyword
host.os.name.text	Multi-field of `host.os.name`.	match_only_text
host.os.platform	Operating system platform (such centos, ubuntu, windows).	keyword
host.os.version	Operating system version as a raw string.	keyword
host.type	Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.	keyword
input.type	Input type	keyword
log.file.path	Full path to the log file this event came from, including the file name. It should include the drive letter, when appropriate. If the event wasn't read from a log file, do not populate this field.	keyword
log.offset	Log offset	long
message	For log events the message field contains the log message, optimized for viewing in a log viewer. For structured logs without an original message field, other fields can be concatenated to form a human-readable summary of the event. If multiple messages exist, they can be combined into one message.	match_only_text
process.name	Process name.	keyword
tags	List of keywords used to tag each event.	keyword

An example event for ec2 looks as following:

{
    "@timestamp": "2020-02-20T07:01:01.000Z",
    "agent": {
        "ephemeral_id": "d86fb0e5-bdd6-4b91-8a3e-3535b80a4118",
        "id": "acba78ef-1401-4689-977c-d8c2e5d6a8fa",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.10.1"
    },
    "aws": {
        "ec2": {
            "ip_address": "ip-172-31-81-156"
        },
        "s3": {
            "bucket": {
                "arn": "arn:aws:s3:::elastic-package-aws-bucket-36908",
                "name": "elastic-package-aws-bucket-36908"
            },
            "object": {
                "key": "ec2.log"
            }
        }
    },
    "cloud": {
        "provider": "",
        "region": "us-east-1"
    },
    "data_stream": {
        "dataset": "aws.ec2_logs",
        "namespace": "ep",
        "type": "logs"
    },
    "ecs": {
        "version": "8.0.0"
    },
    "elastic_agent": {
        "id": "acba78ef-1401-4689-977c-d8c2e5d6a8fa",
        "snapshot": false,
        "version": "8.10.1"
    },
    "event": {
        "agent_id_status": "verified",
        "dataset": "aws.ec2_logs",
        "ingested": "2023-11-03T15:00:09Z",
        "original": "2020-02-20T07:01:01.000Z Feb 20 07:01:01 ip-172-31-81-156 systemd: Stopping User Slice of root."
    },
    "input": {
        "type": "aws-s3"
    },
    "log": {
        "file": {
            "path": "https://elastic-package-aws-bucket-36908.s3.us-east-1.amazonaws.com/ec2.log"
        },
        "offset": 0
    },
    "message": "Stopping User Slice of root.",
    "process": {
        "name": "systemd"
    },
    "tags": [
        "preserve_original_event",
        "forwarded",
        "aws-ec2-logs"
    ]
}

Metrics reference