What is an Elastic integration?

This integration is powered by Elastic Agent. Elastic Agent is a single, unified way to add monitoring for logs, metrics, and other types of data to a host. It can also protect hosts from security threats, query data from operating systems, forward data from remote services or hardware, and more. Refer to our documentation for a detailed comparison between Beats and Elastic Agent.

Prefer to use Beats for this use case? See Filebeat modules for logs or Metricbeat modules for metrics.

Overview

The AWS CloudFront integration allows you to monitor your AWS CloudFront usage.

Use the AWS CloudFront integration to collect and parse logs related to content delivery. Then visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference logs when troubleshooting an issue.

For example, you could use the data from this integration to know when there are more than some number of failed requests for a single piece of content in a given time period. You could also use the data to troubleshoot the underlying issue by looking at additional context in the logs like the number of unique users (by IP address) who experienced the issue, the source of the request, and more.

Data streams

The AWS CloudFront integration collects one type of data: logs.

Logs help you keep a record of every user request that CloudFront receives. These logs are useful for many scenarios, including security and access audits. See more details in the Logs reference.

Requirements

Before using any AWS integration you will need:

  • AWS Credentials to connect with your AWS account.
  • AWS Permissions to make sure the user you're using to connect has permission to share the relevant data.

For more details about these requirements, see the AWS integration documentation.

Setup

Use this integration if you only need to collect data from the AWS CloudFront service.

If you want to collect data from two or more AWS services, consider using the AWS integration. When you configure the AWS integration, you can collect data from as many AWS services as you'd like.

For step-by-step instructions on how to set up an integration, see the Getting started guide.

Logs reference

The cloudfront data stream collects standard logs (also called access logs) from AWS CloudFront. CloudFront standard logs provide detailed records about every request that’s made to a distribution.

Exported fields

FieldDescriptionType
@timestamp
Event timestamp.
date
aws.cloudfront.content_type
The value of the HTTP Content-Type header of the response.
keyword
aws.cloudfront.domain
The domain name of the CloudFront distribution (for example, d111111abcdef8.cloudfront.net).
keyword
aws.cloudfront.edge_detailed_result_type
When the value of the x-edge-result-type field is Error, this field contains the specific type of error. When the object was served to the viewer from the Origin Shield cache, this field contains OriginShieldHit. In all other cases, this field contains the same value as x-edge-result-type.
keyword
aws.cloudfront.edge_location
The edge location that served the request. Each edge location is identified by a three-letter code and an arbitrarily assigned number (for example, DFW3). The three-letter code typically corresponds with the International Air Transport Association (IATA) airport code for an airport near the edge location’s geographic location.
keyword
aws.cloudfront.edge_response_result_type
How the server classified the response just before returning the response to the viewer. See also the x-edge-result-type field.
keyword
aws.cloudfront.edge_result_type
How the server classified the response after the last byte left the server. In some cases, the result type can change between the time that the server is ready to send the response and the time that it finishes sending the response. See also the x-edge-response-result-type field. For example, in HTTP streaming, suppose the server finds a segment of the stream in the cache. In that scenario, the value of this field would ordinarily be Hit. However, if the viewer closes the connection before the server has delivered the entire segment, the final result type (and the value of this field) is Error. WebSocket connections will have a value of Miss for this field because the content is not cacheable and is proxied directly to the origin.
keyword
aws.cloudfront.time_to_first_byte
The number of seconds between receiving the request and writing the first byte of the response, as measured on the server.
float
aws.edge_location
The edge location that served the request. Each edge location is identified by a three-letter code and an arbitrarily assigned number (for example, DFW3). The three-letter code typically corresponds with the International Air Transport Association (IATA) airport code for an airport near the edge location’s geographic location.
alias
cloud.account.id
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
keyword
cloud.availability_zone
Availability zone in which this host is running.
keyword
cloud.image.id
Image ID for the cloud instance.
keyword
cloud.instance.id
Instance ID of the host machine.
keyword
cloud.instance.name
Instance name of the host machine.
keyword
cloud.machine.type
Machine type of the host machine.
keyword
cloud.project.id
Name of the project in Google Cloud.
keyword
cloud.provider
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
keyword
cloud.region
Region in which this host is running.
keyword
container.id
Unique container id.
keyword
container.image.name
Name of the image the container was built on.
keyword
container.labels
Image labels.
object
container.name
Container name.
keyword
data_stream.dataset
Data stream dataset.
constant_keyword
data_stream.namespace
Data stream namespace.
constant_keyword
data_stream.type
Data stream type.
constant_keyword
destination.address
Some event destination addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the .address field. Then it should be duplicated to .ip or .domain, depending on which one it is.
keyword
destination.domain
The domain name of the destination system. This value may be a host name, a fully qualified domain name, or another host naming format. The value may derive from the original event or be added from enrichment.
keyword
ecs.version
ECS version this event conforms to. ecs.version is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events.
keyword
error.message
Error message.
match_only_text
event.dataset
Event dataset
constant_keyword
event.duration
Duration of the event in nanoseconds. If event.start and event.end are known this value should be the difference between the end and start time.
long
event.module
Event module
constant_keyword
host.architecture
Operating system architecture.
keyword
host.containerized
If the host is a container.
boolean
host.domain
Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider.
keyword
host.hostname
Hostname of the host. It normally contains what the hostname command returns on the host machine.
keyword
host.id
Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of beat.name.
keyword
host.ip
Host ip addresses.
ip
host.mac
Host mac addresses.
keyword
host.name
Name of the host. It can contain what hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.
keyword
host.os.build
OS build information.
keyword
host.os.codename
OS codename, if any.
keyword
host.os.family
OS family (such as redhat, debian, freebsd, windows).
keyword
host.os.kernel
Operating system kernel version as a raw string.
keyword
host.os.name
Operating system name, without the version.
keyword
host.os.name.text
Multi-field of host.os.name.
text
host.os.platform
Operating system platform (such centos, ubuntu, windows).
keyword
host.os.version
Operating system version as a raw string.
keyword
host.type
Type of host. For Cloud providers this can be the machine type like t2.medium. If vm, this could be the container, for example, or other information meaningful in your environment.
keyword
http.request.bytes
Total size in bytes of the request (body and headers).
long
http.request.id
A unique identifier for each HTTP request to correlate logs between clients and servers in transactions. The id may be contained in a non-standard HTTP header, such as X-Request-ID or X-Correlation-ID.
keyword
http.request.method
HTTP request method. The value should retain its casing from the original event. For example, GET, get, and GeT are all considered valid values for this field.
keyword
http.request.referrer
Referrer for this HTTP request.
keyword
http.response.body.bytes
Size in bytes of the response body.
long
http.response.bytes
Total size in bytes of the response (body and headers).
long
http.response.status_code
HTTP response status code.
long
http.version
HTTP version.
keyword
network.protocol
In the OSI Model this would be the Application Layer protocol. For example, http, dns, or ssh. The field value must be normalized to lowercase for querying.
keyword
network.type
In the OSI Model this would be the Network Layer. ipv4, ipv6, ipsec, pim, etc The field value must be normalized to lowercase for querying.
keyword
related.hosts
All hostnames or other host identifiers seen on your event. Example identifiers include FQDNs, domain names, workstation names, or aliases.
keyword
related.ip
All of the IPs seen on your event.
ip
source.address
Some event source addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the .address field. Then it should be duplicated to .ip or .domain, depending on which one it is.
keyword
source.as.number
Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet.
long
source.as.organization.name
Organization name.
keyword
source.as.organization.name.text
Multi-field of source.as.organization.name.
match_only_text
source.geo.city_name
City name.
keyword
source.geo.continent_name
Name of the continent.
keyword
source.geo.country_iso_code
Country ISO code.
keyword
source.geo.country_name
Country name.
keyword
source.geo.location
Longitude and latitude.
geo_point
source.geo.region_iso_code
Region ISO code.
keyword
source.geo.region_name
Region name.
keyword
source.ip
IP address of the source (IPv4 or IPv6).
ip
source.port
Port of the source.
long
tags
List of keywords used to tag each event.
keyword
tls.cipher
String indicating the cipher used during the current connection.
keyword
tls.version
Numeric part of the version parsed from the original string.
keyword
tls.version_protocol
Normalized lowercase protocol name parsed from original string.
keyword
url.domain
Domain of the url, such as "www.elastic.co". In some cases a URL may refer to an IP and/or port directly, without a domain name. In this case, the IP address would go to the domain field. If the URL contains a literal IPv6 address enclosed by [ and ] (IETF RFC 2732), the [ and ] characters should also be captured in the domain field.
keyword
url.extension
The field contains the file extension from the original request url, excluding the leading dot. The file extension is only set if it exists, as not every url has a file extension. The leading period must not be included. For example, the value must be "png", not ".png". Note that when the file name has multiple extensions (example.tar.gz), only the last one should be captured ("gz", not "tar.gz").
keyword
url.full
If full URLs are important to your use case, they should be stored in url.full, whether this field is reconstructed or present in the event source.
wildcard
url.full.text
Multi-field of url.full.
match_only_text
url.original
Unmodified original url as seen in the event source. Note that in network monitoring, the observed URL may be a full URL, whereas in access logs, the URL is often just represented as a path. This field is meant to represent the URL as it was observed, complete or not.
wildcard
url.original.text
Multi-field of url.original.
match_only_text
url.path
Path of the request, such as "/search".
wildcard
url.query
The query field describes the query string of the request, such as "q=elasticsearch". The ? is excluded from the query string. If a URL contains no ?, there is no query field. If there is a ? but no query, the query field exists with an empty string. The exists query can be used to differentiate between the two cases.
keyword
url.registered_domain
The highest registered url domain, stripped of the subdomain. For example, the registered domain for "foo.example.com" is "example.com". This value can be determined precisely with a list like the public suffix list (http://publicsuffix.org). Trying to approximate this by simply taking the last two labels will not work well for TLDs such as "co.uk".
keyword
url.scheme
Scheme of the request, such as "https". Note: The : is not part of the scheme.
keyword
url.subdomain
The subdomain portion of a fully qualified domain name includes all of the names except the host name under the registered_domain. In a partially qualified domain, or if the the qualification level of the full name cannot be determined, subdomain contains all of the names below the registered domain. For example the subdomain portion of "www.east.mydomain.co.uk" is "east". If the domain has multiple levels of subdomain, such as "sub2.sub1.example.com", the subdomain field should contain "sub2.sub1", with no trailing period.
keyword
url.top_level_domain
The effective top level domain (eTLD), also known as the domain suffix, is the last part of the domain name. For example, the top level domain for example.com is "com". This value can be determined precisely with a list like the public suffix list (http://publicsuffix.org). Trying to approximate this by simply taking the last label will not work well for effective TLDs such as "co.uk".
keyword
user_agent.device.name
Name of the device.
keyword
user_agent.name
Name of the user agent.
keyword
user_agent.original
Unparsed user_agent string.
keyword
user_agent.original.text
Multi-field of user_agent.original.
match_only_text
user_agent.os.full
Operating system name, including the version or code name.
keyword
user_agent.os.full.text
Multi-field of user_agent.os.full.
match_only_text
user_agent.os.name
Operating system name, without the version.
keyword
user_agent.os.name.text
Multi-field of user_agent.os.name.
match_only_text
user_agent.os.version
Operating system version as a raw string.
keyword
user_agent.version
Version of the user agent.
keyword

An example event for cloudfront looks as following:

{
    "destination": {
        "address": "d111111abcdef8.cloudfront.net",
        "domain": "d111111abcdef8.cloudfront.net"
    },
    "source": {
        "geo": {
            "continent_name": "Europe",
            "region_iso_code": "SE-E",
            "city_name": "Linköping",
            "country_iso_code": "SE",
            "country_name": "Sweden",
            "region_name": "Östergötland County",
            "location": {
                "lon": 15.6167,
                "lat": 58.4167
            }
        },
        "as": {
            "number": 29518,
            "organization": {
                "name": "Bredband2 AB"
            }
        },
        "address": "89.160.20.112",
        "port": 11040,
        "ip": "89.160.20.112"
    },
    "url": {
        "path": "/index.html",
        "extension": "html",
        "registered_domain": "d111111abcdef8.cloudfront.net",
        "scheme": "https",
        "top_level_domain": "cloudfront.net",
        "domain": "d111111abcdef8.cloudfront.net",
        "full": "https://d111111abcdef8.cloudfront.net/index.html"
    },
    "tags": [
        "preserve_original_event"
    ],
    "network": {
        "type": "ipv4",
        "protocol": "https"
    },
    "cloud": {
        "provider": "aws"
    },
    "@timestamp": "2019-12-04T21:02:31.000Z",
    "ecs": {
        "version": "1.12.0"
    },
    "related": {
        "hosts": [
            "d111111abcdef8.cloudfront.net"
        ],
        "ip": [
            "89.160.20.112"
        ]
    },
    "http": {
        "request": {
            "method": "GET",
            "bytes": 23,
            "id": "SOX4xwn4XV6Q4rgb7XiVGOHms_BGlTAC4KyHmureZmBNrjGdRLiNIQ=="
        },
        "version": "2.0",
        "response": {
            "body": {
                "bytes": 78
            },
            "bytes": 392,
            "status_code": 200
        }
    },
    "tls": {
        "cipher": "ECDHE-RSA-AES128-GCM-SHA256",
        "version": "1.2",
        "version_protocol": "tls"
    },
    "event": {
        "ingested": "2022-01-07T06:44:14.262549044Z",
        "original": "2019-12-04\t21:02:31\tLAX1\t392\t89.160.20.112\tGET\td111111abcdef8.cloudfront.net\t/index.html\t200\t-\tMozilla/5.0%20(Windows%20NT%2010.0;%20Win64;%20x64)%20AppleWebKit/537.36%20(KHTML,%20like%20Gecko)%20Chrome/78.0.3904.108%20Safari/537.36\t-\t-\tHit\tSOX4xwn4XV6Q4rgb7XiVGOHms_BGlTAC4KyHmureZmBNrjGdRLiNIQ==\td111111abcdef8.cloudfront.net\thttps\t23\t0.001\t-\tTLSv1.2\tECDHE-RSA-AES128-GCM-SHA256\tHit\tHTTP/2.0\t-\t-\t11040\t0.001\tHit\ttext/html\t78\t-\t-",
        "kind": "event",
        "id": "SOX4xwn4XV6Q4rgb7XiVGOHms_BGlTAC4KyHmureZmBNrjGdRLiNIQ==",
        "category": "web",
        "type": [
            "access"
        ],
        "outcome": "success"
    },
    "aws": {
        "cloudfront": {
            "edge_result_type": "Hit",
            "content_type": "text/html",
            "edge_detailed_result_type": "Hit",
            "domain": "d111111abcdef8.cloudfront.net",
            "edge_response_result_type": "Hit",
            "time_to_first_byte": 0.001,
            "edge_location": "LAX1"
        }
    },
    "user_agent": {
        "name": "Chrome",
        "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36",
        "os": {
            "name": "Windows",
            "version": "10",
            "full": "Windows 10"
        },
        "device": {
            "name": "Other"
        },
        "version": "78.0.3904.108"
    }
}