This integration is powered by Elastic Agent. Elastic Agent is a single, unified way to add monitoring for logs, metrics, and other types of data to a host. It can also protect hosts from security threats, query data from operating systems, forward data from remote services or hardware, and more. Refer to our documentation for a detailed comparison between Beats and Elastic Agent.
Prefer to use Beats for this use case? See Filebeat modules for logs or Metricbeat modules for metrics.
See the integrations quick start guides to get started:
The AWS integration is used to fetch logs and metrics from Amazon Web Services.
Use the AWS integration to collect metrics and logs across many AWS services managed by your AWS account. Visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference data when troubleshooting an issue.
Extra AWS charges on CloudWatch API requests will be generated by this integration. Please see API Requests for more details.
Data streams
The AWS integration collects two types of data, logs and metrics, across many AWS services.
Logs help you keep a record of events that happen in your AWS account. This may include every user request that CloudFront receives, every action taken on your services by an AWS user or role, and more.
Metrics give you insight into the state of your AWS services. his may include understanding where you're spending the most and why, the volume of storage you're using, CPU utilization of your instances, and more.
For a complete list of all AWS services and the data streams available for each, see Reference.
API requests
Overview
The AWS integration uses different AWS API to bootstrap and collect metrics and logs. The following table illustrates which APIs are used by the AWS integration and how.
AWS API Name | AWS API Count | Frequency | Datastream |
---|---|---|---|
IAM ListAccountAliases | 1 | Once on startup | all |
STS GetCallerIdentity | 1 | Once on startup | all |
EC2 DescribeRegions | 1 | Once on startup | all |
CloudWatch ListMetrics | Total number of results / ListMetrics max page size (500, based on AWS API ListMetrics | Per region per collection period | metrics related only |
CloudWatch GetMetricData | Total number of results / GetMetricData max page size (500, based on AWS API GetMetricData | Per region per namespace per collection period | metrics related only |
CloudWatch DescribeLogGroups | Total number of results / DescribeLogGroups max page size (50, based on AWS API DescribeLogGroups | Per region per collection period | logs related only |
CloudWatch FilterLogEvents | Total number of results / FilterLogEvents max page size (1MB or 10'0000 events, based on AWS API FilterLogEvents | Per log group per region per collection period | logs related only |
CostExplorer GetCostAndUsage | Total number of results / GetCostAndUsage max page size (8192, based on AWS API GetCostAndUsage | Per CostExplorer Group Definition per region per collection period | AWS Billing |
S3 ListObjectsV2 | Total number of results / ListObjectsV2 max page size (up to 1,000, based on AWS API FilterLogEvents | Per bucket per region per collection period | logs related only |
S3 GetObject | 1 | Per object per collection period | logs related only |
SecurityHub GetFindings | Total number of results / GetFindings max page size ( 100, based on AWS API GetFindings | Per region per collection period | AWS Security Hub |
SecurityHub GetInsights | Total number of results / GetInsights max page size ( 100, based on AWS API GetInsights | Per region per collection period | AWS Security Hub |
Each of these APIs may generate extra charges on your AWS Account. Please refer to AWS Princing for more information.
Metrics collection and cost considerations
For each AWS service you enable metrics data collection for, the AWS integration will collect metrics in all the AWS regions where there are available metrics for that service. The collection period is also set to sensible defaults that should fit the majority of use cases.
The extra-charges generated by GetMetricData API calls are proportional to the frequency we collect data and the amount of metrics that are queried for. If you are concerned about the cost derived by enabling any metrics collection, we recommend reviewing the following parameters:
Regions
. By selecting only the AWS Regions you are interested in, you can make sure that no unnecessary Cloudwatch API call is performed against irrelevant AWS regions.Collection Period
andData Granularity
. By settingCollection Period
andData Granularity
together, you can control, respectively, how frequently you want your metrics to be collected and how granular they have to be. If you can tolerate an extra delay in retrieving metrics as trade off, you may consider settingdata_granularity
and increase the value forCollection Period
to reduce extra charges. For example, settingData Granularity
to your current value forPeriod
, and doubling the value ofPeriod
, may lead to a 50% savings.Tags Filter
. By specifying a tag, you can ensure that no Cloudwatch API call is performed for AWS resources you are not interested in.
Requirements
Before using the AWS integration you will need:
- AWS Credentials to connect with your AWS account.
- AWS Permissions to make sure the user you're using to connect has permission to share the relevant data.
AWS Credentials
AWS credentials are required for running AWS integrations. There are a few ways to provide AWS credentials:
- Use access keys directly
- Use temporary security credentials
- Use a shared credentials file
- Use an IAM role Amazon Resource Name (ARN)
Use access keys directly
Access keys are long-term credentials for an IAM user or the AWS account root user. To use access keys as credentials, you need to provide:
access_key_id
: The first part of the access key.secret_access_key
: The second part of the access key.
For more details see AWS Access Keys and Secret Access Keys.
Use temporary security credentials
Temporary security credentials can be configured in AWS to last for some period of time.
They consist of an access key ID, a secret access key, and a security token, which is
typically returned using GetSessionToken
.
IAM users with multi-factor authentication (MFA) enabled need to submit an MFA code
while calling GetSessionToken
.
For more details see Temporary Security Credentials.
You can use AWS CLI to generate temporary credentials.
For example, you would use sts get-session-token
if you have MFA enabled:
aws> sts get-session-token --serial-number arn:aws:iam::1234:mfa/your-email@example.com --duration-seconds 129600 --token-code 123456
Then, use the response to provide the following options to the AWS integration:
access_key_id
: The first part of the access key.secret_access_key
: The second part of the access key.session_token
: A token required when using temporary security credentials.
Because temporary security credentials are short term, after they expire, you will need to generate new ones and manually update the package configuration to continue collecting AWS metrics. This will cause data loss if the configuration is not updated with the new credentials before the old ones expire.
Use a shared credentials file
If you use different credentials for different tools or applications, you can use profiles to configure multiple access keys in the same configuration file. For more details see Create Shared Credentials File
Instead of providing the access_key_id
and secret_access_key
directly to the integration,
you will provide two advanced options to look up the access keys in the shared credentials file:
credential_profile_name
: The profile name in shared credentials file.shared_credential_file
: The directory of the shared credentials file.
Note: If you don't provide values for all keys, the integration will use defaults:
- If
access_key_id
,secret_access_key
androle_arn
are all not provided, then the package will check forcredential_profile_name
. - If there is no
credential_profile_name
given, the default profile will be used. - If
shared_credential_file
is empty, the default directory will be used.- In Windows, shared credentials file is located at
C:\Users\<yourUserName>\.aws\credentials
. - For Linux, macOS, or Unix, the file is located at
~/.aws/credentials
.
- In Windows, shared credentials file is located at
Use an IAM role Amazon Resource Name (ARN)
An IAM role ARN is an IAM identity that you can create in your AWS account. You determine what the role has permission to do. A role does not have standard long-term credentials such as a password or access keys associated with it. Instead, when you assume a role it provides you with temporary security credentials for your role session. IAM role ARN can be used to specify which AWS IAM role to assume to generate temporary credentials. For more details see AssumeRole API documentation.
To use an IAM role ARN, you need to provide either a credential profile or
access keys along with the role_arn
advanced option.
role_arn
is used to specify which AWS IAM role to assume for generating temporary credentials.
Note: If role_arn
is given, the package will check if access keys are given.
If they are not given, the package will check for a credential profile name.
If neither is given, the default credential profile will be used.
AWS Permissions
Specific AWS permissions are required for the IAM user to make specific AWS API calls. To enable the AWS integration to collect metrics and logs from all supported services, make sure these permissions are given:
ec2:DescribeInstances
ec2:DescribeRegions
cloudwatch:GetMetricData
cloudwatch:ListMetrics
iam:ListAccountAliases
rds:DescribeDBInstances
rds:ListTagsForResource
s3:GetObject
sns:ListTopics
sqs:ChangeMessageVisibility
sqs:DeleteMessage
sqs:ListQueues
sqs:ReceiveMessage
sts:AssumeRole
sts:GetCallerIdentity
tag:GetResources
Setup
Use the AWS integration to connect to your AWS account and collect data from multiple AWS services. When you configure the integration, you can collect data from as many AWS services as you'd like.
If you only need to collect data from one AWS service, consider using the individual integration (for example, to only collect billing metrics, you can use the AWS CloudFront integration).
For step-by-step instructions on how to set up an integration, see the Getting started guide.
Reference
Below is an overview of the type of data you can collect from each AWS service. Visit the page for each individual AWS integration to see details about exported fields.
Service | Metrics | Logs |
---|---|---|
Billing | x | |
CloudFront | x | |
CloudTrail | x | |
CloudWatch | x | x |
DynamoDB | x | |
EBS | x | |
EC2 | x | x |
ECS | x | |
ELB | x | x |
Fargate | x | |
Kinesis | x | |
Network Firewall | x | x |
Lambda | x | |
NAT Gateway | x | |
RDS | x | |
Route 53 | x | |
S3 | x | x |
S3 Storage Lens | x | |
SNS | x | |
SQS | x | |
Transit Gateway | x | |
Usage | x | |
VPC Flow | x | |
VPN | x | |
WAF | x | |
Redshift | x | |
Custom | x |
Changelog
Version | Details |
---|---|
1.30.0 | View pull request Add dashboards data streams filters. |
1.29.1 | View pull request Drop comments from CloudFront loglines |
1.29.0 | View pull request Add data_granularity parameter and rename period title to Collection Period. |
1.28.3 | View pull request Remove quotes from VPC flow log message field and move dot_expander processor to top |
1.28.2 | View pull request Add dot_expander processor to expand all fields with dot into object fields View pull request Support VPC flow log with message field |
1.28.1 | View pull request Adjust kinesis integration to kinesis data stream |
1.28.0 | View pull request Enhance S3 integration dashboard |
1.27.3 | View pull request Support multiple forwarded IPs in cloudfront integration |
1.27.2 | View pull request Update the pagination termination condition. |
1.27.1 | View pull request Added a Summary Dashboard for AWS Security Hub. |
1.27.0 | View pull request Add Inspector data stream. |
1.25.3 | View pull request Remove duplicate fields from agent.yml and use ecs.yml for ECS fields |
1.25.2 | View pull request Update ec2 fields.yml doc |
1.25.1 | View pull request Remove duplicate 'content_type' config that causes errors while configurating the integration. |
1.25.0 | View pull request Force content type where json content is expected |
1.24.6 | View pull request Enhance Kinesis integration dashboard |
1.24.5 | View pull request Allow adding multiple processors in cloudfront logs. |
1.24.4 | View pull request Do not rely on dynamodb lightweight module metricset. |
1.24.3 | View pull request Fix adding processors in cloudfront logs. |
1.24.2 | View pull request Fix billing datastream agent template. |
1.24.1 | View pull request Fix aws.cloudtrail.request_id parsing |
1.24.0 | View pull request Expose Default Region setting to UI |
1.23.4 | View pull request Set default endpoint to empty string |
1.23.3 | View pull request Fix Billing Dashboard |
1.23.2 | View pull request Fix EC2 dashboard |
1.23.1 | View pull request Update all AWS documentation. |
1.23.0 | View pull request Fix file.path field in cloudtrail data stream to use json.digestS3Object |
1.22.0 | View pull request Update cloud.region parsing |
1.21.0 | View pull request Add Security Hub Findings and Insights data streams |
1.20.0 | View pull request Improve dashboards by removing individual visualizations from library |
1.19.5 | View pull request Move ebs metrics config from beats to integrations |
1.19.4 | View pull request Fix proxy URL documentation rendering. |
1.19.3 | View pull request Update sample_event.json in kinesis data stream |
1.19.2 | View pull request Move NATGateway metrics config from beats to integrations |
1.19.1 | View pull request Move Transit Gateway metrics config from beats to integrations |
1.19.0 | View pull request Add Kinesis metrics datastream |
1.18.2 | View pull request Move s3_request metrics config from beats to integrations View pull request Move s3_daily_storage metrics config from beats to integrations View pull request Move SQS metrics config from beats to integrations View pull request Move SNS metrics config from beats to integrations View pull request Move lambda metrics config from beats to integrations |
1.18.1 | View pull request Release AWS billing integration as GA |
1.18.0 | View pull request Add ECS metricset View pull request Fix incorrect fields on multiple visualizations |
1.17.5 | View pull request Release Amazon Redshift integration as GA |
1.17.4 | View pull request Fix data_stream.dataset indentation on cloudwatch_logs integration |
1.17.3 | View pull request Add missing endpoint config to metrics datasets. View pull request Move usage metrics config from beats to integrations View pull request Move dynamodb metrics config from beats to integrations |
1.17.2 | View pull request Improve support for event.original field from upstream forwarders. |
1.17.1 | View pull request Fix misspelling of Log Stream Prefix variable in manifest for aws-cloudwatch input |
1.17.0 | View pull request Added Redshift integration |
1.16.6 | View pull request Update documentation with additional context for new users. |
1.16.5 | View pull request Move ELB metrics config from beats to integrations |
1.16.4 | View pull request Fix ELB dataset to parse URLs with spaces View pull request Upgrade ECS to 8.2.0 |
1.16.3 | View pull request Move RDS metrics config from beats to integrations |
1.16.2 | View pull request Move EC2 metrics config from beats to integrations |
1.16.1 | View pull request Fix invalid values for ECS fields in vpcflow |
1.16.0 | View pull request Move VPN configuration file into integrations and add tag collection |
1.15.0 | View pull request Deprecate s3 input in cloudwatch integration View pull request Improve description for cloudwatch integration |
1.14.8 | View pull request Fix http.response.status_code to accept 000 |
1.14.7 | View pull request Fix aws.dimensions.* for rds data stream View pull request Fix aws.dimensions.* for sns data stream View pull request Add aws.dimensions.* for dynamodb data stream |
1.14.6 | View pull request Improve s3 integration tile title and description |
1.14.5 | View pull request Fix duplicate titles for integrations |
1.14.4 | View pull request Fix cloudfront integration grok pattern |
1.14.3 | View pull request Add new pattern to VPC Flow logs including all 29 v5 fields |
1.14.2 | View pull request Fix billing dashboard. |
1.14.1 | View pull request Add documentation for multi-fields |
1.14.0 | View pull request Add configuration for max_number_of_messages to the aws.firewall_logs S3 input. |
1.13.1 | View pull request Fix metricbeat- reference in dashboard |
1.13.0 | View pull request Compress dashboard screenshots. |
1.12.1 | View pull request Fix field mapping conflicts in the elb_logs data stream relating to ECS fields ( trace.id , source.port , and a few others). |
1.12.0 | View pull request Add CloudFront Logs Datastream |
1.11.4 | View pull request Add Ingest Pipeline script to map IANA Protocol Numbers |
1.11.3 | View pull request Changing missing ecs versions to 8.0.0 |
1.11.2 | View pull request Add data_stream.dataset option for custom aws-cloudwatch log input |
1.11.1 | View pull request Update permission list |
1.11.0 | View pull request Update to ECS 8.0 |
1.10.2 | View pull request Change cloudwatch metrics and logs default to false |
1.10.1 | View pull request Add description of supported vpcflow formats |
1.10.0 | View pull request Add cloudwatch input into AWS package for log collection |
1.9.0 | View pull request Add Route 53 Resolver Logs Datastream |
1.8.0 | View pull request Add Route 53 Public Zone Logs Datastream |
1.7.1 | View pull request Regenerate test files using the new GeoIP database |
1.7.0 | View pull request Add integration for AWS Network Firewall |
1.6.2 | View pull request Change test public IPs to the supported subset |
1.6.1 | View pull request Fix the value of event.created in CloudTrail data stream. |
1.6.0 | View pull request Add max_number_of_messages config option to AWS S3 input config. |
1.5.1 | View pull request Add missing sample events |
1.5.0 | View pull request Support Kibana 8.0 |
1.4.1 | View pull request Add Overview dashboard for AWS S3 Storage Lens |
1.4.0 | View pull request Add integration for AWS S3 Storage Lens |
1.3.2 | View pull request Uniform with guidelines |
1.3.1 | View pull request Add config parameter descriptions |
1.3.0 | View pull request Add WAF datastream |
1.2.2 | View pull request Prevent pipeline script error |
1.2.1 | View pull request Fix logic that checks for the 'forwarded' tag |
1.2.0 | View pull request Update to ECS 1.12.0 |
1.1.0 | View pull request vpcflow sync with filebeat fileset |
1.0.0 | View pull request Release AWS as GA |
0.10.7 | View pull request Add proxy config |
0.10.6 | View pull request Fix aws.billing.EstimatedCharges field name |
0.10.5 | View pull request Add event.created field |
0.10.4 | View pull request Improve RDS dashboard |
0.10.3 | View pull request Convert to generated ECS fields |
0.10.2 | View pull request update to ECS 1.11.0 |
0.10.1 | View pull request Escape special characters in docs |
0.10.0 | View pull request Update integration description |
0.9.3 | View pull request Fix categories for each policy template |
0.9.2 | View pull request Add linked account information into billing metricset |
0.9.1 | View pull request Fix aws.s3access pipeline when remote IP is a - |
0.9.0 | View pull request Change default credential options to access keys |
0.8.0 | View pull request Set "event.module" and "event.dataset" |
0.7.0 | View pull request Introduce granularity using input_groups |
0.6.4 | View pull request Add support for Splunk authorization tokens |
0.6.3 | View pull request Fix bug in Third Party ingest pipeline |
0.6.2 | View pull request Removed incorrect http.request.referrer field from elb logs |
0.6.1 | View pull request Add support for CloudTrail Digest & Insight logs |
0.6.0 | View pull request Update ECS version, add event.original and preparing for package GA |
0.5.6 | View pull request Fix stack compatability |
0.5.5 | View pull request Allow role_arn work with access keys for AWS |
0.5.4 | View pull request Rename s3 input to aws-s3. |
0.5.3 | View pull request Add missing "geo" fields |
0.5.2 | View pull request update to ECS 1.9.0 |
0.5.1 | View pull request Ignore missing "json" field in ingest pipeline |
0.5.0 | View pull request Moving edge processors to ingest pipeline |
0.4.2 | View pull request Updating package owner |
0.4.1 | View pull request Correct sample event file. |
0.4.0 | View pull request Add changes to use ECS 1.8 fields. |
0.0.3 | View pull request initial release |