You are viewing docs on Elastic's new documentation system, currently in technical preview. For all other Elastic docs, visit

Custom AWS Logs

Collect raw logs from AWS S3 or CloudWatch with Elastic Agent.

Beta feature

This functionality is in beta and is subject to change. The design and code is less mature than official generally available features and is being provided as-is with no warranties. Beta features are not subject to the support service level agreement of official generally available features.

What is an Elastic integration?

This integration is powered by Elastic Agent. Elastic Agent is a single, unified way to add monitoring for logs, metrics, and other types of data to a host. It can also protect hosts from security threats, query data from operating systems, forward data from remote services or hardware, and more. Refer to our documentation for a detailed comparison between Beats and Elastic Agent.

Prefer to use Beats for this use case? See Filebeat modules for logs or Metricbeat modules for metrics.

The custom AWS input integration offers users two ways to collect logs from AWS: from an S3 bucket (with or without SQS notification) and from CloudWatch. Custom ingest pipelines may be added by adding the name to the pipeline configuration option, creating custom ingest pipelines can be done either through the API or the Ingest Node Pipeline UI.

Collecting logs from S3 bucket

When collecting logs from S3 bucket is enabled, users can retrieve logs from S3 objects that are pointed to by S3 notification events read from an SQS queue or directly polling list of S3 objects in an S3 bucket.

The use of SQS notification is preferred: polling list of S3 objects is expensive in terms of performance and costs and should be preferably used only when no SQS notification can be attached to the S3 buckets. This input integration also supports S3 notification from SNS to SQS.

SQS notification method is enabled setting queue_url configuration value. S3 bucket list polling method is enabled setting bucket_arn configuration value and number_of_workers value. Both queue_url and bucket_arn cannot be set at the same time and at least one of the two value must be set.

Collecting logs from CloudWatch

When collecting logs from CloudWatch is enabled, users can retrieve logs from all log streams in a specific log group. filterLogEvents AWS API is used to list log events from the specified log group. Amazon CloudWatch Logs can be used to store log files from Amazon Elastic Compute Cloud(EC2), AWS CloudTrail, Route53, and other sources.


Enhancement View pull request
Remove duplicated number_of_workers settings
Enhancement View pull request
Add permissions to reroute events to logs-- for generic datastream
Enhancement View pull request
Add multiline support for using s3 input
Enhancement View pull request
Added categories and/or subcategories.
Enhancement View pull request
Add required field number of workers to support non aws buckets, and add default value.
Bug fix View pull request
Add latency config parameter for aws-cloudwatch input
Enhancement View pull request
Expose Default Region setting to UI
Bug fix View pull request
Set default endpoint to empty string
Bug fix View pull request
Fix proxy URL documentation rendering.
Bug fix View pull request
Fix misspelling of Log Stream Prefix variable in manifest for aws-cloudwatch input
Bug fix View pull request
update readme file
Bug fix View pull request
Add kibana version constraint
Enhancement View pull request
Move s3 input and cloudwatch input into the same generic data stream
Enhancement View pull request
initial release