Logstash is an open source data collection engine with real-time pipelining capabilities. It supports a wide variety of data sources, and can dynamically unify data from disparate sources and normalize the data into destinations of your choice.
Logstash can collect data using a variety of Logstash input plugins, enrich and transform the data with Logstash filter plugins, and output the data to Elasticsearch using the Logstash Elasticsearch output plugin.
You can use Logstash to extend Beats for advanced use cases, such as data routed to multiple destinations or when you need to make your data persistent.
Logstash for Elasticsearch on serverless
Logstash is a powerful, versatile ETL (Extract, Transform, Load) engine that can play an important role in organizations of all sizes. Some capabilities and features for large, self-managed users aren't appropriate for serverless.
You'll use the Logstash
Elasticsearch output plugin to send data to Elasticsearch on serverless.
Some differences to note between Elasticsearch on serverless and self-managed Elasticsearch:
- Your logstash-output-elasticsearch configuration uses API keys to access Elasticsearch from Logstash. User-based security settings are ignored and may cause errors.
- Elasticsearch on serverless uses data lifecycle management (DLM) instead of index lifecycle management (ILM). If you add ILM settings to your Elasticsearch output configuration, they are ignored and may cause errors.
- Logstash monitoring for serverless is available through the Logstash Integration in Elastic Observability.
- The logstash-output-elasticsearch
hostssetting defaults to port
:9200. Set the value to port
To use Logstash to send data to Elasticsearch on serverless, you must be using:
- Logstash 8.10.1 or later
- Logstash Elasticsearch output plugin 11.18.0 or later
- Logstash Elasticsearch input plugin 4.18.0 or later
- Logstash Elasticsearch filter plugin 3.16.0 or later
Use the Security: API key section in the UI to create an API key for securely connecting the Logstash Elasticsearch output to Elasticsearch on serverless. We recommend creating a unique API key per Logstash instance. You can create as many API keys as necessary.
When you set up your API keys, use the metadata option to tag each API key with details that are meaningful to you. This step makes managing multiple API keys easier.
After you generate an API key, add it to your Logstash Elasticsearch output plugin config file's
Here's an example:
api_key => "TiNAGG4BaaMdaH1tRfuU:KnR6yE41RrSowb0kQ0HWoA"
On this page