- Elastic integrations
- Integrations quick reference
- 1Password
- Abnormal Security
- ActiveMQ
- Active Directory Entity Analytics
- Airflow
- Akamai
- Apache
- API (custom)
- Arbor Peakflow SP Logs
- Arista NG Firewall
- Atlassian
- Auditd
- Auth0
- authentik
- AWS
- Amazon CloudFront
- Amazon DynamoDB
- Amazon EBS
- Amazon EC2
- Amazon ECS
- Amazon EMR
- AWS API Gateway
- Amazon GuardDuty
- AWS Health
- Amazon Kinesis Data Firehose
- Amazon Kinesis Data Stream
- Amazon Managed Streaming for Apache Kafka (MSK)
- Amazon NAT Gateway
- Amazon RDS
- Amazon Redshift
- Amazon S3
- Amazon S3 Storage Lens
- Amazon Security Lake
- Amazon SNS
- Amazon SQS
- Amazon VPC
- Amazon VPN
- AWS Bedrock
- AWS Billing
- AWS CloudTrail
- AWS CloudWatch
- AWS ELB
- AWS Fargate
- AWS Inspector
- AWS Lambda
- AWS Logs (custom)
- AWS Network Firewall
- AWS Route 53
- AWS Security Hub
- AWS Transit Gateway
- AWS Usage
- AWS WAF
- Azure
- Activity logs
- App Service
- Application Gateway
- Application Insights metrics
- Application Insights metrics overview
- Application State Insights metrics
- Azure logs (v2 preview)
- Azure OpenAI
- Billing metrics
- Container instance metrics
- Container registry metrics
- Container service metrics
- Custom Azure Logs
- Custom Blob Storage Input
- Database Account metrics
- Event Hub input
- Firewall logs
- Frontdoor
- Functions
- Microsoft Entra ID
- Monitor metrics
- Network Watcher VNet
- Network Watcher NSG
- Platform logs
- Resource metrics
- Spring Cloud logs
- Storage Account metrics
- Virtual machines metrics
- Virtual machines scaleset metrics
- Barracuda
- BitDefender
- Bitwarden
- blacklens.io
- Blue Coat Director Logs
- BBOT (Bighuge BLS OSINT Tool)
- Box Events
- Bravura Monitor
- Broadcom ProxySG
- Canva
- Cassandra
- CEL Custom API
- Ceph
- Check Point
- Cilium Tetragon
- CISA Known Exploited Vulnerabilities
- Cisco
- Cisco Meraki Metrics
- Citrix
- Claroty CTD
- Cloudflare
- Cloud Asset Inventory
- CockroachDB Metrics
- Common Event Format (CEF)
- Containerd
- CoreDNS
- Corelight
- Couchbase
- CouchDB
- Cribl
- CrowdStrike
- Cyberark
- Cybereason
- CylanceProtect Logs
- Custom Websocket logs
- Darktrace
- Data Exfiltration Detection
- DGA
- Digital Guardian
- Docker
- Elastic APM
- Elastic Fleet Server
- Elastic Security
- Elastic Stack monitoring
- Elasticsearch Service Billing
- Envoy Proxy
- ESET PROTECT
- ESET Threat Intelligence
- etcd
- Falco
- F5
- File Integrity Monitoring
- FireEye Network Security
- First EPSS
- Forcepoint Web Security
- ForgeRock
- Fortinet
- Gigamon
- GitHub
- GitLab
- Golang
- Google Cloud
- Custom GCS Input
- GCP
- GCP Audit logs
- GCP Billing metrics
- GCP Cloud Run metrics
- GCP CloudSQL metrics
- GCP Compute metrics
- GCP Dataproc metrics
- GCP DNS logs
- GCP Firestore metrics
- GCP Firewall logs
- GCP GKE metrics
- GCP Load Balancing metrics
- GCP Metrics Input
- GCP PubSub logs (custom)
- GCP PubSub metrics
- GCP Redis metrics
- GCP Security Command Center
- GCP Storage metrics
- GCP VPC Flow logs
- GCP Vertex AI
- GoFlow2 logs
- Hadoop
- HAProxy
- Hashicorp Vault
- HTTP Endpoint logs (custom)
- IBM MQ
- IIS
- Imperva
- InfluxDb
- Infoblox
- Iptables
- Istio
- Jamf Compliance Reporter
- Jamf Pro
- Jamf Protect
- Jolokia Input
- Journald logs (custom)
- JumpCloud
- Kafka
- Keycloak
- Kubernetes
- LastPass
- Lateral Movement Detection
- Linux Metrics
- Living off the Land Attack Detection
- Logs (custom)
- Lumos
- Lyve Cloud
- Mattermost
- Memcached
- Menlo Security
- Microsoft
- Microsoft 365
- Microsoft Defender for Cloud
- Microsoft Defender for Endpoint
- Microsoft DHCP
- Microsoft DNS Server
- Microsoft Entra ID Entity Analytics
- Microsoft Exchange Online Message Trace
- Microsoft Exchange Server
- Microsoft Graph Activity Logs
- Microsoft M365 Defender
- Microsoft Office 365 Metrics Integration
- Microsoft Sentinel
- Microsoft SQL Server
- Mimecast
- ModSecurity Audit
- MongoDB
- MongoDB Atlas
- MySQL
- Nagios XI
- NATS
- NetFlow Records
- Netskope
- Network Beaconing Identification
- Network Packet Capture
- Nginx
- Okta
- Oracle
- OpenCanary
- Osquery
- Palo Alto
- pfSense
- PHP-FPM
- PingOne
- PingFederate
- Pleasant Password Server
- PostgreSQL
- Prometheus
- Proofpoint TAP
- Proofpoint On Demand
- Pulse Connect Secure
- Qualys VMDR
- QNAP NAS
- RabbitMQ Logs
- Radware DefensePro Logs
- Rapid7
- Redis
- Rubrik RSC Metrics Integration
- Salesforce
- SentinelOne
- ServiceNow
- Slack Logs
- Snort
- Snyk
- SonicWall Firewall
- Sophos
- Spring Boot
- SpyCloud Enterprise Protection
- SQL Input
- Squid Logs
- SRX
- STAN
- Statsd Input
- Sublime Security
- Suricata
- StormShield SNS
- Symantec
- Symantec Endpoint Security
- Sysmon for Linux
- Sysdig
- Syslog Router Integration
- System
- System Audit
- Tanium
- TCP Logs (custom)
- Teleport
- Tenable
- Threat intelligence
- ThreatConnect
- Threat Map
- Thycotic Secret Server
- Tines
- Traefik
- Trellix
- Trend Micro
- TYCHON Agentless
- UDP Logs (custom)
- Universal Profiling
- Vectra Detect
- VMware
- WatchGuard Firebox
- WebSphere Application Server
- Windows
- Wiz
- Zeek
- ZeroFox
- Zero Networks
- ZooKeeper Metrics
- Zoom
- Zscaler
Kubernetes integration
editKubernetes integration
editVersion |
1.68.1 (View all) |
Compatible Kibana version(s) |
8.15.0 or higher |
Supported Serverless project types |
Security |
Subscription level |
Basic |
Level of support |
Elastic |
This integration is used to collect logs and metrics from Kubernetes clusters. Time series index mode enabled for metrics data streams.
This integration requires kube-state-metrics, which is not included with Kubernetes by default. For dashboards to properly populate, the kube-state-metrics service must be deployed to your Kubernetes cluster |
As one of the main pieces provided for Kubernetes monitoring, this integration is capable of fetching metrics from several components:
Some of the previous components are running on each of the Kubernetes nodes (like kubelet
or proxy
) while others provide
a single cluster-wide endpoint. This is important to determine the optimal configuration and running strategy
for the different datasets included in the integration.
Kubernetes endpoints and metricsets
editKubernetes module is a bit complex as its internal datasets require access to a wide variety of endpoints.
This section highlights and introduces some groups of datasets with similar endpoint access needs.
For more details on the datasets see configuration example
and the datasets
sections below.
node / system / pod / container / module / volume
editThe datasets container
, node
, pod
, system
and volume
require access to the kubelet endpoint
in each of
the Kubernetes nodes, hence it’s recommended to include them as part
of an Agent DaemonSet
or standalone Agents running on the hosts.
Depending on the version and configuration of Kubernetes nodes, kubelet
might provide a read only http port (typically 10255),
which is used in some configuration examples. But in general, and lately, this endpoint requires SSL (https
) access
(to port 10250 by default) and token based authentication.
state_* and event
editState_* datasets are enabled by default.
All datasets with the state_
prefix require hosts
field pointing to kube-state-metrics
service within the cluster. As the service provides cluster-wide metrics, there’s no need to fetch them per node,
hence the recommendation is to run these datasets as part of an Agent Deployment
with one only replica.
Generally kube-state-metrics
runs a Deployment
and is accessible via a service called kube-state-metrics
on
kube-system
namespace, which will be the service to use in our configuration.
apiserver
editThe apiserver dataset requires access to the Kubernetes API, which should be easily available in all Kubernetes
environments. Depending on the Kubernetes configuration, the API access might require SSL (https
) and token
based authentication.
proxy
editThe proxy dataset requires access to the proxy endpoint in each of Kubernetes nodes, hence it’s recommended
to configure it as a part of an Agent DaemonSet
.
scheduler and controllermanager
editThese datasets require access to the Kubernetes controller-manager
and scheduler
endpoints. By default, these pods
run only on master nodes, and they are not exposed via a Service, but there are different strategies
available for its configuration:
-
Create
Kubernetes Services
to makekube-controller-manager
andkube-scheduler
available and configure the datasets to point to these services as part of anAgent Deployment
. -
Run these datasets as part an
Agent Daemonset
(with HostNetwork setting) with anodeSelector
to only run on Master nodes.
These datasets are not enabled by default.
In some "As a Service" Kubernetes implementations, like GKE
, the master nodes or even the pods running on
the masters won’t be visible. In these cases it won’t be possible to use scheduler
and controllermanager
metricsets.
container-logs
editThe container-logs dataset requires access to the log files in each Kubernetes node where the container logs are stored.
This defaults to /var/log/containers/*${kubernetes.container.id}.log
.
Routing
editThe container-logs data stream allows routing logs to a different dataset or namespace using pod annotations.
For example, suppose you are running Nginx on your Kubernetes cluster, and you want to drive the Nginx container logs into a dedicated dataset or namespace. By annotating the pod with elastic.co/namespace: nginx
, the integration will send all the container logs to the nginx
namespace.
To learn more about routing container-logs, see https://docs.elastic.co/integrations/kubernetes/container-logs.
Preserve original event
editThe agent can be configured to set the tag preserve_original_event
on container-logs using pod annotation.
For example, suppose you are routing your Nginx container logs into a dedicated dataset or namespace as described above to make use of the Nginx fleet integration. Enabling preserve_original_event on the Nginx integration will have no effect since the logs were shipped via kubernetes integration and not Nginx. As well, you may not want to have all original events from all Nginx pods preserved as well.
By annotating the pod with elastic.co/preserve_original_event: 'true'
, the integration will add the tag preserve_original_event
as it would be done by the nginx
integration otherwise.
audit-logs
editThe audit-logs dataset requires access to the log files on each Kubernetes node where the audit logs are stored.
This defaults to /var/log/kubernetes/kube-apiserver-audit.log
.
Compatibility
editThe Kubernetes package is tested with Kubernetes [1.28.x - 1.31.x] versions
Dashboard
editKubernetes integration is shipped including default dashboards for apiserver
, controllermanager
, overview
, proxy
and scheduler
.
If you are using HA for those components, be aware that when gathering data from all instances the dashboard will usually show the average of the metrics. For those scenarios filtering by hosts or service address is possible.
Cluster selector in overview
dashboard helps in distinguishing and filtering metrics collected from multiple clusters. If you want to focus on a subset of the Kubernetes clusters for monitoring a specific scenario, this cluster selector could be a handy tool. Note that this selector gets populated from the orchestrator.cluster.name
field that may not always be available. This field gets its value from sources like kube_config
, kubeadm-config
configMap, and Google Cloud’s meta API for GKE. If the sources mentioned above don’t provide this value, metricbeat will not report it. However, you can always use processors to set this field and utilize it in the cluster overview
dashboard.
Changelog
editChangelog
Version | Details | Kibana version(s) |
---|---|---|
1.68.1 |
Bug fix (View pull request) |
8.15.0 or higher |
1.68.0 |
Enhancement (View pull request) |
8.15.0 or higher |
1.67.0 |
Enhancement (View pull request) |
8.15.0 or higher |
1.66.4 |
Bug fix (View pull request) |
8.15.0 or higher |
1.66.3 |
Enhancement (View pull request) |
8.15.0 or higher |
1.66.2 |
Bug fix (View pull request) |
8.15.0 or higher |
1.66.1 |
Bug fix (View pull request) |
8.15.0 or higher |
1.66.0 |
Enhancement (View pull request) |
8.15.0 or higher |
1.65.0 |
Enhancement (View pull request) |
8.15.0 or higher |
1.64.0 |
Enhancement (View pull request) |
8.15.0 or higher |
1.63.1 |
Bug fix (View pull request) |
8.15.0 or higher |
1.63.0 |
Enhancement (View pull request) |
8.15.0 or higher |
1.62.1 |
Bug fix (View pull request) |
8.14.0 or higher |
1.62.0 |
Enhancement (View pull request) |
8.14.0 or higher |
1.61.1 |
Bug fix (View pull request) |
8.14.0 or higher |
1.61.0 |
Enhancement (View pull request) |
8.14.0 or higher |
1.60.0 |
Bug fix (View pull request) |
8.14.0 or higher |
1.59.0 |
Enhancement (View pull request) |
8.14.0 or higher |
1.58.0 |
Enhancement (View pull request) |
8.12.0 or higher |
1.57.0 |
Enhancement (View pull request) |
8.12.0 or higher |
1.56.0 |
Enhancement (View pull request) |
8.12.0 or higher |
1.55.1 |
Enhancement (View pull request) |
8.11.0 or higher |
1.55.0 |
Enhancement (View pull request) |
8.11.0 or higher |
1.54.0 |
Enhancement (View pull request) |
8.11.0 or higher |
1.53.0 |
Enhancement (View pull request) |
8.11.0 or higher |
1.52.0 |
Enhancement (View pull request) |
8.11.0 or higher |
1.51.0 |
Enhancement (View pull request) |
8.10.2 or higher |
1.50.0 |
Enhancement (View pull request) |
8.10.2 or higher |
1.49.0 |
Enhancement (View pull request) |
8.10.2 or higher |
1.48.0 |
Enhancement (View pull request) |
8.10.2 or higher |
1.47.0 |
Enhancement (View pull request) |
8.10.2 or higher |
1.46.0 |
Enhancement (View pull request) |
8.10.1 or higher |
1.45.0 |
Enhancement (View pull request) |
8.10.0 or higher |
1.44.0 |
Enhancement (View pull request) |
8.10.0 or higher |
1.43.1 |
Enhancement (View pull request) |
8.8.0 or higher |
1.43.0 |
Enhancement (View pull request) |
8.8.0 or higher |
1.42.0 |
Enhancement (View pull request) |
8.8.0 or higher |
1.41.0 |
Enhancement (View pull request) |
8.8.0 or higher |
1.40.0 |
Bug fix (View pull request) |
8.8.0 or higher |
1.40.0-beta.2 |
Bug fix (View pull request) |
— |
1.40.0-beta.1 |
Bug fix (View pull request) |
— |
1.40.0-beta |
Enhancement (View pull request) |
— |
1.39.0 |
Enhancement (View pull request) |
8.6.1 or higher |
1.38.1 |
Enhancement (View pull request) |
8.6.1 or higher |
1.38.0 |
Enhancement (View pull request) |
8.6.1 or higher |
1.37.0 |
Enhancement (View pull request) |
8.6.1 or higher |
1.36.0 |
Enhancement (View pull request) |
8.6.1 or higher |
1.35.0 |
Enhancement (View pull request) |
8.6.1 or higher |
1.34.1 |
Enhancement (View pull request) |
8.6.1 or higher |
1.34.0 |
Enhancement (View pull request) |
8.6.1 or higher |
1.33.0 |
Enhancement (View pull request) |
8.6.1 or higher |
1.32.2 |
Enhancement (View pull request) |
8.6.1 or higher |
1.32.1 |
Enhancement (View pull request) |
8.6.1 or higher |
1.32.0 |
Enhancement (View pull request) |
8.6.1 or higher |
1.31.2 |
Enhancement (View pull request) |
8.6.1 or higher |
1.31.1 |
Enhancement (View pull request) |
8.6.1 or higher |
1.31.0 |
Enhancement (View pull request) |
8.6.0 or higher |
1.30.0 |
Enhancement (View pull request) |
8.6.0 or higher |
1.29.2 |
Bug fix (View pull request) |
8.5.0 or higher |
1.29.1 |
Bug fix (View pull request) |
8.5.0 or higher |
1.29.0 |
Bug fix (View pull request) |
8.5.0 or higher |
1.28.2 |
Bug fix (View pull request) |
8.5.0 or higher |
1.28.1 |
Enhancement (View pull request) |
8.5.0 or higher |
1.28.0 |
Enhancement (View pull request) |
8.5.0 or higher |
1.27.1 |
Enhancement (View pull request) |
8.5.0 or higher |
1.27.0 |
Enhancement (View pull request) |
8.5.0 or higher |
1.26.0 |
Enhancement (View pull request) |
8.4.0 or higher |
1.25.0 |
Enhancement (View pull request) |
8.4.0 or higher |
1.24.0 |
Enhancement (View pull request) |
8.4.0 or higher |
1.23.1 |
Enhancement (View pull request) |
8.4.0 or higher |
1.23.0 |
Enhancement (View pull request) |
8.4.0 or higher |
1.22.1 |
Enhancement (View pull request) |
8.4.0 or higher |
1.22.0 |
Enhancement (View pull request) |
8.4.0 or higher |
1.21.2 |
Bug fix (View pull request) |
8.3.0 or higher |
1.21.1 |
Enhancement (View pull request) |
8.3.0 or higher |
1.21.0 |
Enhancement (View pull request) |
8.3.0 or higher |
1.20.0 |
Enhancement (View pull request) |
8.2.0 or higher |
1.19.1 |
Enhancement (View pull request) |
8.2.0 or higher |
1.19.0 |
Enhancement (View pull request) |
8.2.0 or higher |
1.18.1 |
Enhancement (View pull request) |
8.2.0 or higher |
1.18.0 |
Enhancement (View pull request) |
8.2.0 or higher |
1.17.3 |
Bug fix (View pull request) |
7.16.0 or higher |
1.17.2 |
Bug fix (View pull request) |
7.16.0 or higher |
1.17.1 |
Enhancement (View pull request) |
— |
1.17.0 |
Enhancement (View pull request) |
— |
1.16.0 |
Enhancement (View pull request) |
— |
1.15.0 |
Enhancement (View pull request) |
— |
1.14.3 |
Bug fix (View pull request) |
— |
1.14.2 |
Bug fix (View pull request) |
— |
1.14.1 |
Bug fix (View pull request) |
— |
1.14.0 |
Enhancement (View pull request) |
— |
1.13.0 |
Enhancement (View pull request) |
— |
1.12.0 |
Enhancement (View pull request) |
— |
1.11.0 |
Enhancement (View pull request) |
— |
1.10.0 |
Enhancement (View pull request) |
— |
1.9.0 |
Enhancement (View pull request) |
7.16.0 or higher |
1.8.1 |
Bug fix (View pull request) |
7.16.0 or higher |
1.8.0 |
Enhancement (View pull request) |
7.16.0 or higher |
1.7.0 |
Enhancement (View pull request) |
7.16.0 or higher |
1.6.0 |
Enhancement (View pull request) |
7.16.0 or higher |
1.5.0 |
Enhancement (View pull request) |
7.16.0 or higher |
1.4.2 |
Enhancement (View pull request) |
— |
1.4.1 |
Enhancement (View pull request) |
8.0.0 or higher |
1.4.0 |
Enhancement (View pull request) |
— |
1.3.3 |
Bug fix (View pull request) |
— |
1.3.2 |
Enhancement (View pull request) |
— |
1.3.1 |
Enhancement (View pull request) |
— |
1.3.0 |
Enhancement (View pull request) |
— |
1.2.1 |
Bug fix (View pull request) |
— |
1.2.0 |
Enhancement (View pull request) |
— |
1.1.1 |
Bug fix (View pull request) |
— |
1.1.0 |
Enhancement (View pull request) |
7.15.0 or higher |
1.0.0 |
Enhancement (View pull request) |
— |
0.14.1 |
Enhancement (View pull request) |
— |
0.14.0 |
Enhancement (View pull request) |
— |
0.13.0 |
Enhancement (View pull request) |
— |
0.12.2 |
Bug fix (View pull request) |
— |
0.12.1 |
Bug fix (View pull request) |
— |
0.12.0 |
Enhancement (View pull request) |
— |
0.11.1 |
Enhancement (View pull request) |
— |
0.11.0 |
Enhancement (View pull request) |
— |
0.10.0 |
Enhancement (View pull request) |
— |
0.9.1 |
Bug fix (View pull request) |
— |
0.9.0 |
Enhancement (View pull request) |
— |
0.8.0 |
Enhancement (View pull request) |
— |
0.7.0 |
Enhancement (View pull request) |
— |
0.6.0 |
Enhancement (View pull request) |
— |
0.5.3 |
Enhancement (View pull request) |
— |
0.5.2 |
Bug fix (View pull request) |
— |
0.5.1 |
Bug fix (View pull request) |
— |
0.5.0 |
Enhancement (View pull request) |
— |
0.4.5 |
Enhancement (View pull request) |
— |
0.4.4 |
Enhancement (View pull request) |
— |
0.4.3 |
Bug fix (View pull request) |
— |
0.4.2 |
Bug fix (View pull request) |
— |
0.4.1 |
Enhancement (View pull request) |
— |
0.1.0 |
Enhancement (View pull request) |
— |
On this page