- Elastic integrations
- Integrations quick reference
- 1Password
- Abnormal Security
- ActiveMQ
- Active Directory Entity Analytics
- Airflow
- Akamai
- Apache
- API (custom)
- Arbor Peakflow SP Logs
- Arista NG Firewall
- Atlassian
- Auditd
- Auth0
- authentik
- AWS
- Amazon CloudFront
- Amazon DynamoDB
- Amazon EBS
- Amazon EC2
- Amazon ECS
- Amazon EMR
- AWS API Gateway
- Amazon GuardDuty
- AWS Health
- Amazon Kinesis Data Firehose
- Amazon Kinesis Data Stream
- Amazon Managed Streaming for Apache Kafka (MSK)
- Amazon NAT Gateway
- Amazon RDS
- Amazon Redshift
- Amazon S3
- Amazon S3 Storage Lens
- Amazon Security Lake
- Amazon SNS
- Amazon SQS
- Amazon VPC
- Amazon VPN
- AWS Bedrock
- AWS Billing
- AWS CloudTrail
- AWS CloudWatch
- AWS ELB
- AWS Fargate
- AWS Inspector
- AWS Lambda
- AWS Logs (custom)
- AWS Network Firewall
- AWS Route 53
- AWS Security Hub
- AWS Transit Gateway
- AWS Usage
- AWS WAF
- Azure
- Activity logs
- App Service
- Application Gateway
- Application Insights metrics
- Application Insights metrics overview
- Application State Insights metrics
- Azure logs (v2 preview)
- Azure OpenAI
- Billing metrics
- Container instance metrics
- Container registry metrics
- Container service metrics
- Custom Azure Logs
- Custom Blob Storage Input
- Database Account metrics
- Event Hub input
- Firewall logs
- Frontdoor
- Functions
- Microsoft Entra ID
- Monitor metrics
- Network Watcher VNet
- Network Watcher NSG
- Platform logs
- Resource metrics
- Spring Cloud logs
- Storage Account metrics
- Virtual machines metrics
- Virtual machines scaleset metrics
- Barracuda
- BitDefender
- Bitwarden
- blacklens.io
- Blue Coat Director Logs
- BBOT (Bighuge BLS OSINT Tool)
- Box Events
- Bravura Monitor
- Broadcom ProxySG
- Canva
- Cassandra
- CEL Custom API
- Ceph
- Check Point
- Cilium Tetragon
- CISA Known Exploited Vulnerabilities
- Cisco
- Cisco Meraki Metrics
- Citrix
- Claroty CTD
- Cloudflare
- Cloud Asset Inventory
- CockroachDB Metrics
- Common Event Format (CEF)
- Containerd
- CoreDNS
- Corelight
- Couchbase
- CouchDB
- Cribl
- CrowdStrike
- Cyberark
- Cybereason
- CylanceProtect Logs
- Custom Websocket logs
- Darktrace
- Data Exfiltration Detection
- DGA
- Digital Guardian
- Docker
- Elastic APM
- Elastic Fleet Server
- Elastic Security
- Elastic Stack monitoring
- Elasticsearch Service Billing
- Envoy Proxy
- ESET PROTECT
- ESET Threat Intelligence
- etcd
- Falco
- F5
- File Integrity Monitoring
- FireEye Network Security
- First EPSS
- Forcepoint Web Security
- ForgeRock
- Fortinet
- Gigamon
- GitHub
- GitLab
- Golang
- Google Cloud
- Custom GCS Input
- GCP
- GCP Audit logs
- GCP Billing metrics
- GCP Cloud Run metrics
- GCP CloudSQL metrics
- GCP Compute metrics
- GCP Dataproc metrics
- GCP DNS logs
- GCP Firestore metrics
- GCP Firewall logs
- GCP GKE metrics
- GCP Load Balancing metrics
- GCP Metrics Input
- GCP PubSub logs (custom)
- GCP PubSub metrics
- GCP Redis metrics
- GCP Security Command Center
- GCP Storage metrics
- GCP VPC Flow logs
- GCP Vertex AI
- GoFlow2 logs
- Hadoop
- HAProxy
- Hashicorp Vault
- HTTP Endpoint logs (custom)
- IBM MQ
- IIS
- Imperva
- InfluxDb
- Infoblox
- Iptables
- Istio
- Jamf Compliance Reporter
- Jamf Pro
- Jamf Protect
- Jolokia Input
- Journald logs (custom)
- JumpCloud
- Kafka
- Keycloak
- Kubernetes
- LastPass
- Lateral Movement Detection
- Linux Metrics
- Living off the Land Attack Detection
- Logs (custom)
- Lumos
- Lyve Cloud
- Mattermost
- Memcached
- Menlo Security
- Microsoft
- Microsoft 365
- Microsoft Defender for Cloud
- Microsoft Defender for Endpoint
- Microsoft DHCP
- Microsoft DNS Server
- Microsoft Entra ID Entity Analytics
- Microsoft Exchange Online Message Trace
- Microsoft Exchange Server
- Microsoft Graph Activity Logs
- Microsoft M365 Defender
- Microsoft Office 365 Metrics Integration
- Microsoft Sentinel
- Microsoft SQL Server
- Mimecast
- ModSecurity Audit
- MongoDB
- MongoDB Atlas
- MySQL
- Nagios XI
- NATS
- NetFlow Records
- Netskope
- Network Beaconing Identification
- Network Packet Capture
- Nginx
- Okta
- Oracle
- OpenCanary
- Osquery
- Palo Alto
- pfSense
- PHP-FPM
- PingOne
- PingFederate
- Pleasant Password Server
- PostgreSQL
- Prometheus
- Proofpoint TAP
- Proofpoint On Demand
- Pulse Connect Secure
- Qualys VMDR
- QNAP NAS
- RabbitMQ Logs
- Radware DefensePro Logs
- Rapid7
- Redis
- Rubrik RSC Metrics Integration
- Salesforce
- SentinelOne
- ServiceNow
- Slack Logs
- Snort
- Snyk
- SonicWall Firewall
- Sophos
- Spring Boot
- SpyCloud Enterprise Protection
- SQL Input
- Squid Logs
- SRX
- STAN
- Statsd Input
- Sublime Security
- Suricata
- StormShield SNS
- Symantec
- Symantec Endpoint Security
- Sysmon for Linux
- Sysdig
- Syslog Router Integration
- System
- System Audit
- Tanium
- TCP Logs (custom)
- Teleport
- Tenable
- Threat intelligence
- ThreatConnect
- Threat Map
- Thycotic Secret Server
- Tines
- Traefik
- Trellix
- Trend Micro
- TYCHON Agentless
- UDP Logs (custom)
- Universal Profiling
- Vectra Detect
- VMware
- WatchGuard Firebox
- WebSphere Application Server
- Windows
- Wiz
- Zeek
- ZeroFox
- Zero Networks
- ZooKeeper Metrics
- Zoom
- Zscaler
Domain Generation Algorithm Detection
editDomain Generation Algorithm Detection
editVersion |
2.1.0 (View all) |
Compatible Kibana version(s) |
8.9.0 or higher |
Supported Serverless project types |
Security |
Subscription level |
Platinum |
Level of support |
Elastic |
The Domain Generation Algorithm (DGA) Detection package contains assets to detect DGA activity in your network data. This package requires a Platinum subscription. Please ensure that you have a Trial or Platinum level subscription installed on your cluster before proceeding. This package is licensed under Elastic License 2.0.
For more detailed information refer to the following blogs:
Installation
edit- Upgrading: If upgrading from a version below v2.0.0, see the section v2.0.0 and beyond.
- Add the Integration Package: Install the package via Management > Integrations > Add Domain Generation Algorithm Detection. Configure the integration name and agent policy. Click Save and Continue.
- Install assets: Install the assets by clicking Settings > Install Domain Generation Algorithm Detection assets.
-
Configure the pipeline: To configure the pipeline you can use one of the following steps:
-
If using Elastic Defend, add a custom pipeline to the data stream. Go to Stack Management > Ingest Pipelines, and check if the pipeline
logs-endpoint.events.network@custom
exists. image::images/dga/custom-pipeline.png[Component Templates] If it does not exist, you can create it by running the following command in the Dev Console. Be sure to replace<VERSION>
with the current package version.PUT _ingest/pipeline/logs-endpoint.events.network@custom { "processors": [ { "pipeline": { "name": "<VERSION>-ml_dga_ingest_pipeline", "ignore_missing_pipeline": true, "ignore_failure": true } } ] }
-
If
logs-endpoint.events.process@custom
already exists, select the three dots next to it and choose Edit. Click Add a processor. Select Pipeline for Processor, enter<VERSION>-ml_dga_ingest_pipeline
for name (replacing<VERSION>
with the current package version), and check Ignore missing pipeline and Ignore failures for this processor. Select Add Processor. -
If using an Elastic Beat such as Packetbeat, add the ingest pipeline to it by adding a simple configuration setting to
packetbeat.yml
and skip to the Add preconfigured anomaly detection jobs section in these instructions.
-
-
Add the required mappings to the index or component template: Go to Stack Management > Index Management > Component Templates. Templates that can be edited to add custom components will be marked with a
@custom
suffix. For instance, the custom component template for Elastic Defend network events islogs-endpoint.events.network@custom
. Note: Do not attempt to edit the@package
template. image::images/dga/component-templates.png[Component Templates]-
If the
@custom
component template does not exist, you can execute the following command in the Dev Console to create it and then continue to the Rollover section in these instructions.PUT _component_template/{COMPONENT_TEMPLATE_NAME}@custom { "template": { "mappings": { "properties": { "ml_is_dga": { "type": "object", "properties": { "malicious_prediction": { "type": "long" }, "malicious_probability": { "type": "float" } } } } } } }
-
If the
@custom
component template already exists, you will need to edit it to add mappings for data to be properly enriched. Click the three dots next to it and select Edit. image::images/dga/component-templates-edit.png[Component Templates] -
On the index settings step, add the following. Be sure to change
<VERSION>
to the current package version.{ "index": { "default_pipeline": "<VERSION>-ml_dga_ingest_pipeline" } }
-
Proceed to the mappings step in the UI. Click Add Field at the bottom of the page and create an an
Object
field forml_is_dga
. image::images/dga/field1.png[Component Templates] -
Finally create two properties under
ml_is_dga
. image::images/dga/field1a.png[Component Templates] -
The first for
malicious_prediction
of typeLong
and then formalicious_probability
or typeFloat
. image::images/dga/field2.png[Component Templates] - Your component mappings should look like the following: image::images/dga/fields-complete.png[Component Templates]
- Click Review then Save Component Template.
-
-
Rollover Depending on your environment, you may need to rollover in order for these mappings to get picked up. The deault index pattern for Elastic Defend is
logs-endpoint.events.network-default
.POST INDEX_NAME/_rollover
- (Optional) Create a data view for your network logs.
-
Add preconfigured anomaly detection jobs: In Machine Learning > Anomaly Detection, when you create a job, you should see an option to
Use preconfigured jobs
with a card forDGA
. When you select the card, you will see a pre-configured anomaly detection job that you can enable depending on what makes the most sense for your environment. Note this job is only useful for indices that have been enriched by the ingest pipeline. -
Enable detection rules: You can also enable detection rules to alert on DGA activity in your environment, based on anomalies flagged by the above ML jobs. As of version 2.0.0 of this package, these rules are available as part of the Detection Engine in Security > Rules, and can be found using the tag
Use Case: Domain Generated Algorithm Detection
. See this documentation for more information on importing and enabling the rules.
In Security > Rules, filtering with the “Use Case: Domain Generation Algorithm Detection” tag
Anomaly Detection Jobs
editJob | Description |
---|---|
dga_high_sum_probability |
Detects potential DGA (domain generation algorithm) activity that is often used by malware command and control (C2) channels. Looks for a source IP address making DNS requests that have an aggregate high probability of being DGA activity. |
v2.0.0 and beyond
editv2.0.0 of the package introduces breaking changes, namely deprecating detection rules from the package. To continue receiving updates to DGA Detection, we recommend upgrading to v2.0.0 after doing the following:
-
Uninstall existing rules associated with this package: Navigate to Security > Rules and delete the following rules:
- Machine Learning Detected DGA activity using a known SUNBURST DNS domain
- Machine Learning Detected a DNS Request Predicted to be a DGA Domain
- Potential DGA Activity
- Machine Learning Detected a DNS Request With a High DGA Probability Score
Depending on the version of the package you’re using, you might also be able to search for the above rules using the tag DGA
- Upgrade the DGA package to v2.0.0 using the steps here
- Install the new rules as described in the Enable detection rules section below
In version 2.0.1 and after, the package ignores data in cold and frozen data tiers to reduce heap memory usage, avoid running on outdated data, and to follow best practices.
Licensing
editUsage in production requires that you have a license key that permits use of machine learning features.
Changelog
editChangelog
Version | Details | Kibana version(s) |
---|---|---|
2.1.0 |
Enhancement (View pull request) |
8.9.0 or higher |
2.0.4 |
Bug fix (View pull request) |
8.9.0 or higher |
2.0.3 |
Bug fix (View pull request) |
8.9.0 or higher |
2.0.2 |
Enhancement (View pull request) |
8.9.0 or higher |
2.0.1 |
Enhancement (View pull request) |
8.9.0 or higher |
2.0.0 |
Enhancement (View pull request) |
8.9.0 or higher |
1.1.0 |
Enhancement (View pull request) |
8.0.0 or higher |
1.0.1 |
Enhancement (View pull request) |
8.0.0 or higher |
1.0.0 |
Enhancement (View pull request) |
8.0.0 or higher |
0.0.5 |
Enhancement (View pull request) |
— |
0.0.4 |
Enhancement (View pull request) |
— |
0.0.3 |
Bug fix (View pull request) |
— |
0.0.2 |
Bug fix (View pull request) |
— |
0.0.1 |
Enhancement (View pull request) |
— |