Amazon VPC

The AWS VPC Flow integration allows you to monitor Amazon Virtual Private Cloud (Amazon VPC) flow logs. Flow logs capture information about the IP traffic going to and from network interfaces in a VPC.

Use the AWS VPC Flow integration to collect logs related to your Amazon VPCs. Then visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference logs when troubleshooting an issue.

For example, you could use this data to:

• Diagnose overly restrictive security group rules
• Monitor the traffic that is reaching your instance
• Determine the direction of the traffic to and from the network interfaces

Then you can alert the relevant project manager about those events by email.

IMPORTANT: Extra AWS charges on AWS API requests will be generated by this integration. Please refer to the AWS integration for more details.

Data streams

The AWS VPC Flow integration collects one type of data: logs.

Logs help you keep a record of events happening in your VPCs. Logs collected by the vpcflow integration include the packet-level (original) source and destination IP addresses for the traffic, accepted traffic, rejected traffic, and more. See more details in the Logs reference.

Requirements

You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.

Before using any AWS integration you will need:

• AWS Credentials to connect with your AWS account.
• AWS Permissions to make sure the user you're using to connect has permission to share the relevant data.

For more details about these requirements, see the AWS integration documentation.

Setup

Use this integration if you only need to collect IP traffic data for your VPCs.

If you want to collect data from two or more AWS services, consider using the AWS integration. When you configure the AWS integration, you can collect data from as many AWS services as you'd like.

For step-by-step instructions on how to set up an integration, see the Getting started guide.

For more information on implementation, see the Amazon documentation on:

• Default Flow Log Format
• Custom Format with Traffic Through a NAT Gateway
• Custom Format with Traffic Through a Transit Gateway

This integration supports various plain text VPC flow log formats:

• The default pattern of 14 version 2 fields
• A custom pattern including all 29 fields, version 2 though 5:

${version} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status} ${vpc-id} ${subnet-id} ${instance-id} ${tcp-flags} ${type} ${pkt-srcaddr} ${pkt-dstaddr} ${region} ${az-id} ${sublocation-type} ${sublocation-id} ${pkt-src-aws-service} ${pkt-dst-aws-service} ${flow-direction} ${traffic-path}

Logs reference

Note: The Parquet format is not supported.

Exported fields

An example event for vpcflow looks as following:

{
    "data_stream": {
        "namespace": "default",
        "type": "logs",
        "dataset": "aws.vpcflow"
    },
    "destination": {
        "port": 22,
        "address": "2001:db8:1234:a102:3304:8879:34cf:4071",
        "ip": "2001:db8:1234:a102:3304:8879:34cf:4071"
    },
    "source": {
        "address": "2001:db8:1234:a100:8d6e:3477:df66:f105",
        "port": 34892,
        "bytes": 8855,
        "packets": 54,
        "ip": "2001:db8:1234:a100:8d6e:3477:df66:f105"
    },
    "tags": [
        "preserve_original_event"
    ],
    "network": {
        "community_id": "1:hXZclvxUJScaVf0xMIJR6yW6tBQ=",
        "transport": "tcp",
        "type": "ipv6",
        "bytes": 8855,
        "iana_number": "6",
        "packets": 54
    },
    "cloud": {
        "provider": "aws",
        "account": {
            "id": "123456789010"
        }
    },
    "@timestamp": "2016-10-31T11:37:00.000Z",
    "ecs": {
        "version": "8.0.0"
    },
    "related": {
        "ip": [
            "2001:db8:1234:a100:8d6e:3477:df66:f105",
            "2001:db8:1234:a102:3304:8879:34cf:4071"
        ]
    },
    "event": {
        "ingested": "2021-09-28T19:10:43.075027100Z",
        "original": "2 123456789010 eni-1235b8ca123456789 2001:db8:1234:a100:8d6e:3477:df66:f105 2001:db8:1234:a102:3304:8879:34cf:4071 34892 22 6 54 8855 1477913708 1477913820 ACCEPT OK",
        "kind": "event",
        "start": "2016-10-31T11:35:08.000Z",
        "end": "2016-10-31T11:37:00.000Z",
        "type": "connection",
        "category": "network",
        "outcome": "success"
    },
    "aws": {
        "vpcflow": {
            "action": "ACCEPT",
            "account_id": "123456789010",
            "log_status": "OK",
            "interface_id": "eni-1235b8ca123456789",
            "version": "2"
        }
    }
}