What is an Elastic integration?

This integration is powered by Elastic Agent. Elastic Agent is a single, unified way to add monitoring for logs, metrics, and other types of data to a host. It can also protect hosts from security threats, query data from operating systems, forward data from remote services or hardware, and more. Refer to our documentation for a detailed comparison between Beats and Elastic Agent.

Prefer to use Beats for this use case? See Filebeat modules for logs or Metricbeat modules for metrics.

Lyve Cloud is your simple, trusted, and efficient on-demand solution for mass-capacity storage.Lyve Cloud is designed to be compatible with Amazon S3.

Lyve Cloud Log Integration

The Lyve Cloud Log Integration offers users a way to collect logs from Lyve Cloud's audit log bucket

When setting up the Lyve Cloud Integration you will need the target bucket name and the secret credentials to access the bucket. You can then visualize that data in Kibana and reference data when troubleshooting an issue.

Using the s3 API audit log information you can identify which events have occurred, when they have occurred and the user who performed the actions.

Setup

Before adding the integration, you must complete the following tasks in the Lyve Cloud console to read the logs that are available in Lyve Cloud bucket:

  1. Login with an administrator account.
  2. Create a target bucket to save logs.
  3. Enable S3 API audit logs.

Configuration

  1. Click on "Add Lyve Cloud" button on the upper right side of the agent configuration screen to create a policy for an elastic agent.
  2. Turn on the switch for Collecting logs from lyve cloud, under "Change defaults" fill in the reqired information for ingesting the correct logs access key, secret key, bucket name and endpoint .
  3. Give A "New agent policy name", click on "Save and continue" and click on "Add to hosts".
  4. Follow Elastic's instructions to add an agent and you're set to go.

Dashboard and log monitoring

Filter out the Lyve Cloud logs using - data_stream.dataset:"lyve_cloud.audit" when creating new dashboard or in other Analytics search fields inside the filter box.

Exported fields

FieldDescriptionType
@timestamp
Date/time when the event originated. This is the date/time extracted from the event, typically representing when the event was generated by the source. If the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline. Required field for all events.
date
client.as.number
Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet.
long
client.as.organization.name
Organization name.
keyword
client.as.organization.name.text
Multi-field of client.as.organization.name.
match_only_text
client.geo.city_name
City name.
keyword
client.geo.continent_name
Name of the continent.
keyword
client.geo.country_iso_code
Country ISO code.
keyword
client.geo.country_name
Country name.
keyword
client.geo.location.lat
Longitude and latitude.
geo_point
client.geo.location.lon
Longitude and latitude.
geo_point
client.geo.region_iso_code
Region ISO code.
keyword
client.geo.region_name
Region name.
keyword
client.ip
IP address of the client (IPv4 or IPv6).
ip
cloud.account.id
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
keyword
cloud.availability_zone
Availability zone in which this host is running.
keyword
cloud.image.id
Image ID for the cloud instance.
keyword
cloud.instance.id
Instance ID of the host machine.
keyword
cloud.instance.name
Instance name of the host machine.
keyword
cloud.machine.type
Machine type of the host machine.
keyword
cloud.project.id
Name of the project in Google Cloud.
keyword
cloud.provider
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
keyword
cloud.region
Region in which this host is running.
keyword
container.id
Unique container id.
keyword
container.image.name
Name of the image the container was built on.
keyword
container.labels
Image labels.
object
container.name
Container name.
keyword
data_stream.dataset
Data stream dataset.
constant_keyword
data_stream.namespace
Data stream namespace.
constant_keyword
data_stream.type
Data stream type.
constant_keyword
ecs.version
ECS version this event conforms to. ecs.version is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events.
keyword
event.dataset
Event dataset
constant_keyword
event.module
Event module
constant_keyword
host.architecture
Operating system architecture.
keyword
host.containerized
If the host is a container.
boolean
host.domain
Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider.
keyword
host.hostname
Hostname of the host. It normally contains what the hostname command returns on the host machine.
keyword
host.id
Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of beat.name.
keyword
host.ip
Host ip addresses.
ip
host.mac
Host mac addresses.
keyword
host.name
Name of the host. It can contain what hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.
keyword
host.os.build
OS build information.
keyword
host.os.codename
OS codename, if any.
keyword
host.os.family
OS family (such as redhat, debian, freebsd, windows).
keyword
host.os.kernel
Operating system kernel version as a raw string.
keyword
host.os.name
Operating system name, without the version.
keyword
host.os.name.text
Multi-field of host.os.name.
text
host.os.platform
Operating system platform (such centos, ubuntu, windows).
keyword
host.os.version
Operating system version as a raw string.
keyword
host.type
Type of host. For Cloud providers this can be the machine type like t2.medium. If vm, this could be the container, for example, or other information meaningful in your environment.
keyword
http.request.body.bytes
Size in bytes of the request body.
long
http.response.body.bytes
Size in bytes of the response body.
long
http.response.mime_type
Mime type of the body of the response. This value must only be populated based on the content of the response body, not on the Content-Type header. Comparing the mime type of a response with the response's Content-Type header can be helpful in detecting misconfigured servers.
keyword
http.response.status_code
HTTP response status code.
long
input.type
Type of Filebeat input.
keyword
log.file.path
Full path to the log file this event came from, including the file name. It should include the drive letter, when appropriate. If the event wasn't read from a log file, do not populate this field.
keyword
lyve_cloud.audit.auditEntry.api.bucket
Bucket for which the opearion was taken upon.
keyword
lyve_cloud.audit.auditEntry.api.name
Represents name of the operation.
keyword
lyve_cloud.audit.auditEntry.api.object
Objects name
keyword
lyve_cloud.audit.auditEntry.api.status
Represents http status explicitly by string instead of code.
keyword
lyve_cloud.audit.auditEntry.api.timeToFirstByte
Represents time to first packet to arrive in Nano seconds.
long
lyve_cloud.audit.auditEntry.api.timeToResponse
Represents time of the response in Nano seconds.
long
lyve_cloud.audit.auditEntry.requestHeader.X-Forwarded-For
Identifying the originating IP address of a client connecting to a web server through a proxy server.
keyword
lyve_cloud.audit.auditEntry.requestHeader.X-Forwarded-Host
Identifying the original host requested by the client in the Host HTTP request heade
keyword
lyve_cloud.audit.auditEntry.requestHeader.X-Forwarded-Port
helps you identify the destination port that the client used to connect to the load balancer
long
lyve_cloud.audit.auditEntry.requestHeader.X-Real-Ip
Represents http request user's ip.
keyword
lyve_cloud.audit.auditEntry.responseHeader.Accept-Ranges
Marker used by the server to advertise its support for partial requests from the client for file downloads.
keyword
lyve_cloud.audit.auditEntry.responseHeader.Last-Modified
Contains a date and time when the resource was last modified
keyword
lyve_cloud.audit.auditEntry.responseHeader.X-Amz-Bucket-Region
Region of which the operation of the log was taken upon.
keyword
lyve_cloud.audit.auditEntry.responseHeader.X-Amz-Object-Lock-Mode
Object retention mode
keyword
lyve_cloud.audit.auditEntry.responseHeader.X-Amz-Server-Side-Encryption
Identifier for the server-side encryption
keyword
lyve_cloud.audit.auditEntry.responseHeader.object_lock_retain_until_date
Object retention duration
date
lyve_cloud.audit.auditEntry.responseHeader.x-amz-version-id
The version of the object. When versioning is enabled.
keyword
lyve_cloud.audit.auditEntry.version
Represents the current version of Audit Log structure.
keyword
os.name
Operating system name, without the version.
keyword
os.name.text
Multi-field of os.name.
match_only_text
related.hosts
All hostnames or other host identifiers seen on your event. Example identifiers include FQDNs, domain names, workstation names, or aliases.
keyword
related.ip
All of the IPs seen on your event.
ip
related.user
All the user names or other user identifiers seen on the event.
keyword
source.as.number
Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet.
long
source.as.organization.name
Organization name.
keyword
source.as.organization.name.text
Multi-field of source.as.organization.name.
match_only_text
source.geo.city_name
City name.
keyword
source.geo.continent_name
Name of the continent.
keyword
source.geo.country_iso_code
Country ISO code.
keyword
source.geo.country_name
Country name.
keyword
source.geo.location.lat
Longitude and latitude.
geo_point
source.geo.location.lon
Longitude and latitude.
geo_point
source.geo.region_iso_code
Region ISO code.
keyword
source.geo.region_name
Region name.
keyword
source.ip
IP address of the source (IPv4 or IPv6).
ip
tags
List of keywords used to tag each event.
keyword
user.email
User email address.
keyword
user.id
Unique identifier of the user.
keyword
user.name
Short name or login of the user.
keyword
user.name.text
Multi-field of user.name.
match_only_text
user_agent.device.name
Name of the device.
keyword
user_agent.name
Name of the user agent.
keyword
user_agent.original
Unparsed user_agent string.
keyword
user_agent.original.text
Multi-field of user_agent.original.
match_only_text
user_agent.os.full
Operating system name, including the version or code name.
keyword
user_agent.os.full.text
Multi-field of user_agent.os.full.
match_only_text
user_agent.os.name
Operating system name, without the version.
keyword
user_agent.os.name.text
Multi-field of user_agent.os.name.
match_only_text
user_agent.os.version
Operating system version as a raw string.
keyword
user_agent.version
Version of the user agent.
keyword

An example event for audit looks as following:

{
    "@timestamp": "2022-10-20T12:52:42.974Z",
    "cloud": {
        "provider": "lyvecloud"
    },
    "ecs": {
        "version": "8.5.1"
    },
    "event": {
        "original": "{\"auditEntry\": {\"api\": {\"name\": \"GetBucketLocation\", \"bucket\": \"user-name-t10\", \"status\": \"OK\", \"statusCode\": 200, \"timeToResponse\": \"27121602ns\", \"timeToFirstByte\": \"27072750ns\"}, \"time\": \"2022-10-20T12:52:42.974686686Z\", \"version\": \"1\", \"requestID\": \"171FC8111B3F560B\", \"userAgent\": \"MinIO (linux; amd64) minio-go/v7.0.15\", \"deploymentid\": \"8fe8887f-d1e2-4918-9e33-52bfba3b0de8\", \"requestQuery\": {\"location\": \"\"}, \"requestHeader\": {\"X-Real-Ip\": \"10.213.135.144:28911\", \"User-Agent\": \"aws-cli/2.7.7 Python/3.9.11 Linux/5.15.0-52-generic exe/x86_64.ubuntu.20 prompt/off command/s3api.head-object\", \"X-Amz-Date\": \"20221024T083808Z\", \"Authorization\": \"AWS4-HMAC-SHA256 Credential=\u003credacted\u003e/20221024/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=\u003credacted\u003e\", \"Accept-Encoding\": \"identity\", \"X-Forwarded-For\": \"1.128.0.0, 10.213.135.144\", \"X-Forwarded-Host\": \"s3.us-east-1.lyvecloud.seagate.com\", \"X-Forwarded-Proto\": \"https\", \"X-Amz-Content-Sha256\": \"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\"}, \"responseHeader\": {\"ETag\": \"b1946ac92492d2347c6235b4d2611184\", \"Vary\": \"Origin\", \"Content-Type\": \"application/octet-stream\", \"Accept-Ranges\": \"bytes\", \"Last-Modified\": \"Sun, 23 Oct 2022 12:51:23 GMT\", \"Content-Length\": \"6\", \"X-Amz-Request-Id\": \"1720F4788755136D\", \"X-Xss-Protection\": \"1; mode=block\", \"x-amz-version-id\": \"ab44978d-0929-4c3a-8d52-17157c1fb6ad\", \"X-Amz-Bucket-Region\": \"us-east-1\", \"X-Amz-Object-Lock-Mode\": \"COMPLIANCE\", \"Content-Security-Policy\": \"block-all-mixed-content\", \"X-Amz-Server-Side-Encryption\": \"AES256\", \"X-Amz-Object-Lock-Retain-Until-Date\": \"2022-10-27T12:51:23.250Z\"}}, \"serviceAccountName\": \"user-name-terraform\", \"serviceAccountCreatorId\": \"name.last@company.com\"}"
    },
    "http": {
        "response": {
            "body": {
                "bytes": 6
            },
            "mime_type": "application/octet-stream",
            "status_code": 200
        }
    },
    "log": {
        "file": {
            "path": "https://s3.us-east-1.lyvecloud.seagate.com/logss001/October-2022/S3-2022-20-10-14-09-31.gz"
        }
    },
    "lyve_cloud": {
        "audit": {
            "auditEntry": {
                "api": {
                    "bucket": "user-name-t10",
                    "name": "GetBucketLocation",
                    "status": "OK",
                    "timeToFirstByte": 27072750,
                    "timeToResponse": 27121602
                },
                "requestHeader": {
                    "X-Forwarded-For": "1.128.0.0, 10.213.135.144",
                    "X-Forwarded-Host": "s3.us-east-1.lyvecloud.seagate.com",
                    "X-Real-Ip": "10.213.135.144:28911"
                },
                "responseHeader": {
                    "Accept-Ranges": "bytes",
                    "Last-Modified": "Sun, 23 Oct 2022 12:51:23 GMT",
                    "X-Amz-Bucket-Region": "us-east-1",
                    "X-Amz-Object-Lock-Mode": "COMPLIANCE",
                    "X-Amz-Server-Side-Encryption": "AES256",
                    "object_lock_retain_until_date": "2022-10-27T12:51:23.250Z",
                    "x-amz-version-id": "ab44978d-0929-4c3a-8d52-17157c1fb6ad"
                },
                "version": "1"
            }
        }
    },
    "os": {
        "name": "Linux"
    },
    "related": {
        "ip": [
            "1.128.0.0",
            "10.213.135.144"
        ],
        "user": "user-name-terraform"
    },
    "tags": [
        "preserve_original_event"
    ],
    "user": {
        "email": "name.last@company.com",
        "id": "name.last@company.com",
        "name": "user-name-terraform"
    },
    "user_agent": {
        "device": {
            "name": "Other"
        },
        "name": "Other",
        "original": "MinIO (linux; amd64) minio-go/v7.0.15"
    }
}

Changelog

VersionDetails
1.0.1
Enhancement View pull request
Fix changelog link
1.0.0
Enhancement View pull request
Initial implementation