Metrics

The storage dataset fetches metrics from Storage in Google Cloud Platform. It contains all metrics exported from the GCP Storage Monitoring API.

You can specify a single region to fetch metrics like us-central1. Be aware that GCP Storage does not use zones so us-central1-a will return nothing. If no region is specified, it will return metrics from all buckets.

Sample Event

An example event for storage looks as following:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "cloud": {
        "account": {
            "id": "elastic-obs-integrations-dev",
            "name": "elastic-obs-integrations-dev"
        },
        "instance": {
            "id": "4751091017865185079",
            "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
        },
        "machine": {
            "type": "e2-medium"
        },
        "provider": "gcp",
        "availability_zone": "us-central1-c",
        "region": "us-central1"
    },
    "event": {
        "dataset": "gcp.storage",
        "duration": 115000,
        "module": "gcp"
    },
    "gcp": {
        "storage": {
            "storage": {
                "total": {
                    "bytes": 4472520191
                }
            },
            "network": {
                "received": {
                    "bytes": 4472520191
                }
            }
        },
        "labels": {
            "user": {
                "goog-gke-node": ""
            }
        }
    },
    "host": {
        "id": "4751091017865185079",
        "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
    },
    "metricset": {
        "name": "storage",
        "period": 10000
    },
    "service": {
        "type": "gcp"
    }
}

Exported fields

Exported fields

Changelog

This change involves project_id, credentials_file and credentials_json variables that are moved from input level configuration to package level configuration (as those variables are reused across all inputs/data streams).

Users with GCP integration enabled will need to input values for these variables again when upgrading the policies to this version.

Enhancement View pull request
Add GCP Billing Data Stream

Enhancement View pull request
Add GCP Compute Data Stream

Enhancement View pull request
Add GCP Firestore Data stream | | 1.10.0 | Enhancement View pull request
Update package to ECS 8.3.0. | | 1.9.2 | Bug fix View pull request
Fix GCP auditlog parsing issue on response status | | 1.9.1 | Enhancement View pull request
Update readme | | 1.9.0 | Enhancement View pull request
Preserve request and response in flattened fields. | | 1.8.0 | Enhancement View pull request
Add missing cloud.provider field. | | 1.7.0 | Enhancement View pull request
Add dashboards for firewall and vpc flow logs.

Bug fix View pull request
Add missing mappings for several event.* fields. | | 1.6.1 | Enhancement View pull request
Clarify the GCP privileges required by the Pub/Sub input. | | 1.6.0 | Enhancement View pull request
Update to ECS 8.2 | | 1.5.1 | Enhancement View pull request
Add documentation for multi-fields | | 1.5.0 | Enhancement View pull request
Improve Google Cloud Platform docs. | | 1.4.2 | Bug fix View pull request
Remove emtpy values, names with only dots, and invalid client IPs. | | 1.4.1 | Bug fix View pull request
Fix quoting of the credentials_json value in policy templates. | | 1.4.0 | Enhancement View pull request
Add gcp.dns integration | | 1.3.1 | Bug fix View pull request
Add Ingest Pipeline script to map IANA Protocol Numbers | | 1.3.0 | Enhancement View pull request
Update to ECS 8.0 | | 1.2.2 | Bug fix View pull request
Regenerate test files using the new GeoIP database | | 1.2.1 | Bug fix View pull request
Change test public IPs to the supported subset | | 1.2.0 | Enhancement View pull request
Add 8.0.0 version constraint | | 1.1.2 | Enhancement View pull request
Update Title and Description. | | 1.1.1 | Bug fix View pull request
Fix logic that checks for the 'forwarded' tag | | 1.1.0 | Enhancement View pull request
Update to ECS 1.12.0 | | 1.0.0 | Enhancement View pull request
Move from experimental to GA

Enhancement View pull request
remove experimental from data_sets | | 0.3.3 | Enhancement View pull request
Convert to generated ECS fields | | 0.3.2 | Enhancement View pull request
update to ECS 1.11.0 | | 0.3.1 | Enhancement View pull request
Escape special characters in docs | | 0.3.0 | Enhancement View pull request
Update integration description | | 0.2.0 | Enhancement View pull request
Set "event.module" and "event.dataset" | | 0.1.0 | Enhancement View pull request
update to ECS 1.10.0 and adding event.original options | | 0.0.2 | Enhancement View pull request
update to ECS 1.9.0 | | 0.0.1 | Enhancement View pull request
initial release |