New

The executive guide to generative AI

Read more

Cassandra Integration

edit

Cassandra Integration

edit

Version

1.16.0 (View all)

Compatible Kibana version(s)

8.13.0 or higher

Supported Serverless project types
What’s this?

Security
Observability

Subscription level
What’s this?

Basic

Level of support
What’s this?

Elastic

This integration periodically fetches metrics from Cassandra using jolokia agent. It can parse System logs.

Compatibility

edit

This integration has been tested against Cassandra version 3.11.11.

Steps to Setup Jolokia

edit

Prerequisites

edit
  • Java Development Kit (JDK) 1.8 or later
  • Apache Cassandra 3.x or 4.x (depending on user’s version)
  • Jolokia agent JAR file

Jolokia Setup

edit

Follow these steps to set up Jolokia for monitoring Apache Cassandra:

  1. Download the Jolokia JVM Agent:

    Visit the Jolokia official download page to obtain the latest version of the Jolokia JVM agent JAR file. Download the jolokia-jvm-<jolokia_version>-agent.jar file.

  2. Copy the Jolokia Agent to Cassandra’s Library Directory:

    Copy the downloaded jolokia-jvm-<jolokia_version>-agent.jar file to the Cassandra library directory on the server where Cassandra is installed.

    For example:

    cp jolokia-jvm-<jolokia_version>-agent.jar /path/to/cassandra/lib/

    Replace /path/to/cassandra/lib/ with the actual path to Cassandra’s library directory.

  3. Configure Cassandra to use the Jolokia Agent:

    Open the cassandra-env.sh file, located in the Cassandra configuration directory, using a text editor, and add the following line at the bottom of the file:

    JVM_OPTS="$JVM_OPTS -javaagent:/path/to/jolokia-jvm-<jolokia_version>-agent.jar=port=<jolokia_port>,host=0.0.0.0"

    Replace /path/to/jolokia-jvm-<version>-agent.jar with the actual path to the Jolokia agent JAR file copied in Step 2. Save the changes and close the cassandra-env.sh file.

  4. Restart Cassandra:

    Restart the Apache Cassandra service to apply the changes made to the configuration.

    Note:

    • Restarting the Apache Cassandra service will temporarily disrupt database connectivity. Ensure that dependent services are designed to handle such interruptions gracefully.
    • Immediately after a restart, Cassandra’s performance may be impacted due to cold caches and commit log replay. Allow some time for the system to stabilize.
    • Before restarting Cassandra, ensure that no cluster maintenance tasks are in progress to prevent any unintended consequences.
    • The exact steps will vary based on the installation type, the setup process might differ based on the specific deployment method or environment.
    • Procedures for restarting Cassandra may vary based on user’s specific setup and configuration.

Verifying the setup

edit

After restarting Cassandra, user can verify that Jolokia is properly set up by accessing the Jolokia endpoint:

http://<cassandra-host>:<jolokia_port>/jolokia

Replace with the hostname or IP address of user’s Cassandra server.

If the setup is successful, user should see a JSON response containing information about the available Jolokia operations and the Cassandra instance.

User can now use Jolokia to monitor and manage Apache Cassandra cluster.

Troubleshooting

edit
  • If log.flags appears conflicted under the logs-* data view, this issue can be resolved by reindexing the Logs data stream.

Logs

edit

Cassandra system logs from cassandra.log files.

Example

An example event for log looks as following:

{
    "@timestamp": "2024-06-18T06:33:32.952Z",
    "agent": {
        "ephemeral_id": "b1e9fa09-5c73-45d9-b26f-184761635dd9",
        "id": "97400795-188c-4140-a1ee-0002078c785d",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.13.0"
    },
    "data_stream": {
        "dataset": "cassandra.log",
        "namespace": "ep",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "97400795-188c-4140-a1ee-0002078c785d",
        "snapshot": false,
        "version": "8.13.0"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "database"
        ],
        "dataset": "cassandra.log",
        "ingested": "2024-06-18T06:34:02Z",
        "kind": "event",
        "module": "cassandra",
        "type": [
            "info"
        ]
    },
    "input": {
        "type": "log"
    },
    "log": {
        "file": {
            "path": "/tmp/service_logs/cassandra/system.log"
        },
        "level": "INFO",
        "offset": 0,
        "origin": {
            "file": {
                "line": 92,
                "name": "YamlConfigurationLoader.java"
            }
        }
    },
    "message": "Configuration location: file:/etc/cassandra/cassandra.yaml",
    "process": {
        "thread": {
            "name": "main"
        }
    },
    "tags": [
        "forwarded",
        "cassandra-systemlogs"
    ]
}

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields
Field Description Type

@timestamp

Event timestamp.

date

cassandra.log.meta

Log meta infos like java stack_trace.

keyword

data_stream.dataset

Data stream dataset.

constant_keyword

data_stream.namespace

Data stream namespace.

constant_keyword

data_stream.type

Data stream type.

constant_keyword

input.type

Type of Filebeat input.

keyword

log.flags

Flags for the log file.

keyword

log.offset

Offset of the entry in the log file.

long

Metrics

edit

Cassandra metrics using jolokia agent installed on cassandra.

Example

An example event for metrics looks as following:

{
    "@timestamp": "2024-06-18T06:35:01.265Z",
    "agent": {
        "ephemeral_id": "51e65675-8699-4d2e-8c14-ecde813096e9",
        "id": "97400795-188c-4140-a1ee-0002078c785d",
        "name": "docker-fleet-agent",
        "type": "metricbeat",
        "version": "8.13.0"
    },
    "cassandra": {
        "metrics": {
            "cache": {
                "key_cache": {
                    "capacity": 104857600,
                    "one_minute_hit_rate": 0,
                    "requests": {
                        "one_minute_rate": 12
                    }
                },
                "row_cache": {
                    "capacity": 0,
                    "requests": {
                        "one_minute_rate": 0
                    }
                }
            },
            "client": {
                "connected_native_clients": 0
            },
            "client_request": {
                "casread": {
                    "one_minute_rate": 0
                },
                "caswrite": {
                    "one_minute_rate": 0
                },
                "range_slice": {
                    "one_minute_rate": 0,
                    "total_latency": 0
                },
                "read": {
                    "count": 0,
                    "one_minute_rate": 0,
                    "timeouts": 0,
                    "timeoutsms": 0,
                    "total_latency": 0,
                    "unavailables": 0,
                    "unavailablesms": 0
                },
                "write": {
                    "count": 0,
                    "one_minute_rate": 0,
                    "timeouts": 0,
                    "timeoutsms": 0,
                    "total_latency": 0,
                    "unavailables": 0,
                    "unavailablesms": 0
                }
            },
            "column_family": {
                "total_disk_space_used": 72566
            },
            "compaction": {
                "completed": 44,
                "pending": 0
            },
            "dropped_message": {
                "batch_remove": 0,
                "batch_store": 0,
                "counter_mutation": 0,
                "hint": 0,
                "mutation": 0,
                "paged_range": 0,
                "range_slice": 0,
                "read": 0,
                "read_repair": 0,
                "request_response": 0,
                "trace": 0
            },
            "gc": {
                "concurrent_mark_sweep": {
                    "collection_count": 1,
                    "collection_time": 26
                },
                "par_new": {
                    "collection_count": 1,
                    "collection_time": 29
                }
            },
            "memory": {
                "heap_usage": {
                    "committed": 4054777856,
                    "init": 4158652416,
                    "max": 4054777856,
                    "used": 481894272
                },
                "other_usage": {
                    "committed": 62337024,
                    "init": 2555904,
                    "max": -1,
                    "used": 60729840
                }
            },
            "storage": {
                "exceptions": 0,
                "load": 72566,
                "total_hint_in_progress": 0,
                "total_hints": 0
            },
            "system": {
                "cluster": "Test Cluster",
                "data_center": "datacenter1",
                "live_nodes": "192.168.247.2",
                "rack": "rack1",
                "version": "3.11.11"
            },
            "table": {
                "all_memtables_heap_size": 4584,
                "all_memtables_off_heap_size": 0,
                "live_disk_space_used": 72566,
                "live_ss_table_count": 11
            },
            "task": {
                "complete": 55,
                "pending": 0,
                "total_commitlog_size": 67108864
            },
            "thread_pools": {
                "counter_mutation_stage": {
                    "request": {
                        "active": 0,
                        "pending": 0
                    }
                },
                "mutation_stage": {
                    "request": {
                        "active": 0,
                        "pending": 0
                    }
                },
                "read_repair_stage": {
                    "request": {
                        "active": 0,
                        "pending": 0
                    }
                },
                "read_stage": {
                    "request": {
                        "active": 0,
                        "pending": 0
                    }
                },
                "request_response_stage": {
                    "request": {
                        "active": 0,
                        "pending": 0
                    }
                }
            }
        }
    },
    "data_stream": {
        "dataset": "cassandra.metrics",
        "namespace": "ep",
        "type": "metrics"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "97400795-188c-4140-a1ee-0002078c785d",
        "snapshot": false,
        "version": "8.13.0"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "database"
        ],
        "created": "2024-06-18T06:35:01.265Z",
        "dataset": "cassandra.metrics",
        "duration": 110507236,
        "ingested": "2024-06-18T06:35:13Z",
        "kind": "event",
        "module": "cassandra",
        "type": [
            "info"
        ]
    },
    "host": {
        "architecture": "x86_64",
        "containerized": true,
        "hostname": "docker-fleet-agent",
        "id": "8259e024976a406e8a54cdbffeb84fec",
        "ip": [
            "192.168.245.7"
        ],
        "mac": [
            "02-42-C0-A8-F5-07"
        ],
        "name": "docker-fleet-agent",
        "os": {
            "codename": "focal",
            "family": "debian",
            "kernel": "3.10.0-1160.102.1.el7.x86_64",
            "name": "Ubuntu",
            "platform": "ubuntu",
            "type": "linux",
            "version": "20.04.6 LTS (Focal Fossa)"
        }
    },
    "metricset": {
        "name": "jmx",
        "period": 10000
    },
    "service": {
        "address": "http://elastic-package-service-cassandra-1:8778/jolokia/%3FignoreErrors=true&canonicalNaming=false",
        "type": "jolokia"
    }
}

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields
Field Description Type Metric Type

@timestamp

Event timestamp.

date

agent.id

Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id.

keyword

cassandra.metrics.cache.key_cache.capacity

long

gauge

cassandra.metrics.cache.key_cache.one_minute_hit_rate

long

gauge

cassandra.metrics.cache.key_cache.requests.one_minute_rate

long

gauge

cassandra.metrics.cache.row_cache.capacity

long

gauge

cassandra.metrics.cache.row_cache.one_minute_hit_rate

long

gauge

cassandra.metrics.cache.row_cache.requests.one_minute_rate

long

gauge

cassandra.metrics.client.connected_native_clients

long

gauge

cassandra.metrics.client_request.casread.one_minute_rate

double

gauge

cassandra.metrics.client_request.caswrite.one_minute_rate

double

gauge

cassandra.metrics.client_request.range_slice.one_minute_rate

double

gauge

cassandra.metrics.client_request.range_slice.total_latency

double

counter

cassandra.metrics.client_request.read.count

long

counter

cassandra.metrics.client_request.read.one_minute_rate

double

gauge

cassandra.metrics.client_request.read.timeouts

Number of read timeouts encountered.

double

counter

cassandra.metrics.client_request.read.timeoutsms

double

gauge

cassandra.metrics.client_request.read.total_latency

double

counter

cassandra.metrics.client_request.read.unavailables

Number of read unavailables encountered.

double

counter

cassandra.metrics.client_request.read.unavailablesms

double

gauge

cassandra.metrics.client_request.write.count

long

counter

cassandra.metrics.client_request.write.one_minute_rate

double

gauge

cassandra.metrics.client_request.write.timeouts

double

counter

cassandra.metrics.client_request.write.timeoutsms

double

gauge

cassandra.metrics.client_request.write.total_latency

double

counter

cassandra.metrics.client_request.write.unavailables

double

counter

cassandra.metrics.client_request.write.unavailablesms

double

gauge

cassandra.metrics.column_family.total_disk_space_used

long

gauge

cassandra.metrics.compaction.completed

compaction completed tasks.

long

gauge

cassandra.metrics.compaction.pending

compaction pending tasks.

long

gauge

cassandra.metrics.dropped_message.batch_remove

long

counter

cassandra.metrics.dropped_message.batch_store

long

counter

cassandra.metrics.dropped_message.counter_mutation

long

counter

cassandra.metrics.dropped_message.hint

long

counter

cassandra.metrics.dropped_message.mutation

long

counter

cassandra.metrics.dropped_message.paged_range

long

counter

cassandra.metrics.dropped_message.range_slice

long

counter

cassandra.metrics.dropped_message.read

long

counter

cassandra.metrics.dropped_message.read_repair

long

counter

cassandra.metrics.dropped_message.request_response

long

counter

cassandra.metrics.dropped_message.trace

long

counter

cassandra.metrics.gc.concurrent_mark_sweep.collection_count

Total number of CMS collections that have occurred.

long

gauge

cassandra.metrics.gc.concurrent_mark_sweep.collection_time

Approximate accumulated CMS collection elapsed time in milliseconds.

long

gauge

cassandra.metrics.gc.par_new.collection_count

Total number of ParNew collections that have occurred.

long

gauge

cassandra.metrics.gc.par_new.collection_time

Approximate accumulated ParNew collection elapsed time in milliseconds.

long

gauge

cassandra.metrics.memory.heap_usage.committed

Committed heap memory usage.

long

gauge

cassandra.metrics.memory.heap_usage.init

Initial heap memory usage.

long

gauge

cassandra.metrics.memory.heap_usage.max

Max heap memory usage.

long

gauge

cassandra.metrics.memory.heap_usage.used

Used heap memory usage.

long

gauge

cassandra.metrics.memory.other_usage.committed

Committed non-heap memory usage.

long

gauge

cassandra.metrics.memory.other_usage.init

Initial non-heap memory usage.

long

gauge

cassandra.metrics.memory.other_usage.max

Max non-heap memory usage.

long

gauge

cassandra.metrics.memory.other_usage.used

Used non-heap memory usage.

long

gauge

cassandra.metrics.storage.exceptions

The number of the total exceptions.

long

counter

cassandra.metrics.storage.load

Storage used for Cassandra data in bytes.

long

counter

cassandra.metrics.storage.total_hint_in_progress

The number of the total hits in progress.

long

counter

cassandra.metrics.storage.total_hints

The number of the total hits.

long

counter

cassandra.metrics.system.cluster

keyword

cassandra.metrics.system.data_center

keyword

cassandra.metrics.system.joining_nodes

keyword

cassandra.metrics.system.leaving_nodes

keyword

cassandra.metrics.system.live_nodes

keyword

cassandra.metrics.system.moving_nodes

keyword

cassandra.metrics.system.rack

keyword

cassandra.metrics.system.unreachable_nodes

keyword

cassandra.metrics.system.version

keyword

cassandra.metrics.table.all_memtables_heap_size

long

gauge

cassandra.metrics.table.all_memtables_off_heap_size

long

gauge

cassandra.metrics.table.live_disk_space_used

long

counter

cassandra.metrics.table.live_ss_table_count

long

gauge

cassandra.metrics.task.complete

completed tasks.

long

gauge

cassandra.metrics.task.pending

pending tasks.

long

gauge

cassandra.metrics.task.total_commitlog_size

total commitlog size of tasks.

long

gauge

cassandra.metrics.thread_pools.counter_mutation_stage.request.active

long

gauge

cassandra.metrics.thread_pools.counter_mutation_stage.request.pending

long

gauge

cassandra.metrics.thread_pools.mutation_stage.request.active

long

gauge

cassandra.metrics.thread_pools.mutation_stage.request.pending

long

gauge

cassandra.metrics.thread_pools.read_repair_stage.request.active

long

gauge

cassandra.metrics.thread_pools.read_repair_stage.request.pending

long

gauge

cassandra.metrics.thread_pools.read_stage.request.active

long

gauge

cassandra.metrics.thread_pools.read_stage.request.pending

long

gauge

cassandra.metrics.thread_pools.request_response_stage.request.active

long

gauge

cassandra.metrics.thread_pools.request_response_stage.request.pending

long

gauge

cloud.account.id

The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.

keyword

cloud.availability_zone

Availability zone in which this host, resource, or service is located.

keyword

cloud.instance.id

Instance ID of the host machine.

keyword

cloud.provider

Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.

keyword

cloud.region

Region in which this host, resource, or service is located.

keyword

container.id

Unique container id.

keyword

data_stream.dataset

Data stream dataset.

constant_keyword

data_stream.namespace

Data stream namespace.

constant_keyword

data_stream.type

Data stream type.

constant_keyword

host.name

Name of the host. It can contain what hostname returns on Unix systems, the fully qualified domain name (FQDN), or a name specified by the user. The recommended value is the lowercase FQDN of the host.

keyword

service.address

Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets).

keyword

Changelog

edit
Changelog
Version Details Kibana version(s)

1.16.0

Enhancement (View pull request)
Allow @custom pipeline access to event.original without setting preserve_original_event.

8.13.0 or higher

1.15.0

Enhancement (View pull request)
Add processor support for metrics data stream.

8.13.0 or higher

1.14.0

Enhancement (View pull request)
ECS version updated to 8.11.0. Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.

8.13.0 or higher

1.13.0

Enhancement (View pull request)
Add global filter on data_stream.dataset to improve performance.

8.12.0 or higher

1.12.2

Enhancement (View pull request)
Add documentation for setting up Jolokia to monitor Apache Cassandra.

8.12.0 or higher

1.12.1

Enhancement (View pull request)
Update README with reindexing steps.

8.12.0 or higher

1.12.0

Enhancement (View pull request)
Enable secrets for sensitive fields. For more details, refer https://www.elastic.co/guide/en/fleet/current/agent-policy.html#agent-policy-secret-values

8.12.0 or higher

1.11.1

Bug fix (View pull request)
Disable secrets for older stack versions due to errors.

8.8.0 or higher

1.11.0

Enhancement (View pull request)
Enable secret for the sensitive fields, supported from 8.12.

8.8.0 or higher

1.10.1

Bug fix (View pull request)
Update the link to the correct reindexing procedure.

8.8.0 or higher

1.10.0

Enhancement (View pull request)
Update the package format_version to 3.0.0.

8.8.0 or higher

1.9.2

Bug fix (View pull request)
Fix the type for log.flags field.

8.8.0 or higher

1.9.1

Bug fix (View pull request)
Add null check and ignore_missing check to the rename processor

8.8.0 or higher

1.9.0

Enhancement (View pull request)
Enable time series data streams for the metrics datasets. This dramatically reduces storage for metrics and is expected to progressively improve query performance. For more details, see https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html.

8.8.0 or higher

1.8.1

Enhancement (View pull request)
Add metric_type mapping for the fields of metrics datastream.

8.3.0 or higher

1.8.0

Enhancement (View pull request)
Add dimension fields for metrics datastream for TSDB enablement.

8.3.0 or higher

1.7.0

Enhancement (View pull request)
Rename ownership from obs-service-integrations to obs-infraobs-integrations

8.3.0 or higher

1.6.0

Enhancement (View pull request)
Migrate System Logs dashboard visualizations to lens.

8.3.0 or higher

1.5.0

Enhancement (View pull request)
Migrate Overview dashboard visualizations to lens.

8.3.0 or higher

1.4.1

Enhancement (View pull request)
Added categories and/or subcategories.

7.15.0 or higher
8.0.0 or higher

1.4.0

Enhancement (View pull request)
Update ECS version to 8.5.1

7.15.0 or higher
8.0.0 or higher

1.3.0

Enhancement (View pull request)
Update cassandra package as per best practices.

7.15.0 or higher
8.0.0 or higher

1.2.3

Bug fix (View pull request)
Fix dashboard issues.

7.15.0 or higher
8.0.0 or higher

1.2.2

Bug fix (View pull request)
Fix typo in config template for ignoring host enrichment

1.2.1

Enhancement (View pull request)
Add documentation for multi-fields

1.2.0

Enhancement (View pull request)
Update to ECS 8.0

1.1.0

Enhancement (View pull request)
Release cassandra package for v8.0.0

7.15.0 or higher
8.0.0 or higher

1.0.0

Enhancement (View pull request)
GA Release

0.0.1

Enhancement (View pull request)
Initial draft of the package

Was this helpful?
Feedback