Custom API using Common Expression Language

Collect custom events from an API with Elastic agent

Version
1.13.0 (View all)
Compatible Kibana version(s)
8.13.0 or higher
Supported Serverless project types

Security
Observability
Subscription level
Basic
Level of support
Elastic

The Common Expression Language (CEL) custom API input integration is used to ingest data from custom HTTP and local file-system APIs that do not currently have an existing integration.

The input itself supports making both HTTP requests and file-system read operations, and keeping running and persistent state on information from the last collected events. The input performs Common Expression Language expression evaluation with a set of standard extensions to both obtain input data from the API and then process the data into events that are published to Elasticsearch.

Configuration

The full documentation for the input are currently available here.

The most commonly used configuration options are available on the main integration page, while more advanced and customizable options currently resides under the "Advanced options" part of the integration settings page.

Configuration is split into two main parts: Program and State, with an associated named Resource.

The program is a CEL program that will obtain data from the API either by an HTTP request or a local file-system read operation, and then transform the data into a set of events and cursor states. The CEL environment that the program is run in is provided with an HTTP client that has been set-up with the user-specified authentication, proxy, rate limit and other options, and with a set of user-defined regular expressions that are available to use during execution.

The state is passed into the program on start of execution and may contain configuration details not included in the standard set of options. The CEL program return value will be the state used during the next cycle of CEL execution, and will include the events to be published to Elasticsearch and then removed from the state. State values in general will not persist over restarts, but it may contain a cursor that is persisted. The cursor part of state is used when there is a need to keep long-lived state that will persist over restarts.

The named resource is a string value that the CEL program can use to identify the location of the API and will usually either be a URL or a file path. It is included in the state as the state.url. CEL programs are not required to make use of this configuration, but its presence is required in the configuration.

Changelog

VersionDetailsKibana version(s)

1.13.0

Enhancement View pull request
Add xsd configuration option inside input.

8.13.0 or higher

1.12.0

Enhancement View pull request
ECS version updated to 8.11.0. Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.

8.13.0 or higher

1.11.0

Enhancement View pull request
Enable use of Digest Authentication.

8.12.0 or higher

1.10.0

Enhancement View pull request
Enable use of Okta OAuth2.0.

8.12.0 or higher

1.9.0

Enhancement View pull request
Set sensitive values as secret.

8.12.0 or higher

1.8.0

Enhancement View pull request
Clarify naming of input.

8.8.0 or higher

1.7.1

Enhancement View pull request
Changed owners

8.8.0 or higher

1.7.0

Enhancement View pull request
Limit request tracer log count to five.

8.8.0 or higher

1.6.0

Enhancement View pull request
ECS version updated to 8.11.0.

8.8.0 or higher

1.5.0

Enhancement View pull request
ECS version updated to 8.10.0.

8.8.0 or higher

1.4.0

Enhancement View pull request
The format_version in the package manifest changed from 2.11.0 to 3.0.0. Removed dotted YAML keys from package manifest. Added 'owner.type: elastic' to package manifest.

8.8.0 or higher

1.3.0

Enhancement View pull request
Add tags.yml file so that integration's dashboards and saved searches are tagged with "Security Solution" and displayed in the Security Solution UI.

8.8.0 or higher

1.2.1

Bug fix View pull request
Fix location of request trace log destination.

8.8.0 or higher

1.2.0

Enhancement View pull request
Update package to ECS 8.9.0.

8.8.0 or higher

1.1.0

Enhancement View pull request
Make debug log field redactions available.

8.8.0 or higher

1.0.0

Enhancement View pull request
Make package GA.

8.8.0 or higher

0.4.0

Enhancement View pull request
Update package to ECS 8.8.0.

0.3.0

Enhancement View pull request
Update package-spec version to 2.7.0.

0.2.0

Enhancement View pull request
Change resource tracer filename

0.1.1

Bug fix View pull request
Added build manifest to indicate the ECS reference.

0.1.0

Enhancement View pull request
Initial Implementation