You are viewing docs on Elastic's new documentation system, currently in technical preview. For all other Elastic docs, visit elastic.co/guide.

Optimizing anomaly results

Learn how to fine-tune and filter anomaly results.

To gain clearer insights into real threats, you can tune the anomaly results. The following procedures help to reduce the number of false positives:

Filter out anomalies from rarely used applications and processes

When anomalies include results from a known process that only runs occasionally, you can filter out the unwanted results.

For example, to filter out results from a housekeeping process, named maintenanceservice.exe, that only executes occasionally you need to:

  1. Create a filter list
  2. Add the filter to the relevant job
  3. Clone and rerun the job (optional)

Create a filter list

  1. Go to Machine learningAnomaly DetectionSettings.

  2. Click Filter Lists and then Create.

    The Create new filter list pane is displayed.

  3. Enter a filter list ID.

  4. Enter a description for the filter list (optional).

  5. Click Add item.

  6. In the Items textbox, enter the name of the process for which you want to filter out anomaly results (maintenanceservice.exe in our example).

  7. Click Add and then Save.

    The new filter appears in the Filter List and can be added to relevant jobs.

Add the filter to the relevant job

  1. Go to Machine learningAnomaly DetectionAnomaly Explorer.

  2. Navigate to the job results for which the filter is required. If the job results are not listed, click Edit job selection and select the relevant job.

  3. In the actions column, click the gear icon and then select Configure rules.

    The Create Rule window is displayed.

  4. Select:

    1. Add a filter list to limit where the rule applies.

    2. The WHEN statement for the relevant detector (process.name in our example).

    3. The IS IN statement.

    4. The filter you created as part of the Create a filter list procedure.

      Tip

      For more information, see Customizing detectors with custom rules.

  5. Click Save.

Note

Changes to rules only affect new results. All anomalies found by the job before the filter was added are still displayed.

Clone and rerun the job

If you want to remove all the previously detected results for the process, you must clone and run the cloned job.

Important

Running the cloned job can take some time. Only run the job after you have completed all job rule changes.

  1. Go to Machine learningAnomaly DetectionJobs.
  2. Navigate to the job for which you configured the rule.
  1. Optionally, expand the job row and click JSON to verify the configured filter appears under custom rules in the JSON code.

  2. In the actions column, click the options menu () and select Clone job.

    The Configure datafeed page is displayed.

  3. Click Data Preview and check the data is displayed without errors.

  1. Click Next until the Job details page is displayed.

  2. Enter a Job ID for the cloned job that indicates it is an iteration of the original one. For example, append a number or a username to the original job name, such as windows-rare-network-process-2.

  3. Click Next and check the job validates without errors. You can ignore warnings about multiple influencers.

  4. Click Next and then Create job.

    The Start <job name> window is displayed.

  5. Select the point of time from which the job will analyze anomalies.

  1. Click Start.

    After a while, results will start to appear on the Anomaly Explorer page.

Define an anomaly threshold for a job

Certain jobs use a high-count function to look for unusual spikes in process events. For some processes, a burst of activity is a normal, such as automation and housekeeping jobs running on server fleets. However, sometimes a high-delta event count is unlikely to be the result of routine behavior. In these cases, you can define a minimum threshold for when a high-event count is considered an anomaly.

Depending on your anomaly detection results, you may want to set a minimum event count threshold for the packetbeat_dns_tunneling job:

  1. Go to Machine learningAnomaly DetectionAnomaly Explorer.

  2. Navigate to the job results for the packetbeat_dns_tunneling job. If the job results are not listed, click Edit job selection and select packetbeat_dns_tunneling.

  3. In the actions column, click the gear icon and then select Configure rules.

    The Create Rule window is displayed.

  4. Select Add numeric conditions for when the rule applies and the following when statement:

    WHEN actual IS GREATER THAN <X>

    Where <X> is the threshold above which anomalies are detected.

  5. Click Save.

  6. To apply the new threshold, rerun the job (Job ManagementActionsStart datafeed).

On this page