Squid Proxy

Collect and parse logs from Squid devices with Elastic Agent.

Version
1.0.2 (View all)
Compatible Kibana version(s)
8.14.1 or higher
Supported Serverless project types

Security
Observability
Subscription level
Basic
Level of support
Elastic

Squid is a caching and forwarding HTTP web proxy. Use the Squid Proxy integration to monitor Squid Proxy access logs.

Data streams

This integration supports ingestion of logs from Squid Proxy, via the Filestream, TCP, and/or UDP inputs.

Log is used to retrieve access log messages generated by Squid Proxy. See more details in the documentation here and here.

Requirements

You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.

Elastic Agent must be installed. For more information, refer to the link here.

Installing and managing an Elastic Agent:

You have a few options for installing and managing an Elastic Agent:

With this approach, you install Elastic Agent and use Fleet in Kibana to define, configure, and manage your agents in a central location. We recommend using Fleet management because it makes the management and upgrade of your agents considerably easier.

Install Elastic Agent in standalone mode (advanced users):

With this approach, you install Elastic Agent and manually configure the agent locally on the system where it’s installed. You are responsible for managing and upgrading the agents. This approach is reserved for advanced users only.

Install Elastic Agent in a containerized environment:

You can run Elastic Agent inside a container, either with Fleet Server or standalone. Docker images for all versions of Elastic Agent are available from the Elastic Docker registry, and we provide deployment manifests for running on Kubernetes.

There are some minimum requirements for running Elastic Agent and for more information, refer to the link here.

The minimum Kibana version required is 8.14.1.

Setup

Configure Squid to export access logs using one of the supported methods (file (Module: Standard I/O), udp (Module: UDP Receiver), or tcp (Module: TCP Receiver)).

The integration supports the following formats:

Enabling the integration in Elastic:

  1. In Kibana go to Management > Integrations.
  2. In "Search for integrations" search bar, type Squid Proxy.
  3. Click on the Squid Proxy integration from the search results.
  4. Click on the "Add Squid Proxy" button to add the integration.
  5. Add all the required integration configuration parameters according to the enabled input type.
  6. Click on "Save and continue" to save the integration.

Logs reference

Log

The log dataset collects Squid logs.

Sample Event

An example event for log looks as following:

{
    "@timestamp": "2006-09-08T04:21:52.049Z",
    "agent": {
        "ephemeral_id": "703e0801-aef8-4d26-aa48-12c7673f6df0",
        "id": "29b8ade0-b4ef-4ce2-ab55-0acc99bbb914",
        "name": "elastic-agent-52603",
        "type": "filebeat",
        "version": "8.15.0"
    },
    "data_stream": {
        "dataset": "squid.log",
        "namespace": "63238",
        "type": "logs"
    },
    "destination": {
        "address": "175.16.199.115",
        "bytes": 19763,
        "geo": {
            "city_name": "Changchun",
            "continent_name": "Asia",
            "country_iso_code": "CN",
            "country_name": "China",
            "location": {
                "lat": 43.88,
                "lon": 125.3228
            },
            "region_iso_code": "CN-22",
            "region_name": "Jilin Sheng"
        },
        "ip": "175.16.199.115"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "29b8ade0-b4ef-4ce2-ab55-0acc99bbb914",
        "snapshot": false,
        "version": "8.15.0"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "web"
        ],
        "dataset": "squid.log",
        "duration": 5006000000,
        "ingested": "2024-09-03T18:27:38Z",
        "kind": "event",
        "original": "1157689312.049   5006 10.105.21.199 TCP_MISS/200 19763 CONNECT login.yahoo.com:443 badeyek DIRECT/175.16.199.115 -",
        "outcome": "success",
        "type": [
            "access"
        ]
    },
    "http": {
        "request": {
            "method": "CONNECT"
        }
    },
    "input": {
        "type": "filestream"
    },
    "log": {
        "file": {
            "device_id": "35",
            "inode": "442644",
            "path": "/tmp/service_logs/squid-log-access.log"
        },
        "offset": 0
    },
    "observer": {
        "product": "Squid",
        "type": "proxy",
        "vendor": "Squid"
    },
    "related": {
        "ip": [
            "10.105.21.199",
            "175.16.199.115"
        ],
        "user": [
            "badeyek"
        ]
    },
    "source": {
        "address": "10.105.21.199",
        "ip": "10.105.21.199",
        "user": {
            "name": "badeyek"
        }
    },
    "squid": {
        "peer_status": "DIRECT",
        "result_code": "TCP_MISS",
        "status_code": 200
    },
    "tags": [
        "preserve_original_event",
        "squid-log",
        "forwarded"
    ],
    "url": {
        "original": "login.yahoo.com:443"
    }
}

Exported Fields

Exported fields

FieldDescriptionType
@timestamp
Event timestamp.
date
container.id
Unique container id.
keyword
data_stream.dataset
Data stream dataset.
constant_keyword
data_stream.namespace
Data stream namespace.
constant_keyword
data_stream.type
Data stream type.
constant_keyword
event.dataset
Event dataset
constant_keyword
event.module
Event module
constant_keyword
input.type
Type of Filebeat input.
keyword
log.file.device_id
ID of the device containing the filesystem where the file resides.
keyword
log.file.fingerprint
The sha256 fingerprint identity of the file when fingerprinting is enabled.
keyword
log.file.idxhi
The high-order part of a unique identifier that is associated with a file. (Windows-only)
keyword
log.file.idxlo
The low-order part of a unique identifier that is associated with a file. (Windows-only)
keyword
log.file.inode
Inode number of the log file.
keyword
log.file.path
Full path to the log file this event came from.
keyword
log.file.vol
The serial number of the volume that contains a file. (Windows-only)
keyword
log.flags
Flags for the log file.
keyword
log.offset
Offset of the entry in the log file.
long
log.source.address
Source address from which the log event was read / sent from.
keyword
squid.content_type
The content type as seen in the HTTP reply header.
keyword
squid.peer_status
A code explaining how the request was handled, by forwarding it to a peer or going straight to the source.
keyword
squid.result_code
The outcome of the request.
keyword
squid.status_code
The status of the result.
long

Changelog

VersionDetailsKibana version(s)

1.0.2

Bug fix View pull request
Tolerate errors with uri_parts, ensure _tmp fields are removed in case of errors.

8.14.1 or higher

1.0.1

Enhancement View pull request
Add documentation on supported formats.

8.14.1 or higher

1.0.0

Enhancement View pull request
Add dashboard and improve documentation. Release integration as GA.

8.14.1 or higher

0.21.0

Bug fix View pull request
Enhance mappings and add geoip enrichments.

0.20.0

Enhancement View pull request
Rewrite integration with standard ingest pipelines.

0.19.3

Bug fix View pull request
Add http.request.body.bytes to ECS mappings.

0.19.2

Enhancement View pull request
Changed owners

0.19.1

Bug fix View pull request
Fix exclude_files pattern.

0.19.0

Enhancement View pull request
ECS version updated to 8.11.0.

0.18.0

Enhancement View pull request
ECS version updated to 8.10.0.

0.17.0

Enhancement View pull request
Update package to ECS 8.9.0.

0.16.0

Enhancement View pull request
Ensure event.kind is correctly set for pipeline errors.

0.15.0

Enhancement View pull request
Update package to ECS 8.8.0.

0.14.0

Enhancement View pull request
Update package-spec version to 2.7.0.

0.13.0

Enhancement View pull request
Update package to ECS 8.7.0.

0.12.1

Enhancement View pull request
Added categories and/or subcategories.

0.12.0

Enhancement View pull request
Update package to ECS 8.6.0.

0.11.2

Bug fix View pull request
Update docs to match field definitions.

0.11.1

Bug fix View pull request
Remove duplicate fields.

0.11.0

Enhancement View pull request
Update package to ECS 8.5.0.

0.10.1

Enhancement View pull request
Use ECS geo.location definition.

0.10.0

Enhancement View pull request
Update package to ECS 8.4.0

0.9.0

Enhancement View pull request
Update package to ECS 8.3.0.

0.8.0

Enhancement View pull request
Update to ECS 8.2.0

0.7.0

Enhancement View pull request
Update to ECS 8.0.0

0.6.1

Bug fix View pull request
Regenerate test files using the new GeoIP database

0.6.0

Enhancement View pull request
Add 8.0.0 version constraint

0.5.4

Enhancement View pull request
Uniform with guidelines

0.5.3

Enhancement View pull request
Update Title and Description.

0.5.2

Bug fix View pull request
Fixed a bug that prevents the package from working in 7.16.

0.5.1

Bug fix View pull request
Fix logic that checks for the 'forwarded' tag

0.5.0

Enhancement View pull request
Update to ECS 1.12.0

0.4.3

Bug fix View pull request
Requires version 7.14.1 of the stack

0.4.2

Enhancement View pull request
Convert to generated ECS fields

0.4.1

Enhancement View pull request
update to ECS 1.11.0

0.4.0

Enhancement View pull request
Update integration description

0.3.0

Enhancement View pull request
Set "event.module" and "event.dataset"

0.2.0

Enhancement View pull request
update to ECS 1.10.0 and adding event.original options

0.1.4

Enhancement View pull request
update to ECS 1.9.0

0.1.0

Enhancement View pull request
initial release