Skip to content

Ingest from systemd / journald

You will learn how to ship systemd service logs from journald to Cordo with OpenTelemetry Collector Contrib.

This path suits bare-metal and VM hosts that already write to the systemd journal. Pattern mining reads journald's unit and syslog identifier as the service, so each unit shows up correctly on Patterns and Findings.

Before you start

  1. Create a stream — Create and manage streams.
  2. Mint a stream API key (lax_…) with the otlp scope (or all scopes) — Manage stream API keys.
  3. Install OpenTelemetry Collector Contrib (otelcol-contrib) on the host. The collector user needs read access to the journal (typically membership of the systemd-journal group).

Use /var/log/journal when the journal is persistent, or /run/log/journal when it is only in memory.

Collector configuration

Save a config such as /etc/otelcol-contrib/cordo-journald.yaml:

receivers:
  journald:
    directory: /var/log/journal
    # No units: list — tail the whole journal and keep matching units below.
    start_at: end
    operators:
      # Join multi-line stack traces into one LogRecord per unit.
      - type: recombine
        combine_field: body.MESSAGE
        is_first_entry: 'body.MESSAGE matches "^\\d{4}-\\d{2}-\\d{2}[ T]\\d{2}:\\d{2}:\\d{2}"'
        source_identifier: body._SYSTEMD_UNIT
        max_unmatched_batch_size: 100

      - type: copy
        from: body._SYSTEMD_UNIT
        to: resource["service.name"]

      - type: copy
        from: body._HOSTNAME
        to: resource["host.name"]

      # Leave the journald map as body so MESSAGE, PRIORITY, and unit fields
      # stay on the event. Do not move body.MESSAGE onto body unless you only
      # need a plain string message (that drops PRIORITY and other fields).

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128

  # Keep only your fleet units by prefix. Change ^myapp- to match your names
  # (for example ^api- or ^checkout-). New matching units are picked up with
  # no collector edit.
  filter/units:
    error_mode: ignore
    logs:
      log_record:
        - 'not IsMatch(resource.attributes["service.name"], "^myapp-")'

  batch:
    timeout: 5s
    send_batch_size: 5000

exporters:
  otlphttp:
    endpoint: https://<your-cordo-host>/v1/otlp
    headers:
      X-API-Key: "lax_<prefix>_<secret>"
    compression: gzip
    sending_queue:
      enabled: true
      num_consumers: 4
      queue_size: 5000
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s

service:
  telemetry:
    metrics:
      # otelcol-contrib ≥0.123. Older builds used metrics.address: 0.0.0.0:8888.
      readers:
        - pull:
            exporter:
              prometheus:
                host: "0.0.0.0"
                port: 8888
  pipelines:
    logs:
      receivers: [journald]
      processors: [memory_limiter, filter/units, batch]
      exporters: [otlphttp]

Run:

otelcol-contrib --config /etc/otelcol-contrib/cordo-journald.yaml

Or install the package's systemd unit and point it at this file.

Logs pipeline only

Cordo accepts OTLP Logs only (POST /v1/otlp/v1/logs). Do not point a hostmetrics, traces, or other non-logs pipeline at the same Cordo otlphttp exporter. Those requests fail; the collector retries and fills its own logs with errors. Send metrics and traces elsewhere if you need them.

How journald fields become Cordo events

The journald receiver puts each journal entry into the LogRecord body as a map (MESSAGE, _SYSTEMD_UNIT, _HOSTNAME, PRIORITY, and the rest). Cordo then stores each LogRecord as one JSON event.

Goal What to do
Set a clear service and host Copy _SYSTEMD_UNIT → resource["service.name"] and _HOSTNAME → resource["host.name"] (as in the config above).
Keep severity and journal fields Leave the journald map as body. Cordo lifts map fields to the top level of the event, so MESSAGE and PRIORITY stay queryable.
Plain string body only Move body.MESSAGE to body. The message still mines, and service.name still comes from the resource — but PRIORITY and other journal fields are dropped from the event.

Fields Cordo uses for patterns

Role Keys (first match wins; capitalisation is ignored for most keys)
Message message, msg, log, body, text, event — including journald's MESSAGE — or the field bound to Message on Fields
Service service.name (top-level or under resource), then service, systemd_unit / _SYSTEMD_UNIT, then SYSLOG_IDENTIFIER
Level severity_text, level, severity, and similar first; otherwise syslog PRIORITY (0–3 → error, 4 → warn, 5–6 → info, 7 → debug)

Copying the unit into service.name and leaving PRIORITY on the event gives the cleanest Patterns rows (service = unit name, level from syslog severity).

Pick up new services automatically

Omit a static units: list on the receiver so the collector tails the whole journal. Keep only the units you care about with the filter processor and a prefix match on service.name.

When you add a new unit that matches the prefix (for example myapp-worker.service), its logs flow without editing the collector config.

If you prefer an explicit allow-list instead, set units: on the journald receiver and drop the filter processor:

receivers:
  journald:
    directory: /var/log/journal
    units:
      - myapp-api.service
      - myapp-worker.service
    start_at: end
    operators:
      # same recombine + copy operators as above

Queueing, retries, and late delivery

The sending_queue and retry_on_failure blocks buffer batches when Cordo is briefly unreachable. Journal timestamps become producer event time on each event.

If the collector later drains a backlog, those events can arrive long after they were produced. Anomaly detection still buckets by receipt time; spikes and new patterns that are mostly late are raised at Info with a late-delivery reason — see Late delivery (replayed or buffered logs).

Verify

  1. Confirm the collector is exporting:
curl -s http://127.0.0.1:8888/metrics | grep -E 'otelcol_exporter_(sent|send_failed)_log_records'

otelcol_exporter_sent_log_records should increase. send_failed should stay at zero after a healthy run.

  1. In Cordo, open the stream's Events page. You should see entries with the unit as the service (and host when _HOSTNAME was present). Trigger a multi-line stack trace from a service; it should appear as one event, not one line per frame.

  2. Open Patterns. Templates should group by message shape, with the systemd unit in the service mix and levels driven by PRIORITY (or severity text).

Next