Skip to content

Detect and alert

You will learn how metrics, monitors, alerts, and notification channels turn a saved query into something Cordo watches for you.

Query and views answer what is happening right now. Detect answers tell me when something changes — without you re-running a search. Detection is built on the same learned fields and aggregates you already use; it records those signals and evaluates rules against them on a schedule.

The four nouns

Noun What it is
Metric A recorded aggregate — a bounded count/sum/percentile query snapshot that Cordo runs on a schedule and stores as a time series.
Monitor A detection rule over a saved view or a recorded metric. Decides when a signal is in breach.
Alert A fired monitor instance with a lifecycle (firing → resolved) and immutable evidence of why it fired.
Notification channel A reusable destination — webhook, email, Slack-compatible, or PagerDuty — that alerts are delivered to.

A metric feeds monitors and keeps history cheaply; a monitor watches a metric or view; an alert is what a monitor produces when its condition holds; a channel is where that alert is sent.

Metrics

A metric is an aggregate query you have chosen to record. Instead of only running on demand, Cordo evaluates it on a fixed recording interval and stores each result as a point in a series, with optional grouping.

  • Query-time aggregates stay available without recording — a metric is for signals you want to keep and watch.
  • Recorded-metric retention is independent of raw event retention, and is typically longer and cheaper.
  • Series are bounded by cardinality and count caps. When a cap prevents a new series from recording, the reason is visible to the stream owner.

Monitors

A monitor evaluates one source — a saved view or a recorded metric — and decides whether it is in breach. Cordo supports four condition types:

Condition Breaches when
Threshold The value is above / below / equal to a target within the window. A zero-valued aggregate is present data, not absence.
Absence An expected signal stops arriving (no data / no rows) in the window.
Change The value moves too far versus the equivalent prior period (delta or percent).
Baseline The value deviates from a learned seasonal baseline. Baseline detection stays explainable — you see the baseline and the deviation, not just a score.

Window vs frequency

Two intervals govern every monitor:

  • Window — how much time each evaluation looks at (for example, the last 5 minutes of data).
  • Frequency — how often the evaluation runs (for example, every 1 minute).

Eval offset shifts that window behind wall clock so recording and materialization can catch up (default 5m). The form explains the offset as looking N minutes behind wall clock.

For metric sources these use the same interval tokens as recording. See Relative lookbacks.

Grouping

When the source is grouped (for example, by status or route), a monitor can evaluate per group so one noisy series does not mask another. Each group carries its own breach/OK streak, and groups are never merged into a single value.

Flap suppression and cooldown

Monitors are deliberately slow to cry wolf:

  • A first breach is pending, not an alert — it takes two consecutive breaches to open an alert.
  • It takes two consecutive non-breaches to resolve.
  • After a resolve, a cooldown (one frequency) passes before the monitor can re-open, so a flapping signal does not spam you.

A monitor with no attached channel still evaluates and shows alerts in-app — it just has nowhere to deliver them until you attach one.

Alerts

An alert is what a monitor produces when its condition holds through flap suppression. Alerts have a clear lifecycle — firing then resolved — and one open alert per monitor + group at a time.

Every alert is explainable. It links back to:

  • the originating monitor,
  • the evaluated window,
  • the metric or view values that triggered it (including compare / baseline / deviation for change and baseline conditions), and
  • a bounded sample of contributing events.

Fire-time and resolve-time evidence are captured immutably, so the exact observation that opened an alert survives later evaluations.

Monitor history

Absence of an alert does not prove health — a monitor may be disabled, skipped, starved of data, in cooldown, or failing to evaluate. Monitor history is an immutable, seven-day timeline that answers what has this monitor actually been doing?

  • It records state and observed-value samples, lifecycle edges (fired / resolved / enabled / disabled), skipped or not-ready windows, and bounded evaluation errors.
  • Healthy, pending, firing, cooldown, no-data/hold, disabled, and error periods are shown distinctly.
  • Gaps in coverage are surfaced as unknown, never inferred as healthy.
  • You can inspect ranges of 1h, 6h, 24h, 7d, or a custom range; older detail is purged after seven days, and the UI tells you when a request reaches that boundary.

Notification channels

A channel is a reusable delivery destination you attach to one or more monitors:

Type Delivers to
Webhook An HTTPS endpoint (optionally signed)
Email One or more recipients over SMTP
Slack A Slack-compatible incoming webhook URL
PagerDuty PagerDuty Events API (trigger / resolve)

Channel secrets are encrypted at rest and never shown again after you save them. You can test-fire a channel before relying on it, and delivery retries with backoff — a channel surfaces its own success/error health.

What detect is not

  • Not an external-metrics backend — Cordo records metrics from your logs, it does not scrape Prometheus or ingest OTLP metrics.
  • Not a general APM rules engine — detection is scoped to log-derived signals.
  • Not a replacement for Query — you still explore interactively; detect is for the signals worth watching.

Next