Detect and alert
You will learn how metrics, monitors, alerts, and notification channels turn a saved query into something Cordo watches for you.
Query and views answer what is happening right now. Detect answers tell me when something changes — without you re-running a search. Detection is built on the same learned fields and aggregates you already use; it records those signals and evaluates rules against them on a schedule.
The four nouns
| Noun | What it is |
|---|---|
| Metric | A recorded aggregate — a bounded count/sum/percentile query snapshot that Cordo runs on a schedule and stores as a time series. |
| Monitor | A detection rule over a saved view or a recorded metric. Decides when a signal is in breach. |
| Alert | A fired monitor instance with a lifecycle (firing → resolved) and immutable evidence of why it fired. |
| Notification channel | A reusable destination — webhook, email, Slack-compatible, or PagerDuty — that alerts are delivered to. |
A metric feeds monitors and keeps history cheaply; a monitor watches a metric or view; an alert is what a monitor produces when its condition holds; a channel is where that alert is sent.
Metrics
A metric is an aggregate query you have chosen to record. Instead of only running on demand, Cordo evaluates it on a fixed recording interval and stores each result as a point in a series, with optional grouping.
- Query-time aggregates stay available without recording — a metric is for signals you want to keep and watch.
- Recorded-metric retention is independent of raw event retention, and is typically longer and cheaper.
- Series are bounded by cardinality and count caps. When a cap prevents a new series from recording, the reason is visible to the stream owner.
Monitors
A monitor evaluates one source — a saved view or a recorded metric — and decides whether it is in breach. Cordo supports four condition types:
| Condition | Breaches when |
|---|---|
| Threshold | The value is above / below / equal to a target within the window. A zero-valued aggregate is present data, not absence. |
| Absence | An expected signal stops arriving (no data / no rows) in the window. |
| Change | The value moves too far versus the equivalent prior period (delta or percent). |
| Baseline | The value deviates from a learned seasonal baseline. Baseline detection stays explainable — you see the baseline and the deviation, not just a score. |
Window vs frequency
Two intervals govern every monitor:
- Window — how much time each evaluation looks at (for example, the last 5 minutes of data).
- Frequency — how often the evaluation runs (for example, every 1 minute).
Eval offset shifts that window behind wall clock so recording and materialization can catch up (default 5m). The form explains the offset as looking N minutes behind wall clock.
For metric sources these use the same interval tokens as recording. See Relative lookbacks.
Grouping
When the source is grouped (for example, by status or route), a monitor can
evaluate per group so one noisy series does not mask another. Each group
carries its own breach/OK streak, and groups are never merged into a single
value.
Flap suppression and cooldown
Monitors are deliberately slow to cry wolf:
- A first breach is pending, not an alert — it takes two consecutive breaches to open an alert.
- It takes two consecutive non-breaches to resolve.
- After a resolve, a cooldown (one frequency) passes before the monitor can re-open, so a flapping signal does not spam you.
A monitor with no attached channel still evaluates and shows alerts in-app — it just has nowhere to deliver them until you attach one.
Alerts
An alert is what a monitor produces when its condition holds through flap suppression. Alerts have a clear lifecycle — firing then resolved — and one open alert per monitor + group at a time.
Every alert is explainable. It links back to:
- the originating monitor,
- the evaluated window,
- the metric or view values that triggered it (including compare / baseline / deviation for change and baseline conditions), and
- a bounded sample of contributing events.
Fire-time and resolve-time evidence are captured immutably, so the exact observation that opened an alert survives later evaluations.
Monitor history
Absence of an alert does not prove health — a monitor may be disabled, skipped, starved of data, in cooldown, or failing to evaluate. Monitor history is an immutable, seven-day timeline that answers what has this monitor actually been doing?
- It records state and observed-value samples, lifecycle edges (fired / resolved / enabled / disabled), skipped or not-ready windows, and bounded evaluation errors.
- Healthy, pending, firing, cooldown, no-data/hold, disabled, and error periods are shown distinctly.
- Gaps in coverage are surfaced as unknown, never inferred as healthy.
- You can inspect ranges of 1h, 6h, 24h, 7d, or a custom range; older detail is purged after seven days, and the UI tells you when a request reaches that boundary.
Notification channels
A channel is a reusable delivery destination you attach to one or more monitors:
| Type | Delivers to |
|---|---|
| Webhook | An HTTPS endpoint (optionally signed) |
| One or more recipients over SMTP | |
| Slack | A Slack-compatible incoming webhook URL |
| PagerDuty | PagerDuty Events API (trigger / resolve) |
Channel secrets are encrypted at rest and never shown again after you save them. You can test-fire a channel before relying on it, and delivery retries with backoff — a channel surfaces its own success/error health.
What detect is not
- Not an external-metrics backend — Cordo records metrics from your logs, it does not scrape Prometheus or ingest OTLP metrics.
- Not a general APM rules engine — detection is scoped to log-derived signals.
- Not a replacement for Query — you still explore interactively; detect is for the signals worth watching.
Next
- Hands-on: Detect and alert
- Query model behind metrics: Query and aggregates
- Saved sources for monitors: Views
- Return to Concepts