Streams and ingest

You will learn how streams receive and store events.

A stream is the named destination for every ingest path. Each stream owns its field catalog, shapes, views, ingest API keys, and retention settings.

What a stream holds

Piece	Role
Raw events	Durable JSON payloads for the stream
Field catalog	Learned paths, types, and samples
Shapes	Recurring structural fingerprints
Analytics projection	Fast query/aggregate store (may lag ingest briefly)
Views	Saved query configurations
API keys	Secrets used only for ingest

Usage appears on Streams and Events as event counts and storage sizes (raw and analytics). Retention and optional size caps control how long data stays.

How events get in

All production methods share one pipeline after accept:

Ingest → observe (fields + shapes) → materialize (analytics)

Method	Typical use	Auth
Native HTTP JSON	Apps, scripts, custom collectors	Stream API key (`X-API-Key`)
Native HTTP plain text	Unstructured one-line-per-event logs	Stream API key (`X-API-Key`)
OpenSearch-compatible `_bulk`	DigitalOcean App Platform and similar	HTTP Basic (password = API key)
OTLP Logs (HTTP)	OpenTelemetry Collector / exporters	Stream API key (`X-API-Key` or Bearer)
Vector / Fluent Bit	Sidecar / DaemonSet collectors	Same as native, OpenSearch, or OTLP
MCP `ingest_events`	Agents writing sample events	Personal access token

Traces and metrics are out of scope — Cordo accepts OTLP Logs only.

For step-by-step setup of each method, see Ingest your logs.

What you wait for

After…	What is ready
A successful ingest response	Events are durable; Events and live tails can show them immediately
Observe	Fields and Insights shapes update as paths and structures are learned
Materialize	Query, aggregates, and field stats over the analytics store catch up

If Query looks empty for a few seconds after curl, refresh — materialization is asynchronous. Query and Views distinguish lag from a true empty match, and from a truncated page.

Schema learning limits

Field and shape catalogs have soft and hard caps. Streams shows Schema learning (Fields and Insights banner the same state). A hard cap skips new paths or fingerprints; ingest still stores the event. This is not a type conflict.

Ingest refusals

When ingest is rejected (storage cap, rate or byte budget, oversized request, or plaintext-on-JSON-only path), the stream records an Ingest refused reason and count for owners. The payload is not stored.

Retention and size

When you create or edit a stream:

Retention — how long events are kept (UI presets such as 3 days, 7 days, 30 days, 90 days, 1 year). Default is typically 30 days.
Max stored bytes — optional size safety valve. Empty means no size cap. When set, oldest events are removed first after time-based retention.

A background worker enforces retention; expired data is removed from both raw storage and the analytics projection.

Parse user agents

Optional stream setting Parse user agents. When enabled, ingest detects common user-agent field names and adds a sibling {field}_parsed object with browser, OS, and device details (for example user_agent → user_agent_parsed.browser_family).

Leave this off unless you need those derived fields.

Redact sensitive fields

Optional stream setting Redact sensitive fields on ingest. When enabled, ingest rewrites event payloads before they are stored (Postgres and analytics):

Preset denylist paths such as password, token, authorization, and similar (same rules as catalog sample redaction)
Values that look like secrets (API keys, JWTs, etc.)
Extra dotted paths you configure on the stream (for example headers.cookie)

String values become [REDACTED]; other types become null. Sibling _raw / _cordo_raw fields are cleared when nested children are redacted.

This is irreversible for events already written — turn it on before sending secrets. Sample-only redaction (Fields / Insights) does not protect stored payloads; use this setting for that.

REST create/update expose redact_sensitive and redact_fields. Preview a sample without storing it via POST /v1/streams/{id}/redact/preview.