Configure direct alerts
AlertBird evaluates supported local observations and sends notifications directly from its worker. Slack, Teams Workflows, SMTP email, PagerDuty and generic webhooks are included. No AlertBird cloud relay is required.
Rules and destinations start disabled. Set exact scopes and thresholds before enabling them. A rule can evaluate with no destination selected; attach a destination when you want outbound notifications.
Mount a destination credential
The installation owner manages credentials outside the app. The app selects
versioned logical references, such as primary-slack@v1, and never accepts raw
provider credentials.
Create a protected secrets.json file with your actual destination configuration.
The following shows its shape; the URL placeholder is not a working destination:
{"secrets":[{"ref":"primary-slack@v1","kind":"slack","config":{"url":"REPLACE_WITH_YOUR_HTTPS_WEBHOOK"}}]}
kubectl -n monitoring create secret generic owner-destinations \
--from-file=secrets.json=/secure/path/to/secrets.json
Add destinationSecret: owner-destinations under core in owner-values.yaml,
then upgrade using the same database and collection profiles as installation.
Verify both API and worker roll out. If changing the contents of an existing
Secret, restart API and worker after the updated projection is available so
configuration and dispatch use the intended mounted map.
Keep old credential versions mounted while active episodes, pending deliveries or planned replays reference them. To rotate a credential, create a new logical version and destination version. Replacing a registered version's contents is rejected. Updating a destination does not silently retarget existing rules or episodes; save a new rule version to bind the new destination version.
| Provider kind | Exact credential config fields |
Delivery boundary |
|---|---|---|
slack |
url |
Incoming webhook acceptance; separate firing/recovery messages. |
teams |
url |
Teams Workflows with Anyone authentication via secret URL. Tenant/user OAuth and retired Office365 connectors are unsupported. |
smtp |
host, port, username, password, from_address, to_addresses, tls_mode |
Verified relay TLS; mode tls (implicit TLS) or starttls. No bundled relay/OAuth. |
pagerduty |
routing_key |
Events v2 trigger/acknowledge/resolve with stable episode deduplication keys. |
webhook |
url, signing_secret |
HTTPS POST with Standard Webhooks signatures. |
SMTP recipient addresses are an array of at most 32. Username/password may be
empty for an authenticated-by-network relay. Generic signing secrets use
whsec_ plus canonical base64 encoding of 32 random bytes. Keep a Teams Workflow
co-owner and check the target channel's support before relying on it.
Test a destination
In Integrations, create and enable a destination using a mounted reference. Use its test action, then inspect delivery activity and the receiving system. Tests queue durable work and perform a real outbound send to your configured provider. A PagerDuty test may create an incident requiring manual closure.
Check both provider acceptance and the intended channel/inbox. SMTP DATA acceptance means the relay accepted the message, not inbox delivery. Partial recipient acceptance is reported by counts; accepted recipients are not automatically resent. A provider accepting a request does not establish human receipt.
Choose a measurement
| Rule | Source and units |
|---|---|
| Error rate | trace_server, HTTP-error percentage from 0 to 100. |
| p95 latency | trace_server, milliseconds from supported histograms. |
| Error burst | span_exception or log_exception, observed exceptions. |
| Metric threshold | Supported scalar native_metric from the catalog. |
| Log count | log_rule, matching arrivals after the matcher is enabled. |
| Pipeline stale | One chosen source, receipt age in seconds. |
Rules require cluster and environment, with optional exact Kubernetes namespace, service namespace and service name. They do not accept arbitrary queries or edge selectors. Log matching supports body or severity text, using equals or contains; there is no historical replay. Missing log arrivals do not invent a zero-match observation.
Numeric rules require eligible coverage, a whole-minute evaluation window, minimum samples and sustained duration. Keep the source fresh while allowing the historical window to close. Sampling, stale data and loss can suspend numeric evaluation; no-data is not measured recovery. See collection.
For an error-rate exercise on a test service, use a 5% threshold, 60-second window, zero sustained duration and 10 minimum samples. Send a known mixture of successes and HTTP 5xx requests above the threshold for complete minutes, while continuing telemetry through the closure delay. Confirm eligible coverage and a firing episode. Then keep sending successful traffic through a fresh closed window and confirm observed recovery and its delivery. Perform this with your own test endpoint; changing a rule or turning it off is administrative closure, not telemetry recovery.
Delivery and failure handling
The worker stores events, leases, attempts, retries and exact configuration
versions in PostgreSQL. UI states distinguish queued work, attempts, scheduled
retry, provider acceptance and terminal failure. Eligible terminal work can be
explicitly replayed without erasing its attempt history. Retries can produce
duplicates after an ambiguous provider outcome; exactly-once notification is
not promised. Generic receivers should verify signatures, check timestamp
tolerance and durably deduplicate webhook-id.
Default egress permits public HTTPS on 443 and public TLS SMTP on 465/587.
Private generic webhooks/SMTP need the owner's explicit exact-host, port and
RFC1918/ULA policy, mounted as core.deliveryPolicyConfigMap with policy.json.
There are no wildcard, redirect or TLS-verification exceptions. See the
transport policy contract
and delivery operations for the schema and retry
limits. Restart the worker after changing egress policy.
Acknowledgement and notification silences do not establish recovery. Editing a rule creates a new version and administratively closes the old evaluation; an outstanding provider incident may still need action. Run an independent dead-man check outside this installation: a stopped API or worker cannot reliably report its own outage.
AlertBird