Notifications¶

Overview¶

The notification system informs users and external systems about events occurring in the platform. It consists of three phases:

Emission: Writing notifications to an outbox table
Relay: Polling the outbox, routing notifications to matching rules, and scheduling workflows
Publishing: Delivering notifications to destination systems via durable workflows

---
title: Notification system overview
---
flowchart LR
    subgraph Emission
        D["Domain logic"]
        O[("NOTIFICATION_OUTBOX")]
    end
    subgraph Relay
        R["Outbox relay"]
        RT["Router"]
    end
    subgraph Publishing
        W["dex engine"]
        P["Publishers"]
        E["External systems"]
    end

    D -->|"INSERT"| O
    R -->|"DELETE RETURNING"| O
    R --> RT
    RT -->|"Schedule workflow"| W
    W --> P
    P --> E

This design follows the transactional outbox pattern, as described in ADR-011. It enables atomic emission of notifications, removing the need for external message brokers in the critical path.

Emission¶

Domain logic emits notifications by inserting records into the NOTIFICATION_OUTBOX table. The insert can happen within the same database transaction that runs the business logic triggering the notification, thus avoiding the dual write problem.

erDiagram
    NOTIFICATION_OUTBOX {
        UUID ID PK
        TIMESTAMPTZ TIMESTAMP
        TEXT SCOPE
        TEXT GROUP
        TEXT LEVEL
        BYTEA PAYLOAD
    }

The ID column uses UUIDv7, which combines global uniqueness with sortability. The PAYLOAD column contains the serialized notification in Protobuf format. The notification schema reference describes the schema.

Preliminary filtering¶

To reduce load on the outbox table, the system only inserts a notification if at least one enabled alert could potentially match it. The INSERT operation runs this check using an EXISTS subquery against the NOTIFICATIONRULE table.

Notifications that have no matching rules drop immediately.

Relay¶

The outbox relay is a background process that continuously polls the NOTIFICATION_OUTBOX table, routes notifications to matching rules, and schedules publishing workflows. The polling interval is configurable via dt.notification.outbox-relay.poll-interval-ms, and defaults to 1 second.

---
title: Relay cycle
---
sequenceDiagram
    participant D as Database
    participant R as Relay
    participant RT as Router
    participant DE as dex engine

    activate R
    R ->> D: Begin TX
    R ->> D: Acquire advisory lock
    D -->> R: Lock acquired
    R ->> D: DELETE FROM NOTIFICATION_OUTBOX<br/>RETURNING PAYLOAD
    D -->> R: Notifications
    R ->> RT: Route notifications
    RT -->> R: Matched rules per notification
    R ->> DE: Schedule workflows (atomic)
    R ->> D: Commit TX
    deactivate R

Concurrency control¶

Transaction-level advisory locks prevent concurrent relay cycles across API server instances. This ensures the relay processes notifications in approximately the order the system emitted them. Batch processing offsets the lack of concurrency. Operators configure the batch size via dt.notification.outbox-relay.batch-size; the default is 100.

Routing¶

The router evaluates each notification against all enabled alerts. A rule matches if:

Its scope matches the notification's scope
Its level is equal to or less verbose than the notification's level
The notification's group is in the rule's configured groups
If the rule limits delivery to specific projects or tags, the notification's subject matches

The router runs in batches, using a single SQL query with UNNEST, reducing database round trips.

Large notification handling¶

Notifications exceeding a configurable size threshold (default: 64KiB) move to file storage instead of passing directly to the durable execution engine. This prevents large payloads (for example, PROJECT_VULN_ANALYSIS_COMPLETED) from bloating the workflow history. The publishing workflow retrieves the notification from file storage and deletes the file upon completion.

Publishing¶

For each notification with at least one matched rule, the relay schedules a Publish Notification workflow. The workflow coordinates delivery for all matched rules.

---
title: Publish notification workflow
---
sequenceDiagram
    participant W as Workflow
    participant A as Activity
    participant D as Database
    participant DE as Destination

    activate W
    par for each matched rule
        W ->> A: Invoke activity
        activate A
        A ->> D: Get rule config
        D -->> A: Publisher, template, config
        A ->> DE: Publish notification
        DE -->> A: Response
        A -->> W: Result
        deactivate A
    end
    deactivate W

A separate activity publishes each rule concurrently, allowing independent retries per rule. If delivery to one destination fails, it does not affect delivery to others.

A workflow succeeds if at least one rule's publishing succeeded. Conversely, if publishing for all rules failed, the entire workflow fails.

Publishers¶

Publishers are implementations that deliver notifications to specific destination types. Built-in publishers include email, Kafka, Slack, Microsoft Teams, Jira, Webhook, and more. Refer to the publishers documentation for details.

Publishers receive:

The notification payload
Rule-specific configuration (for example, destination URL, credentials)
A template renderer for formatting the notification

Retries¶

Activities can signal retryable failures (for example, rate limiting or temporary unavailability). The durable execution engine transparently handles retries with backoff. Non-retryable failures are terminal for a specific rule but do not affect other rules.