How We Built an AI-Assisted Sentry Digest to Reduce Alert Fatigue

At Endowus, hundreds of Sentry notifications flow daily into dedicated Slack alert channels. These alerts are essential for reliability and incident response, but keeping up with the volume is a different challenge altogether. We’ve written about our Sentry & Slack alerting setup in a previous post: Solving the Shared Ownership Alerting Challenge.

Our initial setup worked well, but as alert volumes grew, the strain started to show. During a particularly busy period of releases and activity, working closely with our engineering teams as a Tech Program Manager, I could see how much time and attention was being diverted from product development into manually reviewing alerts. Useful signals were present — recurring errors, gradual spikes, subtle correlations — but extracting them required sustained focus across multiple Slack channels and time windows.

Although I’m not a developer, I wanted to help the team solve this. Using AI-assisted development tools, I built an initial working prototype myself, iterating on data extraction, formatting and analysis logic. That first version made the concept tangible: structured interpretation could reduce review effort without disrupting existing monitoring workflows.

Building the prototype also sharpened the problem definition. The alerts themselves were not broken: they were working exactly as intended. What was missing was a structured way to interpret them at scale.

AI as interpreter, not decision-maker

For the prototype, I experimented with using AI to assess alert severity and recommend next actions.

In practice, this proved unreliable.

Severity assessment depends on contextual factors — business impact thresholds, service dependencies, institutional knowledge — that are often implicit rather than documented. While the model could summarise issues accurately, its prioritisation judgements were not consistently trustworthy enough for operational use.

This led to an important design decision: AI’s role would be interpretive, not directive. Its purpose was to organise and contextualise alerts so engineers could evaluate them more efficiently.

AI would not:

• Assign or override severity
• Recommend operational actions
• Suppress alerts
• Replace on-call decision-making

Monitoring and incident response would remain human-led. AI would provide assistance, not outcomes — at least for now.

One digest, every morning

With those boundaries in place, I built a daily digest.

Each weekday morning, engineers receive a consolidated summary of recent Sentry alerts in Slack.

This daily digest:

• Groups alerts by service or error category
• Summarises frequency and recent activity
• Surfaces unusual spikes or recurring patterns
• Flags potential noise candidates for future filtering
• Preserves direct links to underlying Sentry issues

Raw alerts remain unchanged. Monitoring workflows, escalation paths and incident processes are not modified. The digest sits on top of existing signals — less to wade through, but nothing hidden.

Over time, triage became quicker. Engineers could see patterns at a glance rather than piecing them together manually.

How it works

The workflow runs as a scheduled process per Slack channel.

At a high level:

1. Recent Slack messages are retrieved from alert channels.
2. Messages are filtered for Sentry-related content.
3. Issue links and discussion context are extracted.
4. Content is normalised into a predictable input format.
5. The cleaned data is passed to the LLM with strict instructions.
6. A formatted digest is posted back to Slack.

Each step is simple on its own. The care went into how they fit together.

Constraining AI for Reliability

Reliability begins before the LLM is called.

We do not send raw Slack payloads directly to the model. Slack messages are complex objects containing blocks, attachments, formatting markup and thread replies. Rather than passing this through unchanged, we reshape it into a predictable structure.

For each alert, we:

• Filter for Sentry-related messages
• Extract and reconstruct canonical issue URLs
• Flatten Slack blocks and attachments into clean text
• Preserve timestamps and discussion context in a consistent format

By the time the prompt is constructed, the input is already structured, ordered and traceable. The model receives a clean dataset rather than loosely formatted chat content.

The output needs the same care. We give the model a fixed format to work within — set sections, Slack-compatible markdown, a strict brevity limit — and tell it explicitly not to assign severity, declare urgency or recommend actions.

Clean inputs and a bounded output format: that combination is what makes the digest consistent day to day.

What this taught me

This project reinforced three things.

Add, don’t replace

Because the digest layered on top of existing alerts rather than replacing them, engineers could try it without changing anything about how they worked. Adoption happened naturally.

Guardrails build trust

Defining what the system would not do proved as important as defining what it could. Keeping prioritisation and action with engineers reduced friction and made the tool easier to trust.

Structure drives reliability

LLMs are good at making sense of messy data, but they’re more reliable when the data isn’t messy to begin with. Cleaning inputs and bounding outputs cuts down on surprises — and in an operational context, consistency matters more than cleverness.

There’s more to build. The most immediate focus is a lightweight feedback loop within Slack to sharpen digest quality, and pulling in deployment metadata to spot connections between releases and error patterns.

Expanding what’s possible

One thing I didn’t anticipate was how much this project would shift my sense of what’s possible in my role. As a Tech Program Manager, I’ve always worked to connect engineering work to business outcomes — identifying problems, aligning teams, clearing blockers. But actually building a working automation to address one of those problems? That felt firmly out of reach without a developer’s help.

AI-assisted development changed that. Being able to iterate on real, working code meant an idea I’d have normally pitched to a developer and waited on became something I could make tangible myself, quickly enough for the concept to earn its own momentum. That shift, more than any technical outcome in this post, is what I’ll carry forward. If you’re sitting on a problem you’ve always assumed needed an engineer to solve, it might be worth experimenting — the tools have moved further than you think.