Alert Triage using Agent Intelligence Toolkit

This example demonstrates how to build an intelligent alert triage system using AIQ toolkit and LangGraph. The system analyzes system monitoring alerts, performs diagnostic checks using various tools, and generates structured triage reports with root cause categorization. It showcases how to combine LLMs with domain-specific diagnostic tools to create an automated troubleshooting workflow.

Use case description

This example provides an agentic system designed to automate the triage of server-monitoring alerts. The system aims to address several key challenges in alert management:

High alert volume overwhelms security teams and makes timely triage difficult.
Institutional knowledge dependency limits scalability and consistency.
Manual context gathering from scattered systems slows down investigations.
Tedious documentation process make it hard to track or audit triage outcomes.

To solve the problems, the system introduces an event-driven alert triage agent that initiates automated investigations when new alerts are generated by a monitoring platform. Rather than relying on human prompts, the agent autonomously:

Analyzes incoming alerts to identify alert type and affected host
Selects appropriate diagnostic tools from available options:
- Hardware checks via IPMI
- Host performance metrics (CPU, memory)
- Process monitoring status
- Network connectivity tests
- Telemetry metrics analysis
Correlates data from multiple source and iteratively reasons around it to determine root cause
Generates structured reports with:
- Alert summary
- Collected metrics
- Analysis and interpretation
- Recommended actions
- Alert status classification
Categorizes root causes into predefined types like hardware, software, network, etc.

Why use an agentic design?

An agentic design powered by LLMs provides key benefits over traditional rule-based systems:

Handles many alert types: Traditional triage systems break down when alert types grow in number and complexity. Agentic systems adapt on the fly—no need to hard-code every investigation path.
Chooses the right tools dynamically: Based on the alert context, the system can select the most relevant tools and data sources without manual intervention.
Built-in Reporting: Every investigation ends with a natural language summary (with analysis, findings, and next steps), saving time and providing traceability.

How it works

Here's a step-by-step breakdown of the workflow:

1. Alert Received

A new alert is triggered by a monitoring system, containing details like host_id and timestamp
Initiates the investigation process by passing a JSON-formatted alert message

2. Maintenance Check

Before deeper investigation, a Maintenance Check tool queries a maintenance database to see if the alert coincides with scheduled maintenance
If maintenance is ongoing, a summary report is generated explaining the maintenance context
If no maintenance is found, the response NO_ONGOING_MAINTENANCE_STR allows for further agentic investigation

3. Alert Triage Agent

If not under maintenance, the Alert Triage Agent orchestrates the investigation
It analyzes the alert JSON to identify the alert type and affected host
Based on this analysis, it dynamically selects appropriate diagnostic tools

4. Dynamic Tool Invocation

The triage agent may call one or more of the following tools based on the alert context:

Telemetry Metrics Analysis Agent
- Collects and analyzes host-level telemetry data:
  - Host Performance Check: Pulls and analyzes CPU usage patterns
  - Host Heartbeat Check: Monitors host's heartbeat signals
Network Connectivity Check
- Verifies if the host is reachable over the network.
Monitoring Process Check
- Connects to the host to verify monitoring service status (e.g. telegraf)
- Checks if monitoring processes are running as expected
Host Performance Check
- Retrieves system performance metrics like:
  - CPU utilization
  - Memory usage
  - System load
- Analyzes metrics in relation to the alert context
Hardware Check
- Interfaces with IPMI for hardware-level diagnostics
- Monitors environmental metrics:
  - Temperature readings
  - Power status
  - Hardware component health

5. Root Cause Categorization

The agent correlates data gathered from all diagnostic tools
The Categorizer uses LLM reasoning capabilities to determine the most likely root cause
Classifies the issue into predefined categories (see the categorizer prompt):
- software: Malfunctioning or inactive monitoring services
- network_connectivity: Host unreachable or connection issues
- hardware: Hardware failures or degradation
- repetitive_behavior: Recurring patterns like CPU spikes
- false_positive: No clear signs of failure, system appears healthy
- need_investigation: Insufficient information for clear root cause

6. Report Generation

Produces a markdown-formatted report containing:
- Alert details and context
- Maintenance status if applicable
- Results from each diagnostic tool
- Root cause analysis and classification
- Recommended next steps

7. Analyst Review

The final report is presented to an Analyst for review, action, or escalation.

Understanding the config

Functions

Each entry in the functions section defines a tool or sub-agent that can be invoked by the main workflow agent. Tools can operate in test mode, using mocked data for simulation.

Example:

hardware_check:
  _type: hardware_check
  llm_name: tool_reasoning_llm
  test_mode: true

_type: Identifies the name of the tool (matching the names in the tools' python files.)
llm_name: LLM used to support the tool’s reasoning of the raw fetched data.
test_mode: If true, the tool uses predefined mock results for offline testing.

Some entries, like telemetry_metrics_analysis_agent, are sub-agents that coordinate multiple tools:

telemetry_metrics_analysis_agent:
  _type: telemetry_metrics_analysis_agent
  tool_names:
    - telemetry_metrics_host_heartbeat_check
    - telemetry_metrics_host_performance_check
  llm_name: telemetry_metrics_analysis_agent_llm

Workflow

The workflow section defines the primary agent’s execution.

workflow:
  _type: alert_triage_agent
  tool_names:
    - hardware_check
    - ...
  llm_name: ata_agent_llm
  test_mode: true
  test_data_path: ...
  benign_fallback_data_path: ...
  test_output_path: ...

_type: The name of the agent (matching the agent's name in register.py).
tool_names: List of tools (from the functions section) used in the triage process.
llm_name: Main LLM used by the agent for reasoning, tool-calling, and report generation.
test_mode: Enables test execution using predefined input/output instead of real systems.
test_data_path: CSV file containing test alerts and their corresponding mocked tool responses.
benign_fallback_data_path: JSON file with baseline healthy system responses for tools not explicitly mocked.
test_output_path: Output CSV file path where the agent writes triage results. Each processed alert adds a new output column with the generated report.

LLMs

The llms section defines the available LLMs for various parts of the system.

Example:

ata_agent_llm:
  _type: nim
  model_name: meta/llama-3.3-70b-instruct
  temperature: 0.2
  max_tokens: 2048

_type: Backend type (e.g., nim for NVIDIA Inference Microservice).
model_name: LLM mode name.
temperature, top_p, max_tokens: LLM generation parameters (passed directly into the API).

Each tool or agent can use a dedicated LLM tailored for its task.

Installation and setup

If you have not already done so, follow the instructions in the Install Guide to create the development environment and install AIQ toolkit.

Install this workflow

From the root directory of the AIQ toolkit library, run the following commands:

uv pip install -e ./examples/alert_triage_agent

Set up environment variables

As mentioned in the Install Guide, an NVIDIA_API_KEY environment variable is required to run AIQ toolkit.

If you have your key in a .env file, use the following command to load it:

export $(grep -v '^#' .env | xargs)

Example Usage

You can run the agent in test mode or live mode. Test mode allows you to evaluate the agent in a controlled, offline environment using synthetic data. Live mode allows you to run the agent in a real environment.

Running in a live environment

In live mode, each tool used by the triage agent connects to real systems to collect data. These systems can include:

Cloud APIs for retrieving metrics
On-premises endpoints for hardware monitoring
Target hosts accessed via SSH to run diagnostic playbooks to gather system command outputs

To run the agent live, follow these steps:

Configure all tools with real environment details

By default, the agent includes placeholder values for API endpoints, host IP addresses, credentials, and other access parameters. You must:
- Replace these placeholders with the actual values specific to your systems
- Ensure the agent has access permissions to query APIs or connect to hosts
- Test each tool in isolation to confirm it works end-to-end
Add custom tools if needed

If your environment includes unique systems or data sources, you can define new tools or modify existing ones. This allows your triage agent to pull in the most relevant data for your alerts and infrastructure.
Disable test mode

Set test_mode: false in the workflow section and for each tool in the functions section of your config file to ensure the agent uses real data instead of synthetic test datasets.

You can also selectively keep some tools in test mode by leaving their test_mode: true for more granular testing.

Run the agent with a real alert

Provide a live alert in JSON format and invoke the agent using:

aiq run --config_file=examples/alert_triage_agent/configs/config_live_mode.yml --input {your_alert_in_json_format}

This will trigger a full end-to-end triage process using live data sources.

Note on credentials and access

We recommend managing secrets (for example, API keys, SSH keys) using a secure method such as environment variables, secret management tools, or encrypted .env files. Never hard-code sensitive values into the source code.

Running live with a HTTP server listening for alerts

The example includes a Flask-based HTTP server (run.py) that can continuously listen for and process alerts. This allows integration with monitoring systems that send alerts via HTTP POST requests.

To use this mode, first ensure you have configured your live environment as described in the previous section. Then:

Start the Alert Triage Server

From the root directory of the AIQ toolkit library, run:

python examples/alert_triage_agent/src/aiq_alert_triage_agent/run.py \
  --host 0.0.0.0 \
  --port 5000 \
  --env_file examples/alert_triage_agent/.your_custom_env

The server will start and display:

---------------[ Alert Triage HTTP Server ]-----------------
Protocol   : HTTP
Listening  : 0.0.0.0:5000
Env File   : examples/alert_triage_agent/.your_custom_env
Endpoint   : POST /alerts with JSON payload

Send Alerts to the Server

In a separate terminal, you can send alerts using curl. The server accepts both single alerts and arrays of alerts.

Example: Send a single alert:

curl -X POST http://localhost:5000/alerts \
  -H "Content-Type: application/json" \
  -d '{
    "alert_id": 1,
    "alert_name": "InstanceDown",
    "host_id": "test-instance-1.example.com",
    "severity": "critical",
    "description": "Instance test-instance-1.example.com is not available for scrapping for the last 5m. Please check: - instance is up and running; - monitoring service is in place and running; - network connectivity is ok",
    "summary": "Instance test-instance-1.example.com is down",
    "timestamp": "2025-04-28T05:00:00.000000"
  }'

Example: Send multiple alerts:

curl -X POST http://localhost:5000/alerts \
  -H "Content-Type: application/json" \
  -d '[{
    "alert_id": 1,
    "alert_name": "InstanceDown",
    "host_id": "test-instance-1.example.com",
    "severity": "critical",
    "description": "Instance test-instance-1.example.com is not available for scrapping for the last 5m. Please check: - instance is up and running; - monitoring service is in place and running; - network connectivity is ok",
    "summary": "Instance test-instance-1.example.com is down",
    "timestamp": "2025-04-28T05:00:00.000000"
  }, {
    "alert_id": 2,
    "alert_name": "CPUUsageHighError",
    "host_id": "test-instance-2.example.com",
    "severity": "critical",
    "description": "CPU Overall usage on test-instance-2.example.com is high ( current value 100% ). Please check: - trend of cpu usage for all cpus; - running processes for investigate issue; - is there any hardware related issues (e.g. IO bottleneck)",
    "summary": "CPU Usage on test-instance-2.example.com is high (error state)",
    "timestamp": "2025-04-28T06:00:00.000000"
  }]'

Server Response

The server will respond with:
```
{
  "received_alert_count": 2,
  "total_launched": 5
}
```
Where:
- received_alert_count shows the number of alerts received in the latest request
- total_launched shows the cumulative count of all alerts processed
Each alert will trigger an automated triage process.
Monitoring the Process

The server logs will show:
- When alerts are received
- The start of each triage process
- Any errors that occur during processing
You can monitor the progress of the triage process through these logs and the generated reports.

Running in test mode

Test mode lets you evaluate the triage agent in a controlled, offline environment using synthetic data. Instead of calling real systems, the agent uses predefined inputs to simulate alerts and tool outputs, ideal for development, debugging, and tuning.

To run in test mode:

Set required environment variables

Make sure test_mode: true is set in both the workflow section and individual tool sections of your config file (see Understanding the config section).
How it works

The main test CSV provides both alert details and a mock environment. For each alert, expected tool return values are included. These simulate how the environment would behave if the alert occurred on a real system.
The benign fallback dataset fills in tool responses when the agent calls a tool not explicitly defined in the alert's test data. These fallback responses mimic healthy system behavior and help provide the "background scenery" without obscuring the true root cause.

Run the agent in test mode

Run the agent with:
```
aiq run --config_file=examples/alert_triage_agent/configs/config_test_mode.yml --input "test_mode"
```
Note: The --input value is ignored in test mode.

The agent will:
- Load alerts from the test dataset specified in test_data_path in the workflow config
- Simulate an investigation using predefined tool results
- Iterate through all the alerts in the dataset
- Save reports as a new column in a copy of the test CSV file to the path specified in test_output_path in the workflow config
Understanding the output

The output file will contain a new column named output, which includes the markdown report generated by the agent for each data point (i.e., each row in the CSV). Navigate to that rightmost output column to view the report for each test entry.

Sample output snippet:

## Alert Summary
The alert received was for an "InstanceDown" event, indicating that the instance "test-instance-0.example.com" was not available for scraping for the last 5 minutes.

## Collected Metrics
The following metrics were collected:
- Network connectivity check: Successful ping and telnet tests indicated that the host is reachable and the monitoring service is in place and running.
- Monitoring process check: The telegraf service was found to be running and reporting metrics into InfluxDB.
- Hardware check: IPMI output showed that the system's power status is ON, hardware health is normal, and there are no observed anomalies.
- Telemetry metrics analysis: The host is up and running, and CPU usage is within normal limits.

## Analysis
Based on the collected metrics, it appears that the alert was a false positive. The host is currently up and running, and its CPU usage is within normal limits. The network connectivity and monitoring process checks also indicated that the host is reachable and the monitoring service is functioning.

## Recommended Actions
No immediate action is required, as the host is up and running, and the alert appears to be a false positive. However, it is recommended to continue monitoring the host's performance and investigate the cause of the false positive alert to prevent similar incidents in the future.

## Alert Status
The alert status is "False alarm".

## Root Cause Category
false_positive

The alert was categorized as a false positive because all collected metrics indicated the host "test-instance-0.example.com" is up, reachable, and functioning normally, with no signs of hardware or software issues, and the monitoring services are running as expected.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!