Busk: Created page with "To alert on VM‐level errors and push them into PagerDuty, you’ll use Azure Monitor’s Log Alerts (formerly “Log Search Alerts”) against a Log Analytics workspace,..."

2025-07-08T19:30:05Z

Created page with "To alert on VM‐level errors and push them into PagerDuty, you’ll use Azure Monitor’s **Log Alerts** (formerly “Log Search Alerts”) against a Log Analytics workspace,..."

New page

To alert on VM‐level errors and push them into PagerDuty, you’ll use Azure Monitor’s **Log Alerts** (formerly “Log Search Alerts”) against a Log Analytics workspace, plus an **Action Group** wired to PagerDuty via a webhook. In broad strokes the steps are:

1. **Ingest your VM logs** into a Log Analytics workspace.
2. **Create an Action Group** that calls PagerDuty’s Events API v2 webhook.
3. **Define a log‐query alert rule** that fires when “Error” events appear and attaches the PagerDuty Action Group.

---

### 1. Send VM logs to Log Analytics

Ensure your VM is reporting its event logs (Windows) or syslog (Linux) to a Log Analytics workspace:

* In the **Azure portal**, go to your VM → **Diagnostic settings** → **Add diagnostic setting**.
* Check **Send to Log Analytics** and select (or create) your workspace.
* Under **Log** categories include **Syslog** (for Linux) or **Windows Event Logs** (Application, System, Security).
* Save.

Once configured, you’ll see data flow into the workspace’s **Logs** blade .

---

### 2. Create a PagerDuty–backed Action Group

An **Action Group** bundles one or more notification “receivers.” To route Azure alerts into PagerDuty:

1. In the Azure portal search for **Monitor** → **Alerts** → **Action groups** → **+ Create**.
2. Fill in name/resource group/short name.
3. Under **Actions** click **Add action**:

* **Action Type**: **Webhook**
* **Name**: e.g. `PagerDutyWebhook`
* **URI**: your PagerDuty Events API v2 endpoint, e.g.
`https://events.pagerduty.com/v2/enqueue?integration_key=<YOUR_KEY>` ([learn.microsoft.com][1])
4. Review + Create.

> **Tip:** You obtain that Integration Key by adding a “Custom Events API v2” integration to a PagerDuty Service in the PagerDuty UI ([pagerduty.com][2]).

---

### 3. Define a Log-query alert rule

You can do this in the portal or via CLI/ARM/Terraform. We’ll show both:

---

#### A. Portal approach

1. In **Monitor** → **Alerts** → **+ New alert rule**.

2. **Scope**: click **Select resource**, choose your Log Analytics workspace.

3. **Condition**: click **Add condition** → under **Log**, pick **Custom log search**.

4. In the query editor enter a Kusto query that filters for errors. For example:

```kusto
union
(Syslog | where SeverityLevel >= 3),
(Event | where EventLevelName == "Error")
| where TimeGenerated > ago(5m)
```

5. Set **Alert logic**: “When the number of results is **Greater than** **0**.”

6. **Action group**: click **Select action groups**, pick the one you created with the PagerDuty webhook.

7. Define **Alert rule details** (severity, name, description) and **Review + Create** ([learn.microsoft.com][3]).

---

#### B. Azure CLI example

```bash
# 1. Variables
rg="MyResourceGroup"
ws="MyLogAnalyticsWorkspace"
ag="PagerDutyAG"
key="<YOUR_PAGERDUTY_KEY>"

# 2. Create action group with webhook
az monitor action-group create \
--resource-group $rg \
--name $ag \
--short-name pd \
--action webhook PagerDutyWebhook \
https://events.pagerduty.com/v2/enqueue?integration_key=$key \
--verbose

# 3. Get workspace resource ID
ws_id=$(az monitor log-analytics workspace show \
--resource-group $rg \
--workspace-name $ws \
--query id --output tsv)

# 4. Create scheduled (log) query alert
az monitor scheduled-query create \
--resource-group $rg \
--name "VMErrorAlert" \
--scopes $ws_id \
--description "Alert on VM error logs" \
--severity 2 \
--evaluation-frequency 5m \
--window-size 5m \
--condition "count 'ErrorQuery' > 0" \
--condition-query 'union (Syslog | where SeverityLevel>=3),(Event | where EventLevelName=="Error") | where TimeGenerated>ago(5m)' \
--action "/subscriptions/<subId>/resourceGroups/$rg/providers/microsoft.insights/actionGroups/$ag" \
--verbose
```

> In this CLI snippet, `--condition` references a placeholder name (`'ErrorQuery'`) defined by `--condition-query`, so Azure knows when the alert should fire ([learn.microsoft.com][4]).

---

### Validation & Next Steps

* **Test** by generating an error on the VM (e.g. an event in Windows Event Viewer or a logged `logger "something error"` on Linux). Within 5 minutes you should see an incident in PagerDuty.
* **Tune** your Kusto query to filter only the logs you care about (by computer name, log source, text pattern, etc.).
* **Manage** via IaC: if you use Terraform, you can use [`azurerm_monitor_action_group`](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/monitor_action_group) and [`azurerm_monitor_scheduled_query_rules_alert`](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/monitor_scheduled_query_rules_alert).

That’s it – you now have VM‐level log monitoring in Azure, with real‐time error incidents routed straight into your PagerDuty on-call rotation.

[1]: https://learn.microsoft.com/en-us/azure/service-health/service-health-alert-webhook-pagerduty?utm_source=chatgpt.com "Send Azure service health alerts with PagerDuty using webhooks"
[2]: https://www.pagerduty.com/docs/guides/azure-integration-guide/?utm_source=chatgpt.com "Microsoft Azure Alerts Integration Guide - PagerDuty"
[3]: https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/tutorial-log-alert?utm_source=chatgpt.com "Tutorial - Create a log search alert for an Azure resource"
[4]: https://learn.microsoft.com/en-us/cli/azure/monitor/scheduled-query?view=azure-cli-latest&utm_source=chatgpt.com "az monitor scheduled-query - Learn Microsoft"

Azure monitor logs - Revision history

Busk: Created page with "To alert on VM‐level errors and push them into PagerDuty, you’ll use Azure Monitor’s **Log Alerts** (formerly “Log Search Alerts”) against a Log Analytics workspace,..."

Busk: Created page with "To alert on VM‐level errors and push them into PagerDuty, you’ll use Azure Monitor’s Log Alerts (formerly “Log Search Alerts”) against a Log Analytics workspace,..."