I'm looking to integrate with PagerDuty for advanced notification scheduling. While it is possible to simply dispatch an email, I'd prefer to use their API to avoid notification problems from email server issues.
Their API takes JSON-encoded input data - it would be fantastic if the HTTP notification option could be expanded to send JSON post data.
Article Comments
Did some further testing on this.
The server I am sending to does not require Content-type set for JSON, so this should work as-is (but does not).
My notification has HTTP Action enabled with the following:
| URL: | http://events.pagerduty.com/generic/2010-04-15/create_event.json |
| Postdata: | { "service_key": "ServiceKeyDeleted", "incident_key": "prtg/test", "event_type": "trigger", "description": "Test event without placeholders", "details": { "status": "status", "device": "device" } } |
If I telnet to their server directly and post as below, it works.
----------------------Submit by telnet to events.pagerduty.com 80----------------------
POST /generic/2010-04-15/create_event.json HTTP/1.1
Host: events.pagerduty.com
Content-Length: 215
{ "service_key": "ServiceKeyDeleted", "incident_key": "prtg/test", "event_type": "trigger", "description": "Test event without placeholders", "details": { "status": "status", "device": "device" } }
----------------------End----------------------
Can you shed any light on what else PRTG might be adding to the postdata? PRTG logs only show "HTTP/1.1 400 Bad Request" for a test notification. Any way I can get to the data being returned?
Dec, 2011 - Permalink
I think the problem is, that PRTG encodes the POSTDATA and that may break pagerduty's stuff. We will look into creating pagerduty support in one of the next versions.
Jan, 2012 - Permalink
Thanks for the update and for giving this some consideration.
Even a simple option to encode or not encode the postdata on an HTTP notification would be a convenient shorter-term solution.
It's also be helpful to have state change notification options for "Down Acknowledged" and "Paused" (this would let us do our acknowledgments in PRTG and then pass that notification out ).
Thanks again.
Jan, 2012 - Permalink
Would love to have this capability with PagerDuty, we ran into the encode issue too with Leftronic.
Jul, 2012 - Permalink
Hello,
Is there anything new on this subject?
I need to do HTTP Notification with PagerDuty and I have only HTTP/1.1 400 Bad Request logs when i try to test http notification on pagerduty.
Thanks.
Oct, 2012 - Permalink
Any update on "We will look into creating pagerduty support in one of the next versions." ?
Jan, 2013 - Permalink
I'm sorry, but due to a low number of requests it was postponed, but it is still on the list. Please bear with us.
Jan, 2013 - Permalink
I'm a little late to the party, and this isn't exactly a clean way of doing this, but the PagerDuty Nagios API will accept other content-types than JSON. You would just create a Nagios service within PagerDuty.
http://events.pagerduty.com/nagios/2010-04-15/create_event?pd_nagios_object=host&CONTACTPAGER=<pagerduty_api_key>&NOTIFICATIONTYPE=PROBLEM&HOSTNAME=<hostname>&HOSTSTATE=down
Cheers, Ryan
Apr, 2013 - Permalink
We are also very interested in using this feature, and would like to see it implemented as soon as possible.
-Groundspeak IT
Apr, 2013 - Permalink
Hi
I would like this too. I have had a bit of a stab at it and come up with the following powershell v3 script
You need to setup a generic api service in pagerduty and update the script with your personal key
Param(
[string]$Sitename,
[string]$Device,
[string]$Name,
[string]$Status,
[string]$Down,
[string]$message
)
$Description = "$Device $Name $Status $Down"
$Body = @{
"service_key"="<servicekey>";
"event_type"="trigger";
"description"=$Description;
"client"=$Sitename;
"client_url"="<url>";
"details"="$Message"
} | ConvertTo-Json
Invoke-RestMethod -Uri "https://events.pagerduty.com/generic/2010-04-15/create_event.json" -ContentType application/json -Method POST -Body $Body
#$Body | out-file ".\pager.txt"
Now this works flawlessly when running via powershell, when running it via the notification test, the logs come up with "Status sending EXE:Ok".
However I do not get anything at all.
Any ideas on how to debug this ?
Thanks
Oct, 2014 - Permalink
Ok, have modified the script a little bit
The trick seems to be that the PRTG notification setting does not enclose each passed variable in single quotes, so items with spaces in them break the script.
It now looks like (this is what works for us, and yes more can be added, but at present we don't need the extra info, we just need to know something has happened)
Param(
[string]$probe,
[string]$device,
[string]$name,
[string]$status,
[string]$down,
[string]$message
)
$Description = "CRITICAL: $probe $device $name $status $down"
$Body = @{
"service_key"="<apiservicekey>";
"event_type"="trigger";
"description"=$Description;
"client"="$probe";
"client_url"="<url>";
"details"="$message"
} | ConvertTo-Json
Invoke-RestMethod -Uri "https://events.pagerduty.com/generic/2010-04-15/create_event.json" -ContentType application/json -Method POST -Body $Body
#$Body | out-file ".\pager.txt"
In the prtg notification settings the variables under EXE action need to be specified as follows (in the parameter field)
'%probe' '%device' '%name' '%status' '%down' '%message'
Use at your on risk etc
Hopefully this helps someone Thanks
Oct, 2014 - Permalink
PRTG Guys, I don't understand why you don't deliver a simple integration to REST API's it doesn't require that much and they are all the same json based API's
please consider them.
Oct, 2014 - Permalink
Integration with PagerDuty, OpsGenie, and VictorOps native WebAPI calls would be an excellent feature. It feels like a majority of PRTG's competitors have exiting integration with these products. It would be a good sales feature, and also helpful to the pre-existing userbase.
Oct, 2014 - Permalink
If anyone is still looking for a resolution for this, I was able to overcome the headers. I configured a Nagios Service in PagerDuty, then I hit the nagios create_event. Below is how I was able to succesfully send a new trigger and resolve a trigger from PRTG
URL - Make sure it's HTTPS
https://events.pagerduty.com/nagios/2010-04-15/create_event
Postdata to trigger event
NOTIFICATIONTYPE=PROBLEM&pd_nagios_object=service&HOSTNAME=HOSTNAME&SERVICESTATE=DOWN&HOSTADDRESS=HOSTADDRESS&CONTACTPAGER=<YOUR API KEY>&pd_version=1.0&SERVICEDESC=TESTING
Postdata to resolve the event
NOTIFICATIONTYPE=RECOVERY&pd_nagios_object=service&HOSTNAME=HOSTNAME&SERVICESTATE=DOWN&HOSTADDRESS=HOSTADDRESS&CONTACTPAGER=<YOUR API KEY>&pd_version=1.0&SERVICEDESC=TESTING
Dec, 2014 - Permalink
Agreed. Bi-directional VictorOps, OpsGenie, and PagerDuty integration is something other solutions have and we are considering switching platforms for this support.
Jul, 2015 - Permalink
Dear response3
We collect user feedback to help us decide which features would help the most users. We continue to improve our notification emails, however integration with PagerDuty and similar tools is not something we are going for in the foreseeable future.
That does not mean it is off the table forever.
Jul, 2015 - Permalink
OpsGenie has instructions on how to integrate PRTG and OpsGenie: https://www.opsgenie.com/docs/integrations/prtg-integration
Jul, 2016 - Permalink
To expand (or update) on Rhyse posted way above:
I have 2 notifications, a "create pagerduty incident" and "resolve pagerduty incident"
Each of these has a executes a custom script.
The script for "create" is
Param(
[string]$probe,
[string]$device,
[string]$name,
[string]$status,
[string]$down,
[string]$message
)
$Description = "CRITICAL: $probe $device $name $status $down"
$Body = @{
"service_key"="<api key>";
"event_type"="trigger";
"incident_key"=$device;
"description"=$Description;
"client"="$probe";
"client_url"="<url>";
"details"="$message"
} | ConvertTo-Json
Invoke-RestMethod -Uri "https://events.pagerduty.com/generic/2010-04-15/create_event.json" -ContentType application/json -Method POST -Body $Body
#$Body | out-file ".\pager.txt"
and for the "resolve":
Param(
[string]$probe,
[string]$device,
[string]$name,
[string]$status,
[string]$down,
[string]$message
)
$Description = "CRITICAL: $probe $device $name $status $down"
$Body = @{
"service_key"="<api key>";
"event_type"="resolve";
"incident_key"=$device;
"description"=$Description;
"client"="$probe";
"client_url"="<url>";
"details"="$message"
} | ConvertTo-Json
Invoke-RestMethod -Uri "https://events.pagerduty.com/generic/2010-04-15/create_event.json" -ContentType application/json -Method POST -Body $Body
#$Body | out-file ".\pager.txt"
You will note the main difference is I am submitting the incident_key (using the deviceID). This means Pageryduty can resolve the incident if PRTG tells it.
Jan, 2017 - Permalink
I grew tired of the issue of email and the not so great integration into pagerduty, difficult to read and sometimes dodgy on it's functionality. So I wrote a script to handle this and then had to update the script for their new API. You would need to create an API key with PagerDuty then switch the notification type to custom EXE in PRTG and just use a powershell script to call pagerduty. I am sure I left some things defaulted for my environment (I tried to remove all the defining stuff I just hard coded it since it doesn't change). This hopefully will help a few others get this rolling and I have suggested this to pagerduty as the preferred or alternative integration. If the sensor comes back down the script checks to see if alert exists and if it does then it skips saying it already exists unless it's paused or warning, or anything but down really, in which case it closes the ticket. I am sure there are optimizations that could be made it works right now.
Hope this helps.
################# Start Here! #########################################
Param(
[string]$Probe,
[string]$Device,
[string]$Name,
[string]$Status,
[string]$Down,
[string]$DateTime,
[string]$LinkDevice,
[string]$Message
)
$Hour = Get-Date -format "HH:mm"
$text = "$Probe-$Device-$Name"
$Title = $text
$APIKey = "API_Key_20_character"
$servicekey = "service_key_32_characters_string"
$companyname = "subdomain"
$PagerdutyEventURL = "https://events.pagerduty.com/generic/2010-04-15/create_event.json"
$PagerdutyAPIURL = "https://api.pagerduty.com/incidents"
$pagerdutystatus1 = "triggered"
$pagerdutystatus2 = "acknowledged"
$Service = "SERVICE"
$data= "?statuses[]=$pagerdutystatus1`&statuses[]=$pagerdutystatus2`&time_zone=UTC&sort_by:desc&include[]=services:$Service"
$CALL0 = Invoke-WebRequest -Method GET -ContentType "application/vnd.pagerduty+json;version=2" $PagerdutyAPIURL/count$data -Headers @{"Authorization"=("Token token=" + $APIKey)}
$Response0 = $CALL0.content | ConvertFrom-JSON
$CALL1 = Invoke-WebRequest -Method GET -ContentType "application/vnd.pagerduty+json;version=2" $PagerdutyAPIURL$data -Headers @{"Authorization"=("Token token=" + $APIKey)}
$Response1 = $CALL1.content | ConvertFrom-JSON
write-host $Response0.triggered "Triggered and" $Response0.acknowledged "acknowledged"
write-host "Searching for incident_key or title matching ""$Title"""
function createincident
{
$Body= (
New-Object PSObject |
Add-Member -PassThru NoteProperty "service_key" "$servicekey" |
Add-Member -PassThru NoteProperty "event_type" "trigger" |
Add-Member -PassThru NoteProperty "description" "$Title" |
Add-Member -PassThru NoteProperty "incident_key" "$Title" |
Add-Member -PassThru NoteProperty "client" "PRTG" |
Add-Member -PassThru NoteProperty "client_url" "$LinkDevice" |
Add-Member -PassThru NoteProperty details (
New-Object PSObject |
Add-Member -PassThru NoteProperty "Message" "$Message" |
Add-Member -PassThru NoteProperty "Device" "$Device" |
Add-Member -PassThru NoteProperty "Name" "$Name" |
Add-Member -PassThru NoteProperty "Down" "$Down" |
Add-Member -PassThru NoteProperty "DateTime" "$DateTime" |
Add-Member -PassThru NoteProperty "Status" "$Status" |
Add-Member -PassThru NoteProperty "Probe" "$Probe"
) |
Add-Member -PassThru NoteProperty contexts @(
(
New-Object PSObject |
Add-Member -PassThru NoteProperty -name "type" -value "link" |
Add-Member -PassThru NoteProperty -name "href" -value "$LinkDevice"
)
(
New-Object PSObject |
Add-Member -PassThru NoteProperty -name "type" -value "link" |
Add-Member -PassThru NoteProperty -name "href" -value "https://$companyname.pagerduty.com" |
Add-Member -PassThru NoteProperty -name "text" -value "View the incident on PagerDuty"
)
)
) | ConvertTo-JSON -Compress
$CALL2 = Invoke-WebRequest -Method POST -ContentType "application/json" -Body $Body -URI $PagerdutyEventURL
$Response2 = $CALL2.content | ConvertFrom-JSON
write-host $Response2.incident_key "created at $Hour"
}
If ($Response0.triggered -eq 0 -and $Response0.acknowledged -eq 0) {
If ($Status -match "Down" -or $Status -match "Escalation" -and $Status -notmatch "ended") {
write-host "No open or acknowledged alerts! Whoo Hoo... D'oh!"
createincident
}
else
{
write-host "No incidents found resolving impossible"
}
}
else
{
If ($Response1.incidents.incident_key -contains $Title) {
$Incidents = $Response1.incidents
If ($Status -match "Down" -or $Status -match "Escalation" -and $Status -notmatch "ended") {
write-host "PagerDuty Ticket ""$Title"" already exists exiting"
}
else
{
Foreach ($Incident in $Incidents | Where {$_.incident_key -eq $Title}) {
write-host "******************** Start Of Incident Information ********************"
write-host "id = " $Incident.id
write-host "incident_number =" $Incident.incident_number
write-host "incident_key =" $Incident.incident_key
write-host "created_at =" $Incident.created_at
write-host "status =" $Incident.status
write-host "html_url =" $Incident.html_url
write-host "urgency =" $Incident.urgency
write-host "title =" $Incident.title
write-host "description =" $Incident.description
write-host "urgency =" $Incident.urgency
write-host "********************* End Of Incident Information *********************"
$Resolve = (
New-Object PSObject |
Add-Member -PassThru NoteProperty "service_key" "$servicekey" |
Add-Member -PassThru NoteProperty "event_type" "resolve" |
Add-Member -PassThru NoteProperty "description" "$Title" |
Add-Member -PassThru NoteProperty "incident_key" "$Title"
) | ConvertTo-JSON -Compress
write-host "PagerDuty Ticket $Title exists, resolving"
$CallResolve = Invoke-WebRequest -Method POST -ContentType "application/json" -Body $Resolve -URI $PagerdutyEventURL
$ResponseResolve = $CallResolve.content | ConvertFrom-JSON
write-host "Ticket with incident_key"$ResponseResolve.incident_key"attempted ticket closing result:" $ResponseResolve.status
exit
}
}
}
else
{
If ($Status -match "Down" -or $Status -match "Escalation" -and $Status -notmatch "ended") {
write-host "PagerDuty Ticket ""$Title"" does not already exist"
createincident
}
else
{
write-host "PagerDuty Ticket ""$Title"" does not exist and cannot be closed"
}
}
}
################## End Here! ##########################################
Mar, 2017 - Permalink
Another vote for implementing bi-directional acks with PRTG and PagerDuty. The PD-PTRG integration with de-dedupe using a RegEx works OK, but it'd be so much better to only have to ack an incident in one place...
Feb, 2018 - Permalink
Another vote for implementing bi-directional alerts/acks with PagerDuty. It's been almost 10 years, any way this is coming any time soon?
Apr, 2021 - Permalink
Hello,
we will put it onto the wishlist.
best regards.
Dec, 2011 - Permalink