How-to: Light up your @NewRelic Apdex

the super-bright blink(1) mk2

Earlier this month I wrote about my new toy, my blink(1) mk2 from thingm. (There's no financial stake here for me, I just think it's really cool.)

My problem was that I didn't have, well, a really practical use. I already had methods for learning about new mail and I really didn't care to constantly be informed of new Facebook updates, it's not like I'm using Facebook at work anyhow.

But I realized yesterday I did have a good use for it - passive/ambient real-time information about the health of our systems at work.

Several of the members of my team handle On-Call responsibilities. We use NewRelic, KeyNote, WebMetrics integrated with Zapier, PagerDuty, StatusPage.io, Atlassian, etc. for our work.  But for the most part, unless it's a "hard down," an incident must be happening for a period of time before notifications start going off. If we're in the tools, we might see an alert sooner, but otherwise, we're all busy working. No one's sitting around watching monitors.  (We have a desire for more of an NOC but right now it's pretty reactive.)

It occurred to me that since NewRelic offered an API I could use my hacky AppleScript skills (ok, more like awesome Googling skills) and translate NewRelic's red/yellow/green lights for our systems into, literally, red/yellow/green lights that I could see as I worked, even if I wasn't in NewRelic.  Since we have that requirement of sustained impairment before notification, sometimes we hear about something in a drive-by before alarms are ringing. That's embarrassing.

I figured if I could have a closer to real-time look into the system but without spending time in the system that we could have a little bit more of a warning if something was starting to go wibbly-wobbly.

And so with a little work I cobbled together a script that pings the NewRelic API, gets the apdex_score of five sites, displaying a red/yellow/green status.  It shows each one for 5 seconds and then shows a blue light (to help you know the cycle is restarting) and then goes again. It can poll all five systems in about 30 seconds and in the AppleScript log window I have a running history of the apdex_scores.

It's also been a great conversation-starter the past two days, carrying my laptop to meetings and then leaving it sitting there with the laptop mostly closed, the little light quietly pulsing, only stopping to open the laptop and look if something starts to trend down. It's building up credibility for our group as having, literally, our finger on the pulse of the system and it was fun to code it. I'm a little dismayed at a few of the shortcuts I took, but my main job isn't programming, so I didn't want to spend a lot of time on it.

So here's the code, it should be pretty clear where you swap in your api key and your transaction or application IDs. Click the photo of the blink(1) to go and buy one for yourself. (Again, I have no financial interest here, I just think it's cool.)