This is the fourth part in a series on shifting our thoughts about Monitoring. All of this came from a talk I gave at GalaxZ’17 in Austin, TX earlier this year. You can find part one, “Ein Minuten Bitte”, part 2, “Empathetic Monitoring”, and part 3, “New Partners” from their respective links. But if you’re ready to read on, we’ll talk about…

Reformation

That’s a heavy word. But I want to be clear about how much work goes into this process. There’s risk involved here, too. Change, being introduced the wrong way, can cause schism instead of reform. You’ll notice in the 16th century the Protestant Reformation didn’t create a new and improved Roman Catholic Church, but rather a whole assortment of new Protestant sects from Lutheran to Baptist to Pentecostal.

(more…)

This is part three in a series about monitoring all coming from a talk that I gave in Austin, TX at GalaxZ’17. In part one I introduced the problem and in part two I discussed how we should start to change. Today we’re looking at who, in our organization are our…

New Partners

Hopefully at this point you’re on board with me. And maybe you’re thinking “Sure, sounds good, but where do I start? Who do I even talk to?” And that’s an awesome place to be, you’re eager to move forward and that energy is going to be needed for really encouraging organizational change.

(more…)

This is part two in a series about monitoring stemming from a talk that I gave in Austin, TX at GalaxZ’17. In part one I initially brought up the point of what I think is terrible in the current world of monitoring and why I think it should be changed. Here we’ll look more closely at one way how to do that. Today we talk about…

Empathetic Monitoring

Where we’re at right now, we’re tracking our System Metrics. Hopefully many of us are monitoring more than just the actual box health and are tracking things like User Latency, Database Performance and page load times, but it’s still fight with our businesses to see the value of the data. We need to start monitoring Business Metrics. When you tell an organization that you want to start monitoring the vital signs they care about it’s a much easier sell.

(more…)

This will be the first in a series of blog entries from me about changing how you think about monitoring in your tech business. All of this comes from the talk I gave at GalaxZ’17 this year in Austin, TX. I’ll setup the general monitoring situation as it is, and present to you my thoughts about what needs to change. In further parts of this series I’ll actually get in a little further about how to make these changes. And now keep reading for Part 1, What sucks and why:

(more…)

Some of the blog readers may know that I (Aaron) have been doing a bit of traveling lately. Some of the conferences I have been attending fall under the DevOpsDays heading. Recently, I was at DevOpsDays Toronto and earlier this year at DevOpsDays Seattle. I've gone to so many (and applied to speak at more) mostly because I love these events. What's awesome, what really makes them stand out, is that they're not just another lineup of Big Names pitching their brand of How To Do DevOps Correctly, but that the organizers push for local and locally relevant content that speaks to their communities directly.

(more…)

I’ve been inaugurated with my first DevOps Days. It was a fantastic, exhausting weekend where I probably gained more perspective on the DevOps community in 2 days than most of my working career.

What’s a DevOps Day anyway?

If you’re not familiar with the concept of the DevOps Days conference, it’s sort of like an industry conference injected with extreme participation. The attendance is kept intentionally low (although we were at about 425 attendees) and a major focus of the event is the Open Space meeting. Even the Keynote Speakers are encouraged to be new voices in the community over the existing and previous-years’ speakers. This leads for an awesome overall message of “your voice matters.” The only downside is that you meet a ton of people and do a lot of brain-stretching in a short period of time; my introverted side was totally spent by Saturday morning.

Technology can’t fix a people problem

My first major takeaway from the convention was this: People are important. If you find yourself “Rubbing DevOps on it” but the rash just won’t go away, it’s not a technology problem, it’s a culture problem. I could repeat this over and over, but helping people is why I fell in love with technology. The primary motivating factor behind leaving my previous jobs has been a culture issue. It was never that I didn’t love the work, it was that I hated the way the organization’s culture permeated their workforce. Culture matters. Your people matter. If you focus on your people, your product will reflect that.

Automate your 💩

Secondly, it’s all about technical debt. This was a new term for me, personally, but the concept isn’t new. The idea focuses mainly on opportunity costs for any task. Every time you cut a corner, there’s work that you’re not doing that needs to be completed. This work can manifest suddenly as unplanned work when multiple band-aid fixes inevitably go wrong, or when something breaks and there’s not enough documentation and the on-call tech has to troubleshoot the entire issue from scratch. Except that tech doesn’t even have information about what the system is supposed to do. So it was crystalized like this for me:

1. If you can automate it, automate it. If a human doesn’t have to be involved, why make the work for them? This task is important because it frees up man-hours for more important work.
2. If you can’t automate it, you need documentation. For incidents, Minimum-Viable Runbooks are crucial. There should be no guessing what systems do and how to fix them at 3am.
3. Any task that improved quality of life for your employees is critical. Getting paged after hours interrupts sleep and reduces quality of life. Removing unplanned work takes priority even if that means your planned feature release gets delayed.

DevOps CT: Challenge Accepted

All in all this was an awesome conference that greatly encouraged me to build a similar community in Connecticut. Dave and I have already begun by planning our first DevOps CT Meetup. So if you are interested in getting connected with the DevOps community, please sign up and join us in conversation. I would love to talk about all the pie-in-the-sky ideals and discuss all of the realistic challenges that we face daily when trying to affect real culture change in CT. It’s a huge challenge to take on major monolithic organizations like the insurance giants in Hartford, but I leave it to Barney Stinson to share my thoughts on this: “Challenge Accepted.”

This was a guest post written for the VictorOps Blog. To help make your on-call suck less, check out VictorOps

In the world of IT Operations, there is no escaping the Dreaded On-Call. Someone has to keep a look out at night to make sure the business continues to run and we’re not all going to get a nasty surprise come the morning.

(more…)