Shadowing on-call engineers feels like watching open-heart surgery over a doctor’s shoulder. Precision and speed are paramount, and mistakes can lead to severe consequences. As part of my, “Welcome-to-PagerDuty” experience, I was added to a PagerDuty engineering team’s on-call rotation as a shadow observer for a week.

During my time on-call, I was on the same escalation policies and received the same notifications as everyone else on the rotation. I would wake up if engineers on-call were woken up by a notification and lurk in the operations Slack channel as they would work through urgent time-sensitive situations. I expected to see stress bring out the worst in people. To my surprise, I was completely wrong.

Phones buzzed to interrupt sleep, meals, thoughts, and so much more — chat channels filled with rapid fire messages and resolution steps unfolded at all hours of the night. I was surprised to see kindness, support, and humor — despite the on-call responsibilities and stress.

Encouragement

Source: https://xkcd.com/1349/

One simple sentence makes it easier to get through hours of stress.

In the late afternoon of my first-day shadowing, I got my first notification. I hopped on the Slack channel and saw that the investigation had already settled on diagnosing an issue from a recent deploy. The team immediately engaged, methodically pausing and provisioning new machines. In the flurry of triaging and diagnosing, I realized that on-call responder was as new to this situation as I was; we had both started less than one month ago. He was fixing a problem that he had never seen before.

In the early evening, things were getting back on track. It had been a long day and people were on their way home. Still, a veteran engineer (P) checked in and cheered on his teammate (K):

P [6:14 PM] Is everything done???

K[6:15 PM] Not yet, just in the middle of restarting the hosts. Everything is going smoothly so far though

P [6:18 PM] Way to go K! This sh*t used to take us half a day!

This type of peer encouragement creates confidence for people new to the project, new to the company, or new to the profession. Encouragement from senior engineers helps less experienced engineers feel brave enough to ask more questions. Asking questions is how new people learn and ramp up to become experts. It starts with empowering people: one “way to go” at a time.

(2) Humility and (3) Gratitude

The newbie on-call engineer in my anecdote immediately responded to the encouragement. He acknowledged and appreciated the senior engineers who had helped him. In three quick sentences, he demonstrated humility and gratitude.

K:[6:21 PM] No way. I would’ve sank if it weren’t for you and M. So much to learn!

I like that K called out another teammate who had helped him. DevOps engineering teams deeply value their teammates. More than just colleagues, they rely on each other to make sure that the software they’re writing today won’t create a hellish on-call rotation tomorrow. No one person can possibly know everything all the time; they have to constantly learn from each other and lean on each other for help.  

In the example above, observe the fine line between humility and self-deprecation. The engineer mentions that he has ‘so much to learn’. This attitude is different from ‘I’ll never do this myself’ or ‘I’ll never understand all of this stuff.’ Being humble and grateful builds trust between teammates. Self-deprecation can sow uncertainty and uneasiness.

Source: https://xkcd.com/1831/

(4) Levity and (5) So many gifs

Source: https://xkcd.com/512/

Being on-call can be repetitive and (blissfully) boring. On-call predictability reduces chaos and stress, but there will always be some inherent anxiety associated with being on-call. Luckily, the Internet granted on-call teams (and the rest of us) gifs to dissipate stressful on-call shifts with delightful bursts of humor on loop.

PagerDuty has a strong culture of humor and a particular passion for custom gifs. I loved observing this team, distributed across three separate time zones and two different countries, have fun and be goofy with each other. I noticed that people who weren’t currently on-call were consistently present and upbeat.

Source: https://xkcd.com/1301/

So, how can you make your team’s on-call experience more positive?

• Be present for your team, even if you’re not currently on-call
• Cheer people on, especially people who are new to your team
• Acknowledge when something is challenging, risky, or time-consuming
• Be specific with your thanks — acknowledge the people who directly impacted you. It will help the team as a whole understand who has which strengths in future situations.
• Be willing to get silly. Everyone loves to laugh.

I saw plenty of problem-solving, some heartache and frustration, and even parts of people’s weekends and evenings absorbed in being on-call. In the midst of it, small gestures from teammates remind you that you’re not alone.

When it feels like the sky is falling, it helps when people are excellent to each other.

For more on how to engender empathy in DevOps, check out our post on #HugOps.