We recently completed our third round of annual security training at PagerDuty. We run two sessions: One for all employees, where we discuss things such as social engineering, password management, and data handling; and another for engineering teams, where we discuss common vulnerabilities and how to exploit and mitigate them.

These are both training courses that we developed in-house and delivered ourselves, and I wanted to tell the story of why we decided to take a different approach to training than most, as well as how it’s worked out for us—both the good parts and the bad.

I’m also very happy to announce that we’ve open-sourced our security training for use by the community. If you’d rather skip the story behind it and jump straight in, you can view it at https://sudo.pagerduty.com.

The Problem With Security Training

Throughout my career, I’ve participated in many types of mandatory security training. This usually means logging in to a website and watching several unskippable videos for over two hours, which more often than not involve contrived and cringeworthy role-playing scenarios. You’re then forced to answer a set of multiple-choice questions where you get an unlimited number of attempts. I’ve yet to meet anyone who actually fully paid attention to these training approaches, and worse, I’ve yet to meet anyone who has come away from them feeling like their time was well spent and that they’d learned anything useful.

I can understand why these types of training are common throughout the industry though: They require little upfront investment and allow companies to track participation (since you may have compliance requirements where you need to track and audit that the training was taken). But if no one pays attention to the training, answers can be brute forced, and everyone sees it as a chore they can ignore, what’s the point in having it at all? Ticking a compliance checkbox isn’t much use if your staff aren’t well trained against common security threats.

These types of training have been a long-standing pet peeve of mine. I vowed that if I were ever in a position where I could do something different, I would. At PagerDuty I had just that opportunity—and took full advantage of it.

Training Philosophy

The first task was to decide how I wanted to approach the training. What were my goals? What have I seen fail in the past? How could I try to improve upon that? These would be my guiding principles when developing the material. I eventually decided on these four:

1. Teach the why, not just the what.
2. Don’t shy away from technical details.
3. Make it accessible for any skill level.
4. It’s OK to be funny.

Teach the Why, Not Just the What

I’ve always found it aggravating when I’m given a rule or instruction to follow, but not given any context as to why I should be following it. When developing our security training program, I wanted to be sure to spend time talking about why certain rules exist, rather than just reciting them verbatim and forcing everyone to follow them.

For example, if I were to tell everyone that they should be using a password manager, and that all of their passwords should be long, random, and unique, I’m likely to get blank stares. Folks will nod their heads, and then immediately ignore the advice since it’s much easier just to keep using the same password they always had. While we might be able to check the little box that says all employees took the training, no real benefit comes from it.

So in our approach, I chose first to show people why good password management is important. I showed real-world examples of how passwords are broken, how easy it is to “crack” stolen passwords, and just how much harder you can make it on an attacker by having a better password.

I took an example stolen database and worked through them to break the passwords in real time.

We started breaking passwords by just guessing based on the information we already had and eventually worked our way up to more complicated methods. In the end, all of the passwords were broken.

I was initially hesitant to include such a section, fearing it would turn non-technical people off. I couldn’t have been more wrong. It was the most engaging section of the entire training. Showing people just how easily passwords can be broken was eye-opening for many employees. The reveal at the end that this is exactly how passwords have been stored in some past breaches was the turning point for a lot of people in the audience. This section alone caused a lot of our employees to change their passwords habits, which I’ll talk more about later.

Likewise, there was a section on phishing attempts. I showed real phishing attempts that have been made against PagerDuty. Some of them were easy to spot, others not so much. We played a short game of “Reel or Phish” (I’ll get to puns later), where the audience took a shot at guessing whether an actual email is genuine or a phishing attempt. I’m usually not a fan of audience participation in presentations, so I made sure that it wouldn’t be an awkward silence if no one said anything, but people really had a go at it and shouting out whether they thought examples were phishing attempts or not.

Of course, I also showed the red flags for how to spot such phishing attempts so that everyone got to learn something about how to spot them. But using real-world examples, and showing why things are important, is one of the key tenets of our training.

Don’t Shy Away From Technical Details

In order to show people how passwords are broken, we needed to discuss password hashing. The mere mention of the word “hashing” is probably enough to make non-technical employees’ eyes gloss over. So instead I just call it “Magic”. Because truth be told, you don’t need to know any of the technical terminology to understand the concepts. We can keep things simple and easily understandable by everyone, rather than scaring people away with technical words.

That said, I didn’t want to mislead people. So we chose to be clear to them that there is a technical term; it’s just not going to be important for the rest of the content. There’s a fine line here, since you don’t want to patronise people. Again it comes down to explaining the “Why”, rather than just the “What”.

Concepts such as rainbow tables can then be explained without having to refer to the actual name; we can just demonstrate that you can create a lookup and call it a “magic list”. Using language that everyone can understand makes sure that the content is accessible to as much of the audience as possible, while still teaching important technical concepts.

Make It Accessible for Any Skill Level

I wanted the content to be accessible to everyone, regardless of their level of expertise in any particular field. This meant giving enough information for those who may be well versed in the content already, but with plain English descriptions for those who might not be.

I tried to do this by building up concepts piece by piece to make sure everyone was on the same page. For example, with Multi-Factor Authentication, I could have just said there are the three factors, “Knowledge”, “Possession”, and “Inherence”, and left it at that. But most people aren’t going to understand those terms. So instead I put it in plain English and added some examples to make it even clearer. Just like with “hashing” earlier, the technical terms are still there for those who are more familiar with them.

It’s OK to Be Funny

Security is a serious topic, but listening to someone talk seriously for an hour is going to make the best of people fatigued. I am a big fan of injecting humour whenever I can. Sometimes the serious times are when you need to laugh the most. Now, I’m no master comical genius (as my family will confirm), and your training doesn’t need to be a stand up-routine. But drop the occasional funny picture or little message in the slides for those paying attention. Getting a laugh from the audience at a terrible pun you’ve made keeps people engaged and makes it more fun for everyone.

I would encourage keeping it contextual though. Don’t just throw in a funny picture of a cat with no context. Make it relevant to the content. For example, here’s a serious slide from the training. It’s showing a phishing attempt where we’ve highlighted some of the indicators. The important information is all there: We show what’s wrong with the email and that it’s a phishing attempt. But there’s a nice little pun for those who are paying attention.

Likewise, in the lead-up the training date, I made some funny posters and stuck them up around our offices. They serve a serious purpose, reminding everyone that security training is coming up, but they also keep the mood light and let folks have a laugh too, with lots of little jokes for those who are paying attention.

What Did It End Up Looking Like?

The presentation lasted 1 hour and 15 minutes, with 15 minutes at the end for Q&A.

Five main topics were covered:

1. Social Engineering: Primarily phishing, but other types of social engineering were also mentioned.
2. Passwords: A crash course in how passwords are cracked, and why it’s important to have strong passwords. Also covered are password managers and how they can help.
3. Physical Security: How to keep PagerDuty offices and equipment secure, and how to report suspicious activity.
4. Data Handling: The types of data we handle, and how to handle that data securely.
5. Compliance: Descriptions of various compliance regulations and how they affect us.

Obviously, there are many more topics we could have covered, but we chose these as we felt they were most relevant to all employees. There’s no such thing as 100% security, and likewise, there’s no such thing as security training that covers everything. We’re constantly evolving our training material and will likely be changing the topics in future years, depending on which threats we feel are most important to cover.

What Was the Feedback?

At the end of the training, attendees were invited to provide optional feedback as part of a survey. Two questions were asked, and there was a field for any additional comments. We had over 300 responses to the survey, which has provided us with some valuable data.

The first question was, “Did you enjoy the training?”, with a rating from 1 to 5; 1 being the worst and 5 being the best. Over 98% of responders gave a rating of 3 or higher, and the average rating was 4.54 out of 5. (It’s worth noting that I didn’t bribe or social engineer anyone to get these ratings).

The second question was, “How relevant or helpful do you think it was for your job?”, with the same rating scale as the previous question. Over 99% of responders gave a rating of 3 or higher, and the average rating was 4.56 out of 5.

What Went Well?

Both of these results are extremely encouraging and show how valuable and enjoyable the vast majority of our staff found the training.

There were some other interesting things that have happened, thanks to the training, which weren’t necessarily captured in the feedback, but that I wanted to call out:

• 30+ employees have commented on how they’ve now spent time switching to a password manager. Not just for their employee accounts, but for their personal accounts too!
• 20+ employees requested an open-source or sanitised version of the slides so they can share with family (more on this in a moment).
• Security awareness appears to have gone up throughout the company.

The biggest win from my point of view was the fact that multiple employees switched to a password manager. This is huge, because we never told people they had to use a password manager! We simply showed how passwords are broken and why we think password managers help make things safer. Employees made the decision to switch on their own, and did so with their personal accounts too.

More questions are being asked to the security team. We now get asked questions about anything, from how to make sure a phone call is legitimate to how to properly secure their Facebook account. It’s not just about PagerDuty security, but how to protect personal accounts too. While I know at previous jobs, this probably would’ve been considered “out of scope” and answers would not have been provided, we love answering these questions on our team. It always starts a good discussion around online protections, builds trust with others in the company, and fosters a great community.

I firmly believe that security teams shouldn’t be the ones who just say “No” to everything. The motto of our team at PagerDuty is: “Make it easy to do the right thing”. We’re not here to get in everyone’s way. We’re here to make the “right” way (i.e., the most secure way) the easiest path, so that folks will naturally use that path.

What Didn’t Go So Well?

It wasn’t all great news though. As you can see from the charts earlier, some rated the training very poorly. We’d be remiss to just pat ourselves on the back and not investigate that data further. Digging into the negative comments in the feedback, we can categorize them into two buckets:

1. The training was a repeat of the previous two years, which meant this was the third time some of our employees had seen the same material.
2. The training was too long.

It’s hard to argue with either of these points. Thanks to the feedback provided, we’re looking to change our course for next year and offer a much shorter “refresher” course, with new material, for those who’ve already attended this training. It’s our hope that this addresses the main areas of concern from this year’s training.

Another area that didn’t go so well was with the preparation. Oh jeez, did I underestimate just how long it would take to create the training. Creating the training didn’t take a few days. It was a few weeks of effort to get the material together and produce the content in such a way that it would be engaging. A lot of effort went into producing the slides—far more than I originally intended. This was a significant investment in both time and money.

We then took away all employees from their jobs for an hour and a half to present the training. Again, this is a significant investment, especially considering the size of PagerDuty today.

Was it worth it? I think so. My team thinks so. And a lot of employees think so. But is it right for your company? Perhaps not. I can definitely understand not wanting to make such an upfront investment. We’re lucky now in that the content is already created. Once that upfront investment is over, the maintenance burden is relatively low. Which brings me to some good news …

I’m Sold! Can I See the Training?

I’m very happy to announce that, as of today, we have open-sourced our security training for use by everyone!

Not all the material is available publicly, as some content is still restricted to PagerDuty employees. However, most of the training is generic and not specific to PagerDuty itself. Any content that has been redacted is clearly labelled and includes a textual description of what the content was.

For now, the site only contains our all-employee training. Stay tuned for the engineer-specific training, which I hope to add to the site in the coming weeks.

If this sounds like something you’d like to work on, or if you’re interested in seeing the internal only slides, we’re always hiring.