Video Technology

Centro Drives Better Business Outcomes through PagerDuty Automation Capabilities

Centro provides digital advertising and buying software that helps advertisers streamline and scale digital campaigns. Jeff Smith, Director of Production Operations, shares how PagerDuty makes Centro’s on-call process more humane, empowers teams to take action in real time, and accelerates response in order to resolve more incidents.

Video Technology

Twilio Uses PagerDuty to Accelerate Innovation

Written Technology

SendGrid Enhances Employee Productivity and Reduces Downtime with PagerDuty

SendGrid is a proven cloud-based customer communication platform that successfully delivers over 25 billion emails each month for Internet and mobile-based customers. The company is headquartered in Colorado with over 300 employees, 23 of those within the operations team and approximately 84 in the development group. Mary Moore-Simmons, Engineering Operations Manager, is in charge of managing the infrastructure at SendGrid, which includes servers and data centers, the network behind it all, virtualization stacks, and backend systems. With the high rate of emails that are sent from SendGrid, there are a multitude of incident alerts generated on a daily basis. Finding a scalable enterprise-grade solution to help streamline and simplify the manual incident alert process was a top initiative for the company. Replacing previous alerting tool and overcoming scalability challenges SendGrid receives up to two thousand incident alerts in a typical day and tens of thousands per minute during technical incidents or outages. With such a large amount, it’s important for the company to address alerts quickly and efficiently. Before making the move to PagerDuty, SendGrid used a different vendor for alerting, but realized they needed a full-scale incident management solution in place to support their high volume of incidents. “When you have a tool in place, you want it to work, especially when there is an outage; that’s when you expect it to work,” said Moore-Simmons. Faced with scalability challenges, SendGrid decided to make the move to a reliable and scalable incident management solution. “PagerDuty helps us respond faster to the alerts that we receive. We’re able to diagnose outages faster, which in turn improves the experience of our customers and reduces downtime as well as any associated costs.” - Mary Moore-Simmons, Engineering Operations Manager, Sendgrid Accelerating MTTA and MTTR by switching to a new incident management platform SendGrid implemented PagerDuty as their new incident management solution and uses the platform for collaboration, scheduling, escalation, and reporting. When on-call, a user is able to acknowledge an incident alert, escalate the alert if needed, or resolve the issue at hand, allowing them to move directly to the next incident without any delay. The main dashboard which reports all incidents is another critical benefit for SendGrid. “The way PagerDuty’s incident management dashboard’s UI is designed allows you to see what’s going on and what kind of alerts you are receiving. This is super helpful for us - no more having a list of alerts moving around at all times and losing focus on them,” said Moore-Simmons. Moore-Simmons finds PagerDuty’s reporting feature to be the most important asset for her role. Reporting on metrics enables her to gather insight around the number of alerts per day, per week, per month, and per year. “We had an estimate of 78,000 alerts happen this year and the company’s goal was to reduce the number of alerts by 50% compared to 2015. So far, we are on track with this metric, thanks to the support of PagerDuty,” stated Moore-Simmons. She was also able to figure out that the team’s average mean-time-to-repair (MTTR) is 19 minutes, while the average mean-time-to-acknowledge (MTTA) is only 2 minutes. Gathering this type of information helps both Moore-Simmons and the other engineering managers identify what’s working, what’s not, and how to fix the problem. The biggest benefit to SendGrid was that their operations and development teams could now resolve outages quickly and prevent them from happening again, thanks to the reliable and rapid incident notifications. Every minute that an outage occurs costs the company thousands of dollars and results in poor customer experience and customer churn, and with fewer outages, there has been less customer churn. Moreover, the team is now more satisfied and productive after switching to PagerDuty. “We have confidence in PagerDuty and no longer have to worry about unnecessarily long outages and revenue loss.” - Mary Moore-Simmons, Engineering Operations Manager, Sendgrid Enhancing employee productivity and improving scalability SendGrid can rely on PagerDuty as a trustworthy solution to support their use cases, critical alerts, and scheduling. “We have confidence in PagerDuty and no longer have to worry about unnecessarily long outages and revenue loss. Everyone on-call at SendGrid uses PagerDuty and knows the solution as an established provider,” said Moore-Simmons. Employees are happy and productive which is important to the business. Overall, the company has seen many advantages after switching to PagerDuty, including faster resolution times for outages, increased employee productivity and happiness, as well as pulling impressive bottom-of-the-line metrics that attest to the company's operational efficiency.  

Written Technology

Ping Identity: Keeping Identities Safe With PagerDuty

Organizations all over the world trust Ping Identity to keep their employees’ identity safe. Their identity and access management platform gives enterprises one-click access to any application from any device. Ping Identity’s purpose is to enable and protect identity, defend privacy and secure the Internet. Early Ping Identity Challenges Keeping identities secure is a difficult task. Ping Identity takes their responsibility seriously and have heavily invested in their infrastructure to make sure their solutions are highly reliable. Their world-class distributed system means that if they experience data center failures because of a hurricane in the East Coast or a earthquake in the West Coast, their customers will not experience a disruption in their service. Ping Identity’s infrastructure team is responsible for ensuring all Ping Identity’s services are always up, extremely fast, all the time. To keep a pulse on how their systems are functioning day-to-day, they have created a multi-layered stack architecture to monitor their systems. Using a variety of solutions from New Relic for applications to Splunk for machine data, Ping Identity takes care of their IT systems so their customers never have to worry. However, having multiple monitoring tools presented a set of challenges. Alerts from various monitoring tools became difficult to manage and distribute. On-call responsibility was rotated weekly, but whenever someone went on-call they had to log into every monitoring system to change the alert notifications to their email. This process was cumbersome and error-prone. Also multiple teams worked with each monitoring tool so routing issues directly to the right person was difficult. Since there was no way to centrally manage alerts, alerts were missed or sent to the wrong person causing delays. This inefficient alerting process extended costly downtime. "As the team scaled, our method of changing email addresses was not going to work.” - Beau Christensen, Manager of Infrastructure Operations To provide a highly reliable service for their customers, Ping Identity needed to add an incident management solution to their infrastructure to resolve issues more quickly. How did PagerDuty Help? Since extreme reliability is important to Ping Identity, they require the same tenancy from the solutions they add to their infrastructure. Like their customers, they need to trust the tools that help keep their business going. Ping Identity has been relying on PagerDuty to reduce their incident response times. With dependable alert routing and escalation, the right person is reached faster, every time. There is no need to go into every monitoring tool to change contact information before every on-call rotation anymore. On-call engineers can maintain their contact information in PagerDuty and when it’s their turn to be on-call, PagerDuty routes alerts directly to them via multiple methods. Missed alerts were one of the biggest pains that Ping Identity experienced before PagerDuty. With multiple alerting methods, every engineer can personalize the combination of alerts that works for them. Additionally, by automatically escalating missed alerts Ping Identity never has worry about extended outages again. Not only are the right alerts going to the right person, but PagerDuty has helped to decrease alerting noise. With alert deduplication, PagerDuty has alleviated alert fatigue by bundling alerts to prevent event storms during an outage. Each incident can be analyzed to report how well each person and team is performing. By visualizing alerts, Ping Identity can spot trends and make informed decisions about how to improve their infrastructure. “If everything is going smoothly, no one should experience an interruption of service. With PagerDuty, it’s easier to keep that promise.” - Beau Christensen, Manager of Infrastructure Operations PagerDuty was initially adopted within the infrastructure team and has expanded to support, developers and help desk teams. PagerDuty has seamlessly integrated into Ping Identity’s culture and has made it easy for cross-functional teams to collaborate on incidents. Easier collaboration combined with more effective alerting has decreased incident repair time by 100%. “PagerDuty has blossomed within Ping Identity. It has become a core piece of our infrastructure.” - Beau Christensen, Manager of Infrastructure Operations If Ping Identity services are down, people cannot do their jobs. With PagerDuty, Ping Identity can deliver a highly reliable service so its customers can always have secure access to their favorite applications.