REA Group Embraces Digital Transformation with PagerDuty

REA Group Replaces Pagers with PagerDuty

REA Group Limited is a multinational digital advertising company specializing in property, operating the leading property website in Australia, and prominent sites across Asia. Their purpose is to ‘change the way the world experiences property’ which they do through developing innovative products and creating a dynamic working culture that fosters inventive thinking.  Millions of people around the world use REA Group’s websites to find property every day, so the platform must always be on and performing well to ensure people can search for properties at any time, from anywhere. It’s therefore mission-critical for REA Group to act and respond to incidents impacting platform performance without their customers noticing. At the same time, REA Group cannot lose focus on operational efficiency for their software development and management team, especially in the face of rapid growth.

Challenges: Monolithic Incident Alerting and Siloed Operations

In 2014, prior to adopting PagerDuty, REA Group’s operations team relied on a monolithic, inefficient alert notification system that required engineers to carry physical pagers at all times. Because a system based on physical pagers was challenging to change and optimize, the REA team couldn’t guarantee that the right alerts were delivered to the right people, which delayed incident response times. Furthermore, on-call engineers were constantly being notified of non-critical or non-actionable alerts especially out of hours. “It was a nightmare during night-time — a really painful process,” Javier Turegano Molina, Global Infrastructure and Architecture Manager at REA Group, said about the on call experience in those early days.

The second major challenge for the team was the siloed structure of the organization. The organization was composed of many different groups who were each responsible for developing distinct parts of the company’s ecosystem, but all incidents were relayed to a centralized operations team. REA focused on breaking down these silos by embracing a DevOps culture, shifting the ownership of operations towards the teams that were building and maintaining the applications. For this change to be successful, alerts needed to be delivered to the team directly and not sent to a separate centralized unit.

“We now have a way of sending the right alerts to the right people, and at the right time.”

– Javier Turegano Molina, Global Infrastructure and Architecture Manager at REA Group

Achieving Agile Incident Management with PagerDuty

In 2014, Turegano and his team implemented PagerDuty to improve incident response time and to fully embrace the DevOps way of working.

With PagerDuty, REA can streamline the way incidents are managed across its entire organization by coordinating incident responses in a tailored, agile fashion. Incident escalation policies are customized so that alerts are delivered to the right people based on the nature of the problem, including the team who owns the affected service and the engineer who is best suited to handle the issue. The teams now put a great emphasis in designing their alert to match the SLAs and ensure the team is not alerted without a real reason. The outcome is that all teams who own the service now have full accountability. This has required a critical shift in mentality with teams now understanding that if you build it, you run it.  “Being able to tune the schedules was a really great feature for us,” Turegano explained.

Physical pagers have become a thing of the past. Incident notifications are now delivered through PagerDuty, allowing engineers (developers, qa’s, systems, etc.) to be notified via their phones and other devices that they already use and own. “Having no more physical pagers has been life-changing,” Turegano said.

With the metrics that PagerDuty automatically collect, Turegano and his team have improved their operations. PagerDuty provides data that helps them determine their Mean Time to Repair (MTTR), which allows REA Group to track how the performance of its operations team evolves over time during an incident. PagerDuty also aggregates metrics from the diverse set of monitoring tools that the team already uses, such as AWS CloudWatch, Nagios, New Relic and Splunk. These aggregated metrics are invaluable for performing post-mortems after an incident in order to prevent similar issues from recurring in the future, Turegano said.

REA now uses PagerDuty to power all of its digital operations. “Anything that can break will send an alert to PagerDuty, and we now have a way of sending the right alerts to the right people,” Turegano said. He added that REA has not just become more efficient in the way they handle alerts, they’ve undergone an entire DevOps cultural change and PagerDuty has been a great enabler in this journey.

If you want to read more of how REA has scaled on-call check this post in their tech blog.