- PagerDuty /
- Engineering Blog /
- Recipe for Meaningful Change
Engineering Blog
Recipe for Meaningful Change
I’ve had the privilege to be with PagerDuty since 2016, and in that time, I’ve seen a lot of change. I’ve seen the company evolve from a private startup to an enterprise that undertook a successful IPO. Being part of a team that consistently delivered projects and a company that grew quickly yet sustainably allowed me to observe and understand how meaningful change happens.
Although I’ve approached this question from an engineering perspective, I tried to keep these ideas generic; they can be applied to many parts of a business: engineering, product, management, and many others. To be able to lead in any arena, one must be capable of driving meaningful change. I believe this is only possible with a fundamental understanding of the business.
What is Meaningful Change?
It’s important to distinguish between simply driving change versus driving meaningful change. Simply driving change may not be good enough. Change can result in accidental complexity or yield short term gains at the expense of dark debt that costs the business more than the value it produced later down the line. Change can also destabilize the company or put the business at risk. I define meaningful change as change that results in sustained or net positive value added for the business over its lifetime.
To drive meaningful change, I believe it’s vital to have a solid understanding of:
- Cost and impact
- Importance of influence
- Challenges of coordination
I believe that individuals who effect meaningful change consistently have a very deep or intuitive understanding of these concepts and use them to make the appropriate tradeoffs.
“Change can result in accidental complexity or yield short term gains at the expense of dark debt that costs the business more than the value it produced later down the line.”
Cost and Impact
When discussing the topic of driving change, we must talk about cost and impact. This is not a new concept, and I’m not going to bore folks with the details of Risk Impact, which is well understood by most functioning organizations, but I believe it’s important to understand value birthed from change.
The potential net value (upside) created by change is a function of its cost and impact:
- What are the resources required to drive said change?
- How many internal/external stakeholders and customers will be affected by this change?
- How will this change affect confidence in the company (internal/external)?
- What are the expected, short-term returns?
- How does the decision play into the business’s strategy (expected long-term returns)?
Most often, impact behaves similar to leverage in financial instruments; both the upside and downside are amplified as exposure increases. Decisions that lead to more visible changes can either pay off immensely or put the company at significant risk.
This is why it’s imperative to understand the risk appetite of the business when evaluating an initiative or plan of action. In general, larger companies are risk averse due to the downside because there is more at stake while tiny startups cannot survive without undertaking significant risk. Even in larger companies, there may be teams or departments whose mandates both afford and necessitate greater risk taking. Because effective risk taking and risk management are tied directly to growth, developing an intuition for and exhibiting these abilities is extremely valuable to the business and crucial to one’s ability to drive meaningful change.
In general, as impact increases, so does risk (and often, but not always, cost). As the monetary stake involved increases, the number of stakeholders increases, and responsibility is afforded to those that are trusted and have the most influence. Because of this, driving high-impact decisions requires significant influence across the company.
Importance of Influence
“Often, they need access to certain people or resources—and that access requires influence.”
Influence is a form of social capital that’s profoundly important in the healthy function of any human system, whether it’s government, schools, nonprofits, or for-profit organizations and is at the very core of the human experience. It’s an idea that is very intuitive and embedded subconsciously in the human psyche—yet not talked about enough.
By and large, almost any meaningful change that happens in an organization is driven by some very motivated individuals. To be able to operate in the capacity that they need to, these individuals need to wield some amount of influence to be able to execute their vision. Often, they need access to certain people or resources—and that access requires influence.
When talking about developing influence, I think the best way to understand it is by discussing incentives. The business and one’s peers operate on different incentives — influence derived from either will depend on the value that they each receive.
Providing Value to the Business
When a business’s interests and incentives are tied directly to delivering value, it follows that employees who provide the most value will be rewarded (whether it’s compensation or social capital). I believe there’s two major ways to provide value to the business:
Tangible Business Value
Short term
- An individual or team increases revenue or decreases cost in a meaningful way
- Many product facing engineering teams work on features that roll up into a product SKU that customers are actively paying for
- As a personal example, in 2016 I led an effort to scale down over-provisioned machines across PagerDuty, saving roughly $270,000 per year
Long term
- Strategic decisions that position the business for massive expansion of customers and potential revenue—which are almost always driven by executives or very senior individuals. A great example at PagerDuty is the evolution of the product from a simple paging tool into a platform consisting of multiple products, such as PagerDuty Analytics, Event Intelligence, etc.
- Contributes to a positive company brand and image (customer satisfaction, evangelism, etc.)
Organizational support
- Any work that provides intangible value and empowers employees either directly or indirectly by increasing efficiency or reducing waste across the org
- As an example, consider an onboarding process that allowed employees to be productive in half the time after joining? Is there a dollar value attached to that in an organization with thousands of employees?
- Another good example of this would be development of tooling and elimination of tech debt. Both enable developers to deploy code much more efficiently and cut down on time spent as a result of inadequate tooling or dark debt
Providing Value to Peers
Peers also have a financial stake in the company and are vested in its success, but less so than executives—their incentives are tied more so with job satisfaction and career development. In my opinion (and speaking from experience), as seniority increases, employees need more competent and capable peers to learn from and help them through their journey. An individual can provide value and develop influence among peers in two main ways:
- Demonstrating expertise
- Valuable employees take pride in their skills and want to surround themselves with other experts in their professions (to learn from, collaborate with, etc.)
- Providing material support
- Employees value and elevate peers who support them in achieving goals
Visibility & Developing a Brand
One of the most important parts of developing influence is establishing one’s brand in the organization and being seen. There’s work that needs to be done to develop expertise and domain knowledge in a certain area, but there’s also work that needs to be done to market one’s self and make the organization aware of one’s interest and abilities. From experience, most engineers focus on the former, while the latter seems to fall by the wayside. Advocating for yourself is a very important part of developing influence!
Let’s say an individual wants to be able to drive highly visible system design decisions in engineering. Before engineering leadership entrusts the individual with the responsibility to drive said decision, they will ask themselves: “What do I know about this person? What is this person’s track record on similar work in the past? How has their previous decisions regarding tradeoffs on different designs and systems panned out?” For the individual to have won the trust of leadership, they need to have worked towards that reputation by focusing on work that allows them to both exercise and demonstrate it on a smaller scale.
We’ve talked a lot about influence and how we may go about achieving it. But we need to understand the function it serves. Is it really necessary? I believe it is. Our need for hierarchy and influence serves to address the problem of coordination.
Challenges of Coordination
Flocks of birds exhibit swarm intelligence to self organize and coordinate movements
At PagerCon 2020, I had the privilege of giving a talk on the topic of Coordination, Effective Consensus, and Avoiding Gridlock. I thought this blog post could be another good opportunity to allude to some of these ideas. When is it acceptable to incur the costs of coordination and when is it essential to minimize them? To understand the tradeoffs, we have to analyze and understand hierarchies.
Every individual has their own perception of reality and the world we live in. Our diverse backgrounds and experiences mean that we will often have different opinions on the solutions to problems. This is where coordination and communication becomes necessary and why it makes sense to organize ourselves in hierarchies.
The vast majority of hierarchies for businesses at scale, including governance at PagerDuty, is centralized. Select organizations have been known to buck the trend — in an attempt to increase transparency and distribute decision-making power, some organizations adopted “Holacracy,” a form of flat hierarchy as their form of governance. This blog post does a phenomenal job explaining the drawbacks with flat hierarchies and why many high-profile companies that adopted it experienced significant challenges and had to for the most part, re-organize their governance. In short, it can be difficult to make decisions that are well informed and efficient due to coordination challenges, especially as a business grows.
This picture from the aforementioned blog post provides a visual into how quickly the cost of coordination (denoted by edges) can increase as more actors (denoted by nodes) are added:
The number of edges (signifying cost of coordination) increases quadratically:
Without a centralized hierarchy, speed and quality of decision-making suffers drastically at scale due to non-linear increase in coordination cost. This is formally referred to as combinatorial explosion.
I have two general takeaways from this discussion.
First, it’s important to take the above with a huge grain of salt, as there is no one-size-fits-all solution. I’ve experienced significant improvement to the morale and performance of my team since we moved to a model where we distributed leadership to some extent and allowed room for less senior members of the team to also lead. In a system with low coordination costs, such as a small team, distributed approaches to leadership can be very empowering for the individuals and the team. At scale, however, is when the drawbacks of a distributed or decentralized leadership model starts to really become apparent and inhibiting.
Secondly, because of rapidly increasing coordination costs resulting from combinatorial explosion, it can be cost prohibitive to achieve absolute consensus among all stakeholders. Not only that, a single dissenting party can cause gridlock, and decision making can grind to a halt. This is why it’s often more preferable to aim for and achieve rough consensus, particularly when there are many stakeholders involved. As long as all concerns have been addressed, it’s important that everyone commits despite disagreements. Once again, this is not a one-size-fits -all approach, but it can be a good strategy to strike a balance between alignment and speed of decision making.
Closing Remarks
As leaders, we should strive for meaningful change; however, it’s important for us to recognize that not all meaningful changes are immediately visible and generate lots of buzz. Small incremental improvements are often overlooked despite being necessary and moving the organization in the right direction and setting the foundation for dramatic improvements.
To drive meaningful change, we have to understand our domain, master our craft, make the right tradeoffs, listen, and influence those around us. We must try to minimize our coordination costs as much as possible without taking away control and autonomy from the people who deliver our successes. At the same time, hierarchies exist and they are a good thing—let’s use them to our advantage. Let’s empower those that love to execute and are good at it. Let’s foster our future leaders, instill these values, and provide them the opportunity—should they seek it.
As a final remark: one of the things that I’ve always appreciated about PagerDuty as a company has been its humanity. Let’s be kind. Let’s be mindful of how our actions may take away influence from others and share stake, success, and recognition as much as reasonably possible. This is how we can grow—all of us, and bask in our successes, unified and as a team.