Truth, Lies and Surveys: A New Methodology to Assess Culture

Several years ago, the chair of a large global financial institution – let’s call him Bill – received some concerning confidential information about potential control breakdowns and ethics violations in parts of the organization, which would put the institution at risk of significant regulatory penalties, as well as reputational damage. To assess the threat, Bill brought in an outside firm to run an employee survey. Reassuringly, the results showed the risks of such violations to be very limited, a finding which the HR department confirmed. 

However, confidential concerns continued to stream in. In response, Bill commissioned another firm that applied a methodology called randomized response to repeat the assessment. The survey did not ask employees to report their own experiences with violations directly but used an indirect method to allow employees to report them in an untraceable way. 

The eye-opening results of the advanced analysis validated the allegations, revealing systemic problems within certain business units due to unethical and illegal behavior. It was a shocking revelation. 

It also arrived too late for Bill. Regulators caught wind of the bad behavior. Both the CEO and Bill were soon out of their roles, and the organization faced significant fines and reputational damage. 

At Russell Reynolds Associates, we often work with CEOs and board leaders who want to know if they have “good” cultures. They have created mission statements, held town halls, and visited factory floors to define what culture should be and personally reinforce it. 

They have run engagement surveys to make sure they are canvassing every corner of the company, asking employees to agree or disagree with questions like “I am proud to work at this company” and “We care about our customers.” 

Often, though, it remains unclear whether there is any relationship between stated company values and what people consider appropriate behavior. In a recent study of nearly 700 US companies, researchers at MIT Sloan School of Management and London Business School found little to no correlation between stated company values and employee perceptions of the organization.¹ Newspaper headlines confirm this lack of alignment, routinely showcasing the gaps between stated corporate values and employee behavior in the form of corporate fraud, regulatory infractions, safety violations, and discriminatory incidents. These issues underscore how complex and multi-faceted any organization’s culture can be – and how essential it is to have a clearer way to understand how employees behave, as well as the values that they espouse.  

To help leaders better understand their organizations, this paper:

Why most culture data is not credible

No single perspective can paint a full picture of an organization’s culture. Instead, culture is the combination of both leaders’ values (tone at the top) and employees’ beliefs, behaviors, and perceptions (echo from the bottom). 

Most leaders have a clear vision of what they want their culture to be and go to great lengths to define and demonstrate it, making the tone at the top relatively easy to gauge. Yet as our story about Bill illustrates, gaining an accurate understanding of the employee perspective is nuanced and challenging. Standard surveys that ask employees how strongly they agree or disagree with statements about their workplace often fail to uncover deep-seated issues. 

Standard culture surveys have several shortcomings: 

1) Few people believe the promises of anonymity that accompany these surveys. As a result, they answer as if their identity was known, which incentivizes them to make themselves look good. They are unlikely to respond honestly to questions on sensitive topics such as ethics, safety, or commitment to diversity for fear of reprisal. 

2) Loyalty is a powerful force. Most people have a natural inclination to protect those they work closely with and report more positively on them as well. 

3) External pressure from managers can come into play, particularly if the manager’s compensation is linked to survey results. 

In combination, these factors tend to give management and boards an overly rosy picture of the strengths and risks in their organization’s culture. Without deeper insights, problematic behaviors and attitudes are difficult to address. 

For example, in the 2011 Mississippi General Election, the results of a public vote on banning abortion demonstrates the wide variation between what people say and what they do. Despite having strong support in pre-election polls, the bill ultimately failed by a wide margin.² Princeton University conducted a research study via phone interviews with voters comparing their aggregate answers against county-level voting data. The results were stunning: While public records showed that 65% of voters in the 19 counties surveyed had voted “no,” only 30% of the 2,655 voters surveyed by phone admitted to doing so, with another 20% refusing to answer the question.

These challenges can make culture seem elusive and immeasurable. It is not. Innovations in survey methods and advanced statistical approaches are creating new and better ways to calibrate culture. A new methodology used by Russell Reynolds Associates involves asking sensitive questions in a protected manner, making behaviors untraceable to specific individuals but providing greater insights across groups. We refer to this as culture imaging. 

Seeing through walls: a new approach to measuring and managing culture 

In an ideal world, leaders would understand what employees truly believe instead of what they think management wants to hear. This would allow leaders to accurately understand the day-to-day behaviors taking place on the ground. They would be able to see hotspots where there is a high degree of misalignment and pinpoint precisely where the breakdowns are occurring. With this information, they could build thoughtful strategies to address concerns, moving the entire organization towards lower risk and better performance. 

Several of our clients recently adopted advanced tactics, using Russell Reynolds Associates’ new culture analytics approach. Developed by behavioral economists, political scientists, and neuroscientists, this approach gets closer to employees’ deep attitudes and behaviors through innovation in survey design, as well as new ways of analyzing responses. 

A key feature of Russell Reynolds’ cultural imaging approach is that employees never have to respond directly to questions about potentially incriminating or embarrassing behavior. Instead, sensitive statements are bundled with neutral, odds-based statements (such as dice rolls or relatives’ birthdates) and employees simply indicate whether any of the statements are true. This survey design makes it impossible for anyone to know the respondent’s true attitudes or behaviors, while providing enough information to calculate precise group-level estimates of behavior. 

Key Features of RRA's Culture Analytics Approach

A brief tutorial at the start of the survey helps respondents understand how and why their answers will not be traceable to them individually. 

Sensitive statements are bundled with neutral, odds-based statements; respondents can indicate how many statements are true without specifying which ones.

Sophisticated algorithms translate survey data into actionable information using probabilities to determine group-level estimates. 

Results offer a complete, multidimensional “cultural MRI” revealing the degree of alignment among leadership values, employee values, and lived experiences, as well as the degree of alignment across business or demographic groups. 

Toward a higher method for scanning culture

Taking a closer look at culture often means bearing difficult news, as leaders learn that compliance with desired norms is a bigger challenge than they may have believed. Yet failing to acknowledge these gaps is ultimately more painful and potentially destructive than proactively uncovering them. In fact, leadership is often the key enabler (or obstacle) to a strong culture, and the benefits it can produce. Leaders who are not afraid to confront hard truths are essential to building resilient cultures that sustain high performance over time. 

We recommend these critical steps as starting points to gaining this authentic understanding: 

As a leadership team, it is essential to align on the organization's strategic direction and the culture required to enable it. This starts by getting crystal clear on the "cultural truths" that need to exist and being explicit on how they link to strategic imperatives. 

To get a clear and accurate picture of an organization’s existing culture, leaders must ensure they have the right assessment tools and methods. These tools should not only use advanced techniques and statistical analyses, they should also reflect the unique language of your organization so that they are seen as relevant to your people. 

Organizations often try to do too much with the insights they get on culture. Keeping strategic needs in mind, avoid the tendency to do too much action planning, and stay focused on a few organization-wide and surgical interventions in the areas that matter most. 

Culture is constantly evolving, whether leaders want it to or not. The ability to regularly assess and refocus interventions is nearly as important as getting accurate information in the first place. 

With a better handle on culture, boards are able to mitigate risk to the organization, executive teams are able to ensure that strategy becomes reality, and human capital leaders have a powerful tool to attract and retain the talent that best fits the organization’s ideals. The right culture starts at the top, but its benefits elevate the entire organization. 

For sensitive topics, randomized response provides a deeper and truer level of insight than standard, direct methods can glean. In the study of Mississippi voter behavior mentioned earlier, the Princeton researchers also tried a variety of indirect survey methods to see if they would elicit more truthful answers than asking people directly about how they voted on a controversial issue. While all of the indirect methods produced more accurate results than direct questions, randomized response was the clear winner: of the nearly 1000 people surveyed with this method, 65% were estimated to have voted against the abortion ban; exactly the number reflected by public voting data. Similar results hold true in studies of Olympic athletes asked about doping, or drug use.³ 

Leaders can use this state-of-the-art culture analytics approach to measure employees’ alignment with a variety of corporate priorities, from inclusion to financial integrity to sustainability. The insights it surfaces then come together in what we call a “culture MRI,” or a visual representation of the degrees of alignment across different aspects of the enterprise, allowing leaders to quickly pinpoint both where and why problems may be emerging. 

Culture Imaging Shows Multiple Layers Of Results Across Functions And Regions  

I am appropriately recognized for my contributions.  

_{Source: Russell Reynolds Associates}

From this visual, it quickly becomes clear that the sales function in the Americas has strength in this topic area, while R&D in the Americas and APAC have work to do. To address these “hotspots,” the organization might consider having sales leaders share best practices with R&D leaders. 

By applying a culture imaging methodology and re-tallying results across the organization, the advanced algorithms that power this approach gave leaders a full, multi-dimensional picture of how the attitudes and behaviors varied within and among groups and levels. This diagnostic ultimately allowed them to see that cultural variation was closely connected to leadership at the general foreman level, creating a clear path for future interventions.

<sub>1. Donald Sull, Stegano Turconi, Charles Sull, “When It Comes to Culture, Does Your Company Walk the Talk? Company practices often conflict with corporate values. Closing the gap starts with communication”, MIT Sloan Management Review, July 21, 2020.
2. Winston Chou, Kosuke Imai, Bryan Rosenfeld, “Sensitive Survey Questions with Auxiliary Information. Sociological Methods & Research”, December 11, 2017. 
3. Roger Pielke, “Assessing Doping Prevalence is Possible. So What Are We Waiting For?”, Sports Med, October 5, 2017.</sub> 

Author

Sean Dineen is a member of Russell Reynolds Associates’ Leadership & Succession Practice. He is based in Boston.

Jamie Hechinger co-leads Russell Reynolds Associates' Social Impact and Education Sector. She is based in Washington DC.

Dee Symons is a member of Russell Reynolds Associates' Board & CEO practice. She is based in London.

Eric Wimpfheimer leads Knowledge for Russell Reynolds Associates’ Leadership & Succession capability. He is based in New York.