A combination of the availability of massive amounts of consumer data and improved computing power, plus the fact that it’s easier and cheaper to randomize participants than ever before, has created the perfect conditions for the boom to flourish. what is called the “experimental revolution” in business. As businesses increasingly demand actionable data to inform decisions by leveraging experiences, MBA programs are uniquely positioned to equip future leaders with the skills they need. The authors present three main actions that business schools should take to better position students to realize their full potential as leaders of the experimental revolution. As they add courses in cybersecurity, AI / machine learning, and fintech, business schools shouldn’t forget to get back to basics. They have stood the test of time to generate reliable and insightful results in science and philosophy.
A fundamental change in the way businesses make decisions is taking place. Traditionally, conducting experiments and respecting the scientific method was the domain of scientists, but managers in all industries have adopted experimental cultures to inform their decisions on an unprecedented scale.
AT Microsoft Bing, for example, about 80% of product modifications are initially tested as controlled experiments. eBay now saves several million dollars a year in advertising thanks to a simple internal study, which found that buying paid search ads was not profitable for them. Google runs experiments to see which interview questions are most likely to match job performance, and then heavily weights candidate responses to those questions.
A combination of the availability of massive amounts of consumer data and improved computing power, along with the fact that it’s easier and cheaper to randomize participants than ever before, has created the perfect conditions for the boom. of what is called the “Experimental revolution” in the business. As businesses increasingly demand actionable data to inform decisions by leveraging experiences, MBA programs need to do more to equip future leaders with the skills they need.
The case of experimentation
The value of experiments in non-scientific organizations is quite high. Instead of calling managers to solve every puzzle or argument, big and small (should we make the background yellow or blue? Should we improve basic functionality or add new functionality? properly supported and prompted to provide quick responses?), teams can experiment and measure the outcomes of interest and, armed with new data, decide for themselves, or at least present a proposal based on relevant information. The data also provides tangible deliverables to show to stakeholders to demonstrate progress and accountability.
Experiences stimulate innovation. They can provide proof of concept and some degree of confidence in new ideas before taking bigger risks and scaling up. When done well, with data collected and interpreted objectively, experiments can also provide a corrective for mistaken intuition, inaccurate assumptions, or overconfidence. The scientific method (which feeds the experiments) is the gold standard of tools to combat bias and answer questions objectively.
But as more companies embrace a culture of experimentation, they face a major challenge: talent. Experiments are difficult to do well. Some challenges include special statistical knowledge, a clear definition of the problem, and interpretation of the results. And it is not enough to have the skills. Experiments should ideally be carried out iteratively, building on previous knowledge and working towards a deeper understanding of the issue at hand. There are also the issues of preparing managers to override their intuition when the data does not agree with it, and their ability to navigate the hierarchy and bureaucracy to implement changes based on the results of experiments.
Some companies seem to be hiring small armies of PhDs to tackle these skill challenges. (Amazon, for example, employs more than 100 economists with doctorates.) This is not surprising, given that doctorate holders receive years of training – and the shrinking tenure track market in academia has created a glut of doctorates. Other companies develop their employees internally, training them in narrow, industry-specific methodologies. For example, General Mills recently hired for their group of innovative incubators, called g-works, advertising employees who “use entrepreneurial skills and an experimental mindset” in what they called a “testing and learning environment , with rapid experimentation to validate or invalidate the hypotheses ”. Other companies, including Fidelity, LinkedIn, and Aetna, have hired consultants to conduct experiments, including Irrational laboratories, co-founded by Dan Ariely of Duke University and behavioral economist Kristen Berman.
The bottom line is that today’s businesses are looking for the person who can do it all – design and run experiments, master the numbers to interpret the results, use their interpersonal skills to implement data-driven change. , and be inspirational and visionary enough to lead a culture of experimentation. These skills, which are at the heart of what business students need to know upon graduation, require training in experimentation and the scientific method. But business schools do not put this training at the heart of their programs. What can they do differently?
Implement the experiment in business schools
Some business schools teach experimentation to a limited extent. Professors who teach marketing courses, for example, often cover basic forms of A / B testing (for example, analyzing whether an advertisement versus a public service advertisement leads to more product purchases and if the advertising campaign was a good investment). Some teachers also teach “Booking.com“, a case of Thomke and Beyersdorfer on a travel accommodation company that has an exemplary culture of experimentation, allowing employees to conduct thousands of experiments per day. In addition to culture, the case teaches a few nuts and bolts and bolts of experiments, including terminology (eg, p-value, null hypothesis, type 1 error). But that’s just a duty and a case – and the teaching notes from “Booking.com” even warn instructors that most business school students “will have little or no experience with online experimentation.”
One of us (Elizabeth) spends time experimenting in a class called Managing and Leading in Organizations in the PMBA program at the David Eccles School of Business, University of Utah: every year in class, she asks the students to divide into groups and each one designs an experiment, which they then discuss in class. The students participating in this exercise learn valuable lessons about experimentation, but the course itself is not devoted to the use of experimentation in managerial decisions. There is one notable course that does, taught at Harvard Business School by Michael Luca, but it isn’t offered every year. The teaching of real experimentation is neither typical nor widespread in business schools.
Doing more would not be difficult. Through their university libraries, MBA students already have access to a comprehensive published literature that they could use to build and test ideas. They also usually have Qualtrics or similar accounts, which allow them to create surveys, and they can access statistical packages to analyze the data. If they learned the basics in a course on business research methods, they could partner with companies while still in school to gain experience with field experiences while providing an advantage to the company.
An MBA-level experimentation course should include fundamental aspects of the scientific method, which can then be tailored to the specifics of a given company or industry. The basics include:
- How to formulate verifiable and falsifiable research questions and build on what has already been done
- How to operationalize constructions and define tasks
- How to think about the random assignment of conditions and threats to the validity of the experiment
- Why a transparent workflow can enable scrutiny and help protect the integrity of the experience
- How to do inferential statistics
Overall, we believe there are three main actions that business schools should take in order to better position students to realize their full potential as leaders of the Experimental Revolution. First, educate students on how to effectively harness academic literature and build on knowledge gained from previous experimental work, then consolidate that as one of the many resources they turn to to generate knowledge. value. Second, to teach the scientific method and how to conduct effective experiments in the business world using best practices. Finally, give students the opportunity to practice these methods while they are at school, so that they are more likely to apply them once they leave the classroom and enter the venue. of work.
But some questions remain: to what extent can or should experimental research methods be integrated into the MBA curriculum? Is one class enough? Should we require a research apprenticeship in a company or with a university researcher? Each school’s faculty responds better to these questions, keeping in mind the school’s mission, existing curriculum, student body, and industry / employer partners.
For centuries, in all kinds of professional fields, we have relied on experimentation and the scientific method to generate reliable and insightful results. But too often in business today, as we try to deal with the onslaught of change, we rely on expensive and savage decision-making based on intuition, and it costs us dearly. It’s time for us, with the help of MBA programs, to get back to basics.