engineering management

experiment ...
early and often

To create a truly innovative product, you have to be willing to tinker. But establishing a culture to support that can be surprisingly hard.

By Jean Thilmany

Wouldn't it be great if you knew at the very beginning of the design cycle that a final product would work perfectly? If you knew that it would instantly meet marketplace approval? There'd be no need to mess around with pesky design details or marketing plans. You'd just dash off a design, ferry it right over to production, and have this miracle product on store shelves—and flying off them—within the week.

Alas, life doesn't work that way. A niggling little detail called uncertainty hounds product development.

"Without uncertainty, you could go straight to making products. You'd know what to do," said Stefan Thomke, an associate professor of business administration at the Harvard Business School in Cambridge, Mass.

He argues, however, that engineering managers can rework their departments to lay a clear path for engineers to walk through uncertainty toward a new and innovative product. The means of getting from start to finish along that trail? Experimentation.

The willingness to experiment can lead companies to the best, most innovative products possible. But it's a method many designers and their managers are leery of undertaking, Thomke admits.

He literally wrote the book on the importance of experimentation. He's the author of Experimentation Matters (2003, Harvard Business School Publishing Corp.), which posits that every company's ability to innovate depends on a series of experiments—and failures—that help create new products and improve old ones.

The period between the earliest point in the design cycle and final product release should be filled with experimentation, failure, analysis, and yet another round of experimentation. Lather, rinse, and repeat until an innovative product is ready to be ushered out the door, Thomke said.

Yet that's just where many companies fall down, he maintains. They're afraid to experiment. Becoming friendly with the uncertainty inherent in product development and being willing to experiment early and often are among the more savvy moves a manager can make. He advises managers to test those experiments with analysis systems that can predict end results.

New technologies—like computer modeling and simulation software that let engineers ask and answer what-if questions—change the economics of experimentation, greatly reducing the cost of prototyping. But to take full advantage of analysis software, engineering managers have to embrace experimentation, to improve the way their departments innovate, and to transform the very structure of the organization itself.

"The best companies are willing to experiment," Thomke said.
But it's not easy to do. Even if engineering managers recognize the importance of experimentation and of the proper analysis software that lets engineers do just that, they're often stymied by the corporate culture or by engineers themselves, who are afraid to fail and afraid of change.

"Managers," Thomke said, "are terrified of failure."
In his book, Thomke describes six principles engineering managers can follow to unlock their departments' innovative potential.

INNOVATION INGREDIENTS

First, managers need to fully exploit analysis technology early in the design cycle, Thomke said.

"New technologies are most powerful when they're deployed to test what works and what doesn't work as early as possible—the front-loading effect," Thomke writes in his book. "These experiments aren't as complete or perfect as late-stage tests, but they're able to direct early attention and integrated problem-solving at potential downstream risks."

According to Thomke, engineering managers ultimately make decisions about how their engineers use technology, and many engineering managers are rooted by the way they've done jobs in the past, whether outdated or not. Their engineers follow suit. It's just human nature to keep doing things the way we're used to and it's also human nature to shy from change, no matter how small.

"Say a company wants to use engineering software tools earlier in the design process," Thomke said. "It's appealing to use them early because you can find problems earlier.

"But engineers at that company are used to doing things their own way," he added. "They're not going to like adjusting to using those tools earlier in the process."

In product development, design changes become costlier as you get closer to the end because the tooling, once made, will need to be reworked.

"Managers can end up devoting an enormous amount of their time to dealing with late-stage problems—to meet launch dates, reallocate resources, unsnarl schedules, and so on," Thomke writes. "Such fire fighting is taken for granted because most product development processes aren't set up, much less optimized, for early experimentation."

Boeing managers were proactive about using software earlier in the design cycle when developing its new aircraft, the 777. The managers wanted a process that encouraged experimentation and problem solving well before final assembly, Thomke writes.

They coupled a three-dimensional, computer-aided design system with in-house software that enabled engineers to assemble and test digital mock-ups for interference problems. With such a mock-up, an engineer can assemble any part of the plane virtually to check for fit and for interferences.

The new software did more than find interferences earlier. The automatic checks also changed the way people interacted with each other, Thomke said. Not only did designers modify their designs earlier than in the past, they also relied on others to track the modifications via the software. Because they received immediate feedback, the designers were emboldened to experiment even more.

Using software earlier also means the structure of the organization will likely need to change. Engineers can change. If managers can just hang on during the learning curve, they'll be rewarded with productive and experimental engineers.

EXPERIMENT—BUT NOT TOO OFTEN

Though Thomke supports early and frequent experimentation, he cautions managers not to overload their organizations. This is his second principle toward innovation.

"A good experimentation strategy balances the values of early information against the cost of repeated testing," he said.

He advises managers to combine both new and traditional technologies in order to fully realize a department's new-product development potential. This is his third principle.

Managers must be aware of human nature as they work toward bringing in new technology to supplement, and eventually replace, the old.

The true potential of new technologies often lies in a company's ability to reconfigure its processes and organization to use them in concert with traditional technologies, Thomke said.

Today, technologies change faster than engineers can get used to working with them. That's why it's important to embrace new software while keeping the old.

"If you work with something 10 or 20 years and you have expertise with that certain technology, you're not going to be willing to quickly adapt to something else," he said. "Across an organization, up to about 10,000 people are going to be used to doing something in a certain way. You introduce a new technology, and it's not like people will suddenly embrace that."

For example, one engineering company that Thomke profiled in his book spent big bucks on high-end digital simulation technology. Yet, even after implementing that digital program, the company made more prototypes than any organization Thomke studied. The making of many prototypes was the very thing the company hoped to avoid. Technically, to get the most from a simulation system, the organization should rely primarily on that system and should build prototypes only after validating and analyzing most design possibilities via the software.

A little sleuthing soon uncovered the reason behind the plethora of prototypes.

The chief engineer told Thomke that, while company executives had asked his department to bring in the simulation technology to cut prototyping costs, his engineers didn't trust the new technology. Yes, they grudgingly used it to run design simulations. But then they built a prototype for every simulation they ran just to double check results. In the end, they built many more prototypes than before they'd had the software program.

A NEW PARTNER

Engineering managers have to set up their departments to support rapid innovation, according to Thomke's fourth principle. But many managers don't bother. As Thomke puts it, organizational structures get in the way of innovation. For instance, mechanical and electrical engineers are usually grouped at work by their expertise or their area of specialization.

"The thought is, if you put a lot of people together that build prototypes, they'll get better and better at doing that and can make better prototypes cheaper," he said.

While the idea seems sound, especially because groups of engineers usually talk to each other about a mutual project, it sidesteps quick product iteration, Thomke maintains.

"Those interfaces between groups actually get in the way," he said. "The design group isn't responsible for testing, so they hand their design over to testing. And then testing hands it back for redesign. All that takes a lot of time.

"By the time engineers get the tests back from the analysts, those engineers have already moved on to another design," he added. "Engineers can move quickly, but as they come up with these new ideas they need them tested. They don't have much time to wait for feedback from analysts and others."

Disparate engineering departments are usually managed separately as well. According to Thomke, the departments are too independent and too separate, although they're expected to interact quickly and efficiently. When a company wants to iterate at lightning speed, walls between departments get in the way, he said.

Thomke outlines one example, at BMW, in his book. The German automaker wanted to simulate vehicle crashes earlier in the design cycle. Crash tests run at roughly the same time as automobile design let analysts see more quickly where a design would fail. Design engineers needn't waste their time working through an entire model if they already know it is flawed.

But the change necessitated that two groups—analysts and engineers—work together in a way they hadn't previously.

"The downstream group, the engineers, was used to the upstream group, the analysts, coming in later in the cycle," he said. "For crash simulation, analysts needed to get some rough parameters from the people designing the doors. But the door designers were reluctant to give that information out. They didn't want to share their information until they were fully done. They didn't want anyone to see information if it was flawed."

Only after convincing the designers that their early rough data sufficed did the crash simulation group get the required information, Thomke writes in his book. But, even so, that exchange took six months. In a new development process, a six-month delay could derail an entire program.

It was incumbent upon crash simulation and design engineers alike to appreciate and understand not only each other's activities, but also the power of the new technologies that could leverage them. This understanding had to be built patiently over time.

BMW had a world-class crash simulation group, but unless the processes were changed and people started working together in new ways, its technical leadership meant little for development performance, Thomke writes.

Engineers eventually learned to change. But they didn't embrace it from the get-go. Managers shouldn't expect that. BMW engineers needed to understand why the new business process was necessary.

FAILURE VS. MISTAKE

Managers shouldn't confuse failures with mistakes. Fail early and often, but avoid mistakes, Thomke states. This is his fifth principle for unlocking innovative potential.

Experiments that result in failures are not failed experiments. Think of them instead as a way to generate new information the engineer couldn't foresee, Thomke suggests. The faster the experimentation-failure cycle, the more feedback the engineer gathers and incorporates into new rounds of development.

Mistakes are a different animal entirely. Thomke defines them as wrong actions that result from poor judgment or inattention. They are failures because they produce little new or useful information.

A poorly planned experiment with ambiguous data, for example, is a mistake, as is repeating a prior failure or learning nothing from experience.

Managers have to promote failure, but also must weed out mistakes.

A BIG EXPERIMENT

As his sixth principle for innovation, thomke advocates that managers take a different way of thinking about experimentation. Projects themselves should be conceived of as experiments, he said. An entire organizational-reform project should be thought of as an experiment.

For optimal organizational reforms, senior managers should have a portfolio of experimental projects they can learn from.

When BMW sought to put its new development process into play, busy managers and engineers balked. A year passed without any significant progress.

To jump-start the process, executives decided to use the latest 7 Series platform car—already one year into development—to test the new development system. The platform project became an experiment in and of itself, Thomke said, and eventually met with success.

It takes a good manager to walk engineers through change, to implement new technology whether or not employees balk, and to understand that there'll be a learning curve as employees adjust to the new tools and change their work procedures.

To unlock innovation and get full use of research and development, managers need to understand the power of experimentation and new technologies. They also have to change their processes, organization, and the way they manage innovation.

It's a tall order, right? But the great products that get to market quickly after engineers experiment and innovate as much as possible make the efforts well worthwhile.



Return to Index

© 2005 by The American Society of Mechanical Engineers