"One Step at a Time," Feature Article, July 2004

one step at a time

Research suggests that this approach can provide a real benefit in engineering experimentation.

by Daniel D. Frey and Edward M. Greitzer

Almost every engineer is involved in some way in the planning of experiments, whether they are laboratory experiments, field tests, or computer simulations. A set of techniques called Design of Experiments, or DOE, has proved to be an extremely useful methodology for enhancing the effectiveness of that planning.

Six Sigma quality programs have been a mechanism for promulgating DOE in industry, but new research shows that a complementary approach, based on a set of adaptive one-factor-at-a-time experiments, leads to better results under many conditions.

The full-factorial Design of Experiments method, which was initially developed to study agriculture, sets up an experiment for each possible combination of the factors that need to be tested. For example, consider the development of a new clutch. One aspect of its performance is drag torque—the amount of torque the clutch transmits when it is disengaged. A performance goal is to minimize drag torque by appropriate selection of materials, geometry, and other parameters.

Sir Ronald Fisher developed the Design of Experiments method in the 1920s for agricultural research.

The design team may decide to consider two factors—surface texture (waffled and smooth) and surface shape (flat and wavy). In a full factorial design of experiments, 2² experiments will test four clutches—one with a flat waffled surface, another with a flat smooth surface, a third with a wavy waffled surface, and yet another with a wavy smooth surface. The study of three factors would require eight experiments, or 2³.

One-factor-at-a-time experiments succeed each other as a series in which, at each step, a single factor is changed while other factors remain constant.

In a one-factor-at-a-time series of experiments, drag torque would be determined for a baseline configuration of the design, perhaps using flat, smooth clutch plates. Then, the experimenter changes a single factor—for example, by introducing a wavy clutch plate—while all other factors remain constant. The drag torque is measured in the new configuration. Once the effect of the flatness is determined, the effect of another parameter, say surface texture, can be evaluated by changing from a smooth to a waffled pattern.

Engineering experiments are frequently done in series (often on a single apparatus). If that is the case, the sequential nature of the experiments creates an opportunity to adapt to the new results that emerge. One way to take advantage of this opportunity is with an adaptive variant of a one-factor-at-a-time experiment. As each factor is assessed, an alternative that brings improvement is retained in studying the effect of the next factor.

If the wavy plates prove effective in reducing drag torque, the waffle pattern is applied to wavy clutch plates rather than to flat ones. If the change in surface texture improves the design, the new texture is retained, but if the waffle pattern increases the drag torque, that change is reversed before proceeding to study other factors, such as plate material or plate spacing. The adaptive one-factor-at-a-time plan thus constantly exploits new information as it is discovered.

FIELD WORK

In the 1920s, Sir Ronald Fisher, motivated by problems of agricultural experimentation, developed the techniques of factorial Design of Experiments. In most agricultural experiments, uncertainty is high, because field conditions are difficult to control. In agricultural experimentation, it is also common to run a large number of trials in parallel since results cannot be obtained quickly. They usually require a full growing season.

Because a large number of trials are run in parallel, they must be planned at one time. Design of Experiments makes use of the ability to plan the experiments to reduce the effects of experimental uncertainty.

In the four tests of the clutch, there are two observations with flat clutch plates, two with wavy, two with waffled, and two with smooth. Averaging the observations reduces the impact of uncertainty on the results.

The full factorial design also enables the assessment of interactions among experimental factors. An interaction is the effect that factors have when applied jointly. In the clutch example, the surface texture reduces drag torque when applied to flat plates, but increases drag torque when applied to wavy plates. A one-factor-at-a-time experiment does not allow the experimenter to determine such interactions.

A disadvantage of the full factorial design is that the minimum number of trials needed grows rapidly with the number of factors to be studied. Every time an additional factor is added, the minimum number of trials doubles at least. As a result, full factorial experiments are rarely conducted with a large number of factors.

To decrease the minimum number of required experimental units, fractional factorial designs are often used. For example, engineers can use just four trials to examine the effects of texture, flatness, and material on drag torque. A full factorial design with three factors would require eight experiments. The fractional factorial design is a selected subset of the full factorial experiment that preserves many of its useful properties.

A risk in using a fractional factorial approach is the possibility of confounding interaction effects (effects that two variables produce in concert) with main effects (effects produced by a factor independent of other factors). It has been shown, for example, that the flatness and texture of clutch plates exhibit such interaction. If texture, flatness, and material type are studied together, the interaction between texture and flatness may be mistakenly attributed to material type. The possible confounding of interactions with main effects is a principal drawback of fractional factorial designs.

ACCOUNTING FOR ERROR

In planning any experiment, it is important to consider the experimental error—the combined effects of many uncertainties or random variations, such as the repeatability of measuring instruments or small fluctuations in conditions like ambient temperature. One approach to dealing with experimental error is replication—running the same experiment in repeated trials and then averaging the measurements. Replication is effective, but increases the cost of the experiment. The size and influence of experimental error is a key aspect in the choice of whether to use a one-at-a-time or a Design of Experiments approach.

The measure of drag torque in our clutch will vary from trial to trial because of the imperfect repeatability of the dynamometer, changes in oil viscosity over time, and differences in manufacture of the clutch plates. If these experimental errors cause large variations in the measurements, many replicates may be required to determine the drag torque with adequate precision. Suppose, for purposes of illustration, that two replications are needed to determine whether one configuration is better. The one-variable approach would need two replications for each of the four determinations, making a total of eight tests in all.

The Design of Experiments approach, which contains the desired two observations with each factor, would need only four trials.

How can engineers decide which alternative—one-factor-at-a-time or DOE—is preferable? Recent research by the authors addresses that question.

Today, DOE, or a variation of it, is applied to mechanical systems. Perhaps to a clutch, for instance.

We examined data from 66 published full factorial experiments on a variety of different engineering components and systems. Each of the 66 data sets was used in simulations of the process of improving a product through experimentation. The clutch data, for example, came from an experiment described in a Society of Automotive Engineers paper written by Frederick Lloyd, who assessed the effects of seven variables on the drag torque of a wet clutch pack.

To simulate the process of adaptive one-at-a-time experimentation, we selected one clutch design from among the 128, or 27, variations evaluated in the paper and looked up its drag torque in the published data table. We added a random number to the tabulated drag torque value to simulate experimental error (e.g., lack of repeatability in the dynamometer's readings). Then, we changed one design variable, such as the clutch plate material, and looked up the drag torque for that design in the table (again adding a random number).

If the change in plate material seemed to reduce the drag torque, that change was adopted and another variable was changed. After all seven design variables had been toggled, the tabulated drag torque for the apparent best design was recorded. The process was repeated a thousand times, with different starting designs and different orders in which the factors were varied, for all 66 data sets.

A fractional factorial approach was also simulated. Eight different clutch designs (the same number as that used for the one-at-a-time approach) were selected according to a fractional factorial design. Each design was subjected to simulated tests by looking up the corresponding drag torque in a table and adding a random number to mimic the effects of experimental error. We analyzed the data to infer the preferred settings of each design variable and recorded the tabulated drag torque for the combination of preferred levels. This process was also repeated 1,000 times for all 66 data sets.

The information from the clutch simulations can be used to illustrate the overall trend of the results. The best clutch design found by Lloyd had a drag torque of 1.4 foot-pounds. When the experimental error was low, the simulation of the one-at-a-time approach led, on average, to 1.5 ft.-lbs. of drag torque. Thus, with only eight experiments, the one-at-a-time approach accomplished almost the same result as a full factorial approach requiring 128 experiments.

By comparison, the fractional factorial approach, also with eight experiments, led to 1.9 ft.-lbs. of drag torque, on average. As the experimental error increased, the results for both approaches were worse (the minimum torque increased), but the fractional factorial design was affected much less than the one-at-a-time approach.

Using the 66 data sets, we could define a generic pattern. The main finding was that the one-factor-at-a-time approach provides better results than a fractional factorial design, if either the experimental error is small or the interaction among factors is strong.

Design of Experiments was originally conceived as a method to allow for the randomness inherent in agricultural studies. A recent example, by Eduardo Blumwald, a professor at University of California, Davis, and colleagues, compared results with tomato plants, some genetically altered to grow in soils too salty for most crop plants and others with no genetic modification. A full demonstration combining two factors needs four experiments, as shown in the photographs. Wild-type plants (top left) and genetically modified plants (top right) were grown in trace salinity; wild-type plants (bottom left) and transgenic plants also were grown in high salinity, equal to about 35 percent of the salt concentration of seawater.

Specifically, if the variance due to simulated experimental error was less than 40 percent of the variance caused by the design changes, the adaptive approach led to better designs than the fractional factorial approach. The adaptive one-at-a-time approach was also superior, if interaction effects accounted for more than 25 percent of the observed variance in the system.

In addition, there is another important feature to an adaptive one-at-a-time approach. Sometimes, a series of experiments is terminated before the plan has been carried out—perhaps because of funding, scheduling, or experimental difficulties. An incomplete one-at-a-time approach can determine, even early on, some changes that lead to improvement. This is especially significant when the development team is able, based on engineering insights, to prioritize their efforts and investigate the most important factors first.

While the results of this study appear to be at odds with current practices, support for the results is not without precedent.

Milton Friedman, the Nobel Prize-winning economist, and the statistician Leonard Savage published a paper in the 1940s, which presented arguments about why factorial design is not as effective as a one-at-a-time plan, when the primary goal is design optimization. Friedman and Savage noted that a factorial design spreads its observations across the design space, while a one-at-a-time plan, by its nature, concentrates observations in areas of the design space that appear promising. This concentration of effort is one explanation for the effectiveness of the one-at-a-time plan.

The statistician Cuthbert Daniel also recognized the benefits of a one-at-a-time approach, but for a different set of reasons. He noted that one-at-a-time approaches allow experimenters to "learn something from each run" and "react to data more rapidly." He also acknowledged the need for a cutoff in terms of degree of experimental error.

WHAT'S IN IT FOR ENGINEERS?

Design of Experiments is a remarkably successful procedure, which has had a profound influence on the professional practice of engineering. The impacts of Design of Experiments on industry are increasing, with more education and training activities, better software support, and more expert statistical assistance.

Although the role of one-factor-at-a-time experiments has diminished, our research suggests that this approach can provide a real benefit in engineering experimentation. Specifically, experimenting one-factor-at-a-time should be considered in engineering situations in which the experimental error is small, or the interactions between factors are strong, or there is a possibility that the planned set of experiments may not be fully completed.

The bottom line is that whatever methods are used in conducting experiments, it is important that design engineers use the full range of tools at their disposal, with knowledge of both their strengths and weaknesses.

This article is based on the authors' 2003 paper, "A Role for One-Factor-at-a-Time Experimentation in Parameter Design," published in Research in Engineering Design.

Daniel D. Frey is an assistant professor of mechanical engineering and engineering systems at MIT. Edward M. Greitzer is the H.N. Slater Professor of aeronautics and astronautics at MIT and a former director of the MIT Gas Turbine Laboratory. Both are ASME members.