the success story

Two researchers say that the best design
teams, like storytellers, share different
takes on the common theme.

by Jean Thilmany

By hunkering down together on a deadline-sensitive design project, engineering design team members inevitably come to create their own little world within a world.

That unique island takes shape over time, starting from the moment the team decides upon an initial design concept. Each member in turn adds his or her unique vision to the project, building as they go. By sharing ideas, disagreeing, and resolving differences, a shared world takes shape over the course of weeks and months, according to Alice Agogino and Shuang Song, professor and student, respectively, at the University of California, Berkeley.

But how to tell from the outset if a team-created world will lead to an innovative product—or a flop? That's the question Agogino and Song would like to answer.

The two have spent the last five years looking at how design teams communicate, in the hope that their research will pinpoint the elements that separate the successful from the also-rans. Agogino is a professor of mechanical engineering at the school, while Song is a Ph.D. student in the department.

Product design tells its own story, although design teams need not agree on that story at every turn. This is a train wheel assembly.

For their research, they've fitted a storytelling model to the process of designing a new consumer product. Engineering team members build the design, or story, much the same way that oral storytellers have traditionally carried their tales through the countryside. The narrative was passed back and forth among the storytellers, who expanded on it.

"Weaving a new design concept is like weaving a story," Agogino said.

Think of team members as deciding en masse on the stark outline of a story—or, in this case, a design—then enlarging on the original by fleshing it out, passing details back and forth in documents, e-mails, and design sketches to eventually come up with the completed product.

The storytellers had to ask themselves who would listen and why.

Engineers have to ask who will use a product and why. The stories that the team generates convey the value they think the product will bring to those who will use it, Agogino said.

She likens the design process to the storyboards filmmakers create before filming begins. Those storyboards provide an overview of how individual scenes will play out and fit together. They detail what will be needed for each scene, and how scenes will be cut to result in a whole. Storyboards can be changed or expanded upon as filming goes forward.

ON THE SAME PAGE

Wwith this storytelling analogy in mind, agogino and Song have looked at how eight graduate design teams at Berkeley interacted throughout the length of a design project. Their ongoing study teases out the traits that high-performance teams share. With those traits in hand, they plan to develop metrics that will help identify high-performing and low-performing teams early in the design process.

"We're really interested in early warnings about problems with teams," Agogino said. "We wanted to come up with indices of what makes a high-performing team and what makes a poor-performing team, and how you can identify those things while you still have time to take action."

They've had some surprises during their five-year research into design-team dynamics. They expected that after team members agreed upon an initial design, members of the successful teams would pass their story back and forth in agreement. That is, once they had a shared product vision, they'd just get clearer and clearer on that vision the closer they came to deadline.

Not so. They found that while high-performing teams have their days of shared vision, there are other days when it seems as if team members have entirely separate goals for the product.

"Our prediction was that when you start out with a new team that doesn't know each other, where members come from different disciplines, you'd expect low coherence," Agogino said. "They're developing a vision collaboratively and it'll take some time to get to that shared vision. Then, you'd expect that vision to get better and better in the high-performance team."

However, the researchers found that members of the most creative teams don't necessarily think alike at all times. Although members agree upon the same product story in the beginning, they often tell different stories during the course of the design process itself. Within the collaborative act of sorting out and merging those stories lies the creative process, Agogino said. Successful teams grow through periods of agreement and disagreement.

"You wouldn't think high-performance teams would disagree so much," she added. "They're telling different stories throughout the entire process.

"But we found multiple stories is a good thing at different stages of the design process," she said. "Low-performing teams start out with a low concept and they refuse to change it."

Although team members in the class that Agogino and Song studied came from different disciplines, they were expected to work together to design an innovative consumer product.

Engineering students represented one-third of each team. Although primarily mechanical engineers, students also came from civil and electrical engineering, as well as from computer and information science backgrounds. Graduate business students made up another third and industrial design students formed a third.

In order to find patterns of communication, Agogino and Song used a computer program that mathematically analyzed the written documents—such as e-mails—that team members generated during the design process.

To maintain the privacy of team members, e-mails and design documents were never read by a human, Agogino stressed.

Weaving a new design concept is like weaving a story, says mechanical engineering professor Alice Agogino of UC-Berkeley.

This analysis tool gave researchers a means of capturing and dispassionately analyzing the real-world context in which design concepts evolved. It provided a way to measure team members' conflicting ideas, their moments of agreement, and points of argument as they moved toward a completed project.

The technique, known as latent semantic analysis, was originally developed by educators to objectively measure the quality and quantity of a student's expression in an essay. It is also used to help determine whether a writer is the author of documents credited to him.

Latent semantic analysis, which Song said is similar to mathematics and control theory, is a technique for comparing text similarity that was developed (and patented) by Bellcore. It was originally developed for information retrieval, for example, searching a large database of texts for those works that match a specific query. It has since been applied to a number of human-related tasks by researchers in psychology.

The technique uses mathematical algorithms to count via computer the frequency of words within documents. Software scans multiple documents generated by one team, searching for how many times the same words appear within the documents. The program also looks for synonyms and relationships between words. The same words, and similar words, reflect points of agreement, which can be charted.

"Different people use different words, but if all the words together are telling the same story, you'll get semantic coherence," Agogino said. This semantic coherence helps establish that team members are talking about the same concepts, even if they describe them differently.

Semantic analysis doesn't infringe on privacy, because the documents go straight into a computer, Agogino said.

With analysis in hand, the researchers could graphically represent the periods when team members agreed upon a shared vision and the places where they diverged. Agogino and Song charted each team's highs and lows in communication as they worked toward the final product.

Supplied with each team's grade from this analysis and with feedback from the class professor, the researchers then separated the high-, low-, and medium-functioning teams and matched each team's latent semantic analysis against its performance. In this way, they discovered the storytelling differences between teams.

They learned that high-functioning teams aren't necessarily on the same page at every moment in the design process.

In another experiment, Song studied the sketches each team produced—from doodles to more developed designs—to discover early-design-concept differences between high- and low-functioning teams. Her goal here was to discover how high-performing teams function in their early stages.

"You can't get a semantic coherence measurement out of the sketching, but you can look at the variety," Song said. "Are they building and refining the idea, or are they redrafting the same idea again and again?"

KILLER TEAMS

T he researchers are betting that engineering and design managers would like to have a way to measure team performance from the get-go. The higher-ups could then step in to suggest changes for teams that seem to be communicating at cross-purposes.

Agogino says that work with latent semantic analysis may provide a way to red-flag teams in trouble. She likens the latent semantic analysis technique to a kind of artificial intelligence method whereby team members' e-mails and documents are analyzed via computer at various stages in the design process. This would give an instructor or manager a series of graphical representations of the team's communication. If members' communication never converges, the manager can consider that an early warning sign that a team isn't performing up to snuff.

Agogino expects that these analysis techniques could also be applied to multidisciplinary design classes, in which student teams develop one semester-long project.

"The problem with a design class is that they don't have an exam after five weeks," Agogino said. "You don't know how a team is doing. An instructor can't know until the very last day, when projects are finally shown, whether a design team is pulling together."

Instructors have a hard time giving their student teams feedback during the semester because they're not with teams each time they meet.

TOP DOWN

Although they were initially surprised to discover that each team member telling a differing story at various points in the product-design cycle wasn't necessarily a bad thing, Agogino and Song say the principle is reflected in some business teachings. Other textbooks, however, still hold that disagreement between team members is best avoided.

"A lot of textbooks will say that, during brainstorming, in the early stages, you'll have a lot of variation between team members, but as soon as you come up with a concept, everyone is on the same page," Agogino said.

"We didn't find that," she added. "We found that even when they're developing the product, there's high levels of variation in how they communicate. You wouldn't think high-performing teams would disagree so much."

The research reflects the principles taught at top-tier design firms. At those firms, she said, designers are encouraged to fail often so they can succeed at the end. They get feedback early that will force them to change their designs.

"You're always challenging your assumptions," she said. "You may get feedback from your customer, manager, or a design coach. You're willing to go back to ground zero if the feedback is strong enough to do so, and you need to make a change."

With research in hand, she and Song hope to move that product design concept out of the top-shelf firms and into the everyday classroom and business environment.

Return to Index

© 2005 by The American Society of Mechanical Engineers