Jamie McDonough has been planning meetings for nearly two decades. And since his very first program, he's been searching for a reliable way to measure the effectiveness of these meetings—to demonstrate their importance to the organizations that hold them.
"As a company, we've been using methodology to evaluate satisfaction with meetings for more than 19 years," says McDonough, manager of meetings and training for Webster, NY-based Fusion Productions. But these days, McDonough says, he and his colleagues are after something more: concrete return on investment. It's become increasingly important to prove that the money put into meetings is money well spent. "Now, especially as more procurement departments get involved, the objective is to express the ROI in dollars."
ROI is not a new concept to the meetings industry, but it is an elusive one. For years, the term has been used so loosely and in such a wide variety of contexts that it's thought to refer to everything from the ergonomics of chairs to the relevance of the keynote address. And while some of these factors affect ROI, none singularly defines it; rather, return on investment measurements must consider all the aspects of a meeting to arrive at an explicit and definitive value.
The good news: After years of searching in vain for a reliable method with which to calculate ROI, the meetings industry finally seems poised to adopt an effective methodology that could demonstrate the value of meetings. And that could finally clearly reveal the value of the meeting professional as a strategic member of any organization's executive team.
SPEAKING THE LANGUAGE
"The term ROI has been bandied about and misunderstood in this industry for years," says Doug McPhee, national account manager for Conferon in San Diego. Although the concept of ROI always carries with it a sense of significance and urgency, it's remained largely illdefined. As a result, the very mention of this topic frustrates some planners. "I'm so sick of hearing ROI, ROI, ROI!" complains Loretta Lowe, an independent planner based in San Francisco. "Many meeting industry associations pay lip service to the discussion but don't really give us tangible samples of reports to management on this," Lowe says. "Most of the talk I've heard about this has been vague," agrees Neil Schwartz, who plans meetings for the California Credit Union League in Rancho Cucamonga, CA. "Most people really don't know how to proceed with the topic."
Indeed, many efforts to measure ROI in the meetings industry have dealt with what are called "intangibles"—such as whether or not the attendees and organizers felt the meeting was a success, and whether it improved employee morale, communication, and satisfaction. And while these benefits, far from being exact measurements, are part of a meeting's return on investment—in terms of both time and money—they are only parts of a much larger whole.
To promote a more complete understanding of that whole, Meeting Professionals International (MPI) has recently partnered with the Birmingham, AL-based ROI Institute, an organization that has been refining the measurement of ROI in various industries for some 25 years. "We initially developed our ROI methodology for development and training, and it quickly grew to include the area of human resources," says Jack Phillips, ROI Institute chairman. "Since then, the methodology has migrated into the areas of quality control, technology, organizational development, and consulting. The meetings industry is a fairly new application." And while several firms currently offer services to help planners measure ROI, Phillips' methodology appears to have achieved the most traction in the meetings industry, where he says it enjoys a particular relevance.
"The meetings industry has escaped this level of accountability for a long time and we've seen budgets come and go with the economic changes. In good times, there are lots of meetings; in tough times, budgets are cut because executives don't know for sure that they're bringing value," he observes. "There's been a disconnect between the logistical planners and the content and strategy developers. How do you bridge that gap? ROI." In other words, when planners employ a methodical approach to measuring ROI, they will inevitably be involved in planning not only the meeting logistics but also the meeting content.
Phillips' methodology introduces a concrete, systematic approach to ROI measurement in place of what has in the meetings industry traditionally been more of a touchy-feely proposition. "This takes it to a completely different level than we've been used to," says McPhee. Indeed, Phillips' methodology enables planners to calculate a precise dollar figure to apply to a meeting's bottom line.
"One of my favorite sayings is that bean-counters need beans to count," says Julia Rutherford Silvers, an author and event planning expert based in Las Vegas. "So when planners talk about having difficulty communicating with their procurement departments, the answer is we've got to show them the beans. We've got to put the numbers before them." And that's exactly what Phillips' methodology is designed to do. "The idea is to take the analysis all the way through to the end result, which is an actual ROI calculation," says McDonough. "And that allows us to speak the language of the CFO."
FIVE EASY PIECES
Phillips' system emphasizes the need to set objectives for return on investment before the meeting, and then determining whether those objectives were met. This process is separated into five levels.
Levels one and two: These first two levels deal with essentially the same information covered in the session evaluation forms that for years planners have been asking attendees to fill out at the end of meetings. The first level involves collecting feedback, or what Phillips calls "reaction data." Here, planners should find out attendees' satisfaction with the meeting's environmental and entertainment aspects. They should also determine whether attendees considered the meeting relevant to their own professional development. "Ask if the attendee plans to use something learned at the meeting," says Phillips. Level two involves what he has termed "learning objectives": Planners set objectives for what new skills or knowledge the attendee should gain from the meeting.
Traditional meeting evaluation has generally stopped at that point. But in Phillips' methodology, this is just the beginning. Unsurprisingly, it's from the second level onward where "planners haven't stepped up to the challenge," Phillips says.
Level three: Next comes "application and implementation." "People who leave the meeting should do something differently, start something, or change something," Phillips says. "Even if it's nothing more than following up on a networking contact and trying to build from that. So here, we like to set up some application objectives, such as what we want attendees to do differently."
Level four: Called "impact objectives," this level evaluates how what was learned or gained at the meeting actually affected business practices. "When you improve something about a person, you can determine the value of that," Phillips argues. "If it's a salesperson getting one more sale, we know the value of that sale. If it's someone getting more work done, or saving time, we can calculate how much that's worth by how much his or her time is worth."
Level five: Only after the first four levels is the planner actually prepared to calculate return on investment. Phillips defines ROI as the monetary benefits compared to the actual, fully loaded cost of the meeting. This mathematical process yields a concrete numerical figure that translates the qualitative evaluations of a meeting's utility into a quantitative percentage of return on investment. This is the level of ROI examination that Phillips says the meeting industry has lacked.
Phillips suggests the ROI target be between 15 and 25 percent for meetings, but emphasizes that even a zero-percent return on investment on a meeting can be considered a success, because "you got your money back and there are still intangibles gained that you can't convert reliably, credibly, or cost-effectively into monetary terms." Adds Phillips: "Notice that we're not talking about the cost side of the meeting here. Planners already know how to control the cost. This is a way to figure out monetary value in a credible, rational, reliable way. It's totally different."
Clearly, completing this type of thorough analysis of return on investment is a time- and labor-intensive process, so much so that it doesn't make sense to complete the process for each and every meeting. "Whether at an association or corporation, only a few of all meetings would warrant this kind of scrutiny," says Phillips. "Typically, the larger meetings, and the higher-profile, more expensive meetings and events, would elevate to this level of analysis." McPhee concurs: "Any meeting whose value is questioned would qualify, including incentives and familiarization tours." For the meetings in question, objectives must be set months ahead of time and impact must be traced for months following. "You have to be willing to say, 'Let's step back, take six months, and find out if there is value in this meeting,' " explains McDonough. "But if I'm a Fortune 500 firm and I do a couple of hundred meetings per year, I couldn't possibly do a full ROI calculation for each one." So one must carefully choose when to conduct a detailed analysis.
A NEW DIRECTION
The significance of introducing this kind of methodology into the industry goes beyond the value of individual meetings. To demonstrate explicitly and accurately the value of meetings is to demonstrate the value of the meeting planner as a professional. "This process forces the meeting planner to challenge those who are asking for the meeting," Phillips says. "The planner must ask, 'Why are we doing this? What do you expect to come out of this? What do you expect the attendees to do differently after the meeting?' " Proponents of introducing ROI methodology into the meetings industry argue that it gives planners the proverbial seat at the table—the opportunity to serve as more than just logistical coordinators within their organizations. "Rather than just being looked at as logistical managers, we're trying to be more strategic," argues McPhee. "It's a matter of meeting planners becoming meeting professionals and really having a stake in the value of their position within the organization."
Monica Myhill, an independent planner based in Littleton, CO, suggests that the growing emphasis on ROI will change the way planners approach their work. "I've observed that planners are mainly concerned with doing evaluations of their cost-cutting rather than getting involved in the content of the meeting," says Myhill. "But once you have that dialogue with the more senior-level management of your organization, once the meeting planner comes to the table with an understanding of the goals and objectives behind the meetings, then that planner can clearly impact the bottom line of the organization."
Of course, not every meeting planner has jumped on the ROI bandwagon. "Sure, we could create another position in our organizations to develop fancy matrices, create spiffy-looking reports, track sales, do pre- and post-education examinations, do an annual report on spend by vendor, and so on. But how necessary is this, really? For me, as a planner, it would increase my already high workload," argues independent planner Lowe. "Let's get away from this ad nauseum use of 'ROI' and start teaching our future industry professionals customer service, technology, and logistical meeting management skills."
Others fear that a growing emphasis on strategy will devalue their logistical skills. "It's encouraging that our expertise is now being included in the planning process rather than us simply taking orders and implementing," says Mary E. de la Fe, conference coordinator for the Department of Community Revitalization for the National Trust of Historic Preservation in Washington, DC. "However, I would not want to see the important job of handling the logistics be somehow 'lowered.' "
The backlash doesn't surprise ROI Institute's Phillips. "Meeting planners are attracted to this type of work for reasons other than this sort of measurement. In fact," he says, "it's often distasteful to them to think like this. The job has been more event planning than anything else—the right ambiance, food, services. And that's all very important to the experiences and you want that, but from a business perspective, you have to work on what happens afterwards."
Finally, there's the cost barrier. "It may cost the organization anywhere from $15,000 to $80,000 to do a complete ROI analysis," concedes Conferon's McPhee. "But if you're talking about a $6-million meeting, then that amount isn't very much on the line-item budget, particularly to see the value in that meeting in the first place." So, it seems, each organization must determine the ROI on its ROI analysis. Still, many planners consider the process more than worthwhile. "Sure, it can definitely seem daunting because it's very detailed," says independent planner Myhill. "However, the more you learn about it, the more it makes sense. There is a procedure to follow, which is reassuring. Yes, it can be costly, but it's worth it to spend less than five percent of your total costs to truly find out the value of your meetings and events."
In the end, meeting professionals themselves must choose the direction in which they want their career—and industry—to grow. "The history of this profession has been much less sophisticated than this. It has been all about taking care of the details," argues McPhee. "Well, that's really not looking at the big picture. We have consider the value of our position in terms of the overall business goals of the organization."
A Case in Point
Jamie McDonough, director of meetings and training at Webster, NY-based Fusion Productions, cites one of his recent ROI projects—for a corporate franchisee conference—to illustrate that getting the right data is all about how you ask for it.
"First, we did a lot of marketing to attendees regarding our collection of preconference data. We wanted them to understand that the questions we were asking them were going to impact directly what they were going to see at the meeting. So we sent out 800 questionnaires and got 600 back. Next, we scripted every one of the workshops and the general sessions in a way that reinforced that the content included came directly from their needs assessment surveys, and therefore we needed their evaluations on sites, too. In other words, the message was, 'You said you needed it; we delivered it; how did we do?'
"We sent 80 invitations for focus groups to meet on site throughout the conference and also to allow us to monitor their business practices for the next six months. We received a 100-percent positive response rate to the invitations. For the follow-up, we wanted to review their business records and conduct periodic phone interviews with those attendees. We told them that along with the economy, we expect their business to go up anyway, but can they identify a percentage of that growth directly related to what they learned at the conference? Finally, at the same time, we had a small control group of people who did not attend the conference whom we also asked about the state of their business processes.
"Although the ROI methodology is very methodical, it's important how you involve the people who give you the feedback. To achieve a higher response rate, you've got to let them know it's all about them, and all for them."