Monday, April 06, 2015

Three Reasons Anecdotes Often Distort Reports

Probably as an attempt to make an already interesting chart in a report I read last week seem even more interesting, someone penciled in the names of a few events.

The notations seemed to suggest anecdotal explanations for a change over last year in passenger growth at the airport jointly owned by Durham, North Carolina, the community where I live.

This is often done in reports made for general consumption either through a lack of understanding of data analysis, or as I will show later, due to the discomfort human beings have with randomness.

Some of the notations in the airport’s report made sense, such as when a significant number of flights were cancelled over the course of several days due to bad weather.

Some made less sense such as when a few direct west coast flights were added.

And notations attributing increases to community events made no sense at all.

While the events were significant to the communities where they were held and surely filled some airline seats, it is questionable that they were each the reason for a bump in traffic for two months.

If they were, the increase in those months would be reflected by a drop in the month following.  Instead the percentage of increase continued without the events.

An “inconvenient truth” for communities hosting the Super Bowl, the mother of all mega-events, is that sales tax collected across those dates varies little from the year prior and the year after.

There are at least three reasons mistakes such as these occur.

Often it is deliberately done as a community relations gesture to please a few stakeholders or even more often, to help justify the hyperbole they have generated around an event. 

Humorously, one community event proclaimed one year that airlines would add flights due to an event to be held shortly after a new terminal was set to open.

The news media is complicit in this hyperbole or else a follow up story would have noted that neither the number of flights nor passenger load factors increased, signaling that the event may have created displacement which would have also lessened its overall economic impact.

Seth Godin recently wrote, “Human-beings are story-making engines, and when we’re confronted with randomness, we make up an egocentric version of what happened and it involves us.”

Applicable to the explanations penciled on that chart I read, he continues with, “Here’s the truth.  There is no reason.  That’s why we define it as random.”

More on randomness as it involves analysis of data later.

I saw the fascination people have with anecdotes during some of my board meetings in my former life in community destination marketing.

Often after seeing a change in a metric that had been generated such as aggregating and comparing hotel occupancies community-wide, a member, seeking an anecdote-fix, would inevitably turn to another who operated a hotel and ask “Is that what you saw?”

Others would leap to conclusions that it was due to a major event,  even though it would take an event many times larger or a phenomenal increase in events overall that month to move that particular needle, and by doing so, would create displacement.

Of course, even in communities that also had access to this data, a community-destination organization (DMO) may be complicit if it is one of those places that subsidizes events to be held there.

Even where they should know better, this can often happen because organizations in a community want to please those excited by the event, but more often it is because they are careless with data analysis or just want to sell sponsorships.

This has happened in the last few weeks as communities hosted NCAA basketball playoff games across the country.  Often, to calculate impact, they would factor in hotels rooms used without deducting for the percentage that would have been filled anyway.

Or they would extrapolate spending estimates without factoring in the number of day-trip vs. overnight attendees and/or without factoring attrition as teams lost and fans returned home when it was too late to retrieve visitors who had been displaced by the event.

Communities are often guilty of grossing up estimates without netting out displacement and leakage when displacement pushes the impact off into nearby communities, or worse, crowds out the flow of air travelers who would have visited those communities for other purposes.

News reporters become enablers either out of enthusiasm for events or because they do not understand the nature of economic impact, unwittingly misleading officials in other communities to go down the same path without rationale based on good analytics.

The reason misattribution of impact occurs involves data analysis itself.  A white paper entitled “The 5 Common Mistakes People Make In the Name of Statistical Analysis,” is available from Mu Sigma, one of the world’s largest decision sciences and analytics firms.

One is “extracting meaning out of randomness.”  Data analysis is the process of looking for meaningful patterns.  This process involves isolating patterns from “noise” which in statistics refers to a “random phenomena” that is “beyond the control of your organization or even known external factors.”

So the penciling on that chart might be a case of what data analysts call the “Problem of Overfitting.”

This is just one observation from an otherwise excellent report that if harvested by any of the five dozen destination communities pushing and pulling traffic through the airport will be immensely valuable, especially to community-destination marketing organizations.

More than half of the airport enplanements are visitor-related.  The report breaks down the top market areas for 2014 including 610 passengers daily via the airport in Boston, 590 via New York-LaGuardia and another 330 via New York-Kennedy.

Those new flights to LA and San Francisco added to non-direct flights carried 594, probably not enough to make a dent where penciled in as the reason for a bump in numbers overall a year ago.

More than 9.5 million passengers transit through RDU International Airport during the course of a year.  That’s about 26,000 a day, 13,000 of whom are visitors either arriving or returning home.

Even for Durham which draws 33% of the visitors who arrive via RDU, a greater proportion than any other city and the equal of Wake County which includes Raleigh, air travelers represent only 8% of visitors traveling from at least 50 miles away, a point shy of the national average and double that of North Carolina.

But from the time I took my first International flight in 1967 as a college kid and when I retired in 2009 after a four-decade career in community destination marketing, the metric for passenger revenue miles worldwide has increased by 18 times.

But the metric for global air cargo miles had increased by more than 22 times.

Of just as much significance to North Carolina's brand though is that RDU represents 5,000 acres, including impeccably maintained natural areas along roadways maintains.

But half is open space unusable for any future aviation needs.

There will be incredible pressure, unfortunately some of it created by airport stewards, to convert much of that land from open space, trees, streams and park-like land into development and impervious surface instead.

Hopefully, any thinking in that direction will factor in the ecosystem services provided by that open space, giving as much consideration for green infrastructure as tarmac.

Very few cities encompass an airport.  Instead, most airports, such as RDU are surrounded by numerous, distinct cities.

The surrounding cities draw travelers through the airport doubling the size justified by their collective populations.  But on average, airports also distribute a third of the value of all of the products we consume.

RDU is at the center of an Aerotropolis, a term coined by Dr. John Kasarda who heads the Center for Air Commerce at UNC-Chapel Hill.

As such, RDU is well advised to remember that it is also the town square for this vast, multi-city Aerotropolis

No comments: