Statistics+Policy=Impact: Meet Hub LA Member Norman Gilmore

As we focus on Civic Innovation this month, it seemed obvious to learn more about the work Norman Gilmore is doing in a professional capacity as well as in his civic engagement in his neighborhood council. I asked Norman about the relationship between statistical modeling and civic engagement - how can we best use statistics and forecasting to affect policy and the decisions that our institutions make? Don’t let that intense question scare you away from reading this post, though. Norman managed to explain the extremely complex systems he’s working with in a palatable and surprisingly understandable way. If you’re interested in the relationship of data and policy, Norman recommends watching this talk from an LA Times Data Desk meetup (pt 2 and pt 3).

Hub LA member Norman Gilmore.

Could you describe what’s involved in making statistical forecasting accessible and understandable to stakeholders, companies, and foundations?

Although my company is TeamForecast, I actually define my research as “collaborative futures modeling”. I thought if I named my company “TeamModeling”, people might get the wrong idea, like maybe I’m developing Tyra Banks’ next reality show.

Also, predicting even narrow aspects of the future accurately is mostly the domain of physicists, astronomers, and Nate Silver.

So allow me to reframe your question as a broader one about effective use of statistics in general. To that I would offer the work of Doug Smith and Ben Welsh at the Los Angeles Times Data Desk. They have done major stories on value added teaching scores, doctors who over-prescribe to addicts, and Los Angeles Fire Department response times, to name only three.

What distinguishes a Data Desk story from other stories is of course the collection and analysis of a data set, the discovery of patterns, and the generation of hypotheses of what narratives might explain those patterns. Importantly, the data is explained not only using statistical charts and possibly infographics, but also with anecdotes that provide context and the potential for empathy with people who have participated in the system analyzed, as decision makers, beneficiaries, or victims. When you imagine a heart attack victim waiting 15 minutes for emergency response, you are able to viscerally identify with the risks and outcomes implied by the system.

All of these stories caused a legislative or executive response at the city or state level so I think the Data Desk does a great job of using statistics to effect policy change.

Let me suggest that if you stripped the statistical data out of those stories, the visceral impact of the selected anecdotes might be as strong, but the policy impact would not be. Ultimately policy makers are allocating finite budget resources, and they know that in general it is unwise to base decisions on a few outliers or anecdotes that were chosen explicitly for their drama. When a budget change can plausibly be associated with improved outcomes justifying the investment, then I think policy makers are more willing to act.

I also believe that these stories would generally have not have been commissioned by the subject of the story! Most people don’t like the use of statistics to measure and assess their performance. People do seem to like statistics to measure, assess, and control other people’s performance though.

So if someone said to me that they wanted to use statistics to measure performance within their organization, that is a political problem first, and a math problem second. Organizations that are analyzing their customers have a lot of frameworks, tools, and MBAs to work with. Policy and advocacy organizations that are analyzing outcomes outside their span-of-control have much more freedom to follow the data, because a negative interpretation does not lead back to the decisions of their leaders. Of course, data is harder to come by when the organization is attempting to analyze data outside its span-of-control. So these are just examples of organizational contexts that influence receptivity to statistical decision making.

How can those of us who are not trained in statistics understand data in which we have so much at stake?

There is a lot of dry, dry material when you google “statistical literacy”, so let me try a couple suggestions of my own. First, it’s just very important to make sure you understand the difference between correlation and causality. Correlation can be easy to establish, but causality is usually much harder to establish to a high standard. By way of example, it’s easy to correlate smoking with some types of cancer, but the industry was able to delay regulation by asking for a high (essentially overwhelming) standard of causal proof. (In fact, this is a common tactic of industries fighting statistical evidence, whether it be anthropogenic global warming or childhood obesity.)

Second, it’s important to understand the sample size of a study, in relation to the size of the population that the study claims to draw inferences about. You should understand the importance of randomization in selecting samples. And you should evaluate if the study is documented well enough that a third party could repeat it and get a similar result.

And that goes to trust. Most of us are never going to read past the executive summary of the Intergovernmental Panel on Climate Change reports, and so we must decide who should be in our aggregated and distributed trust relationships. It’s a hard problem given our finite cognitive resources.

Edward Tufte’s books are an enjoyable introduction to learning about the presentation of statistical data.

In your experience, how have you learned to quantify impact?

First you have to choose the outcome measures of interest and devise measurement instruments and processes. (surveys? lasers? bank balances? thermometers?)

When cost of execution of a plan is low, the intervention or plan should just be implemented in a pilot and outcomes measured. If the outcomes are good, scaling up is your next problem.

If cost of execution is high, then you will probably invest a lot in modeling to estimate impacts of various scenarios. The general challenge here is to avoid group-think by preventing dominating personalities from shutting out diversity of thought in the scenario planning.

A third option is a closed-loop system, which is a decision process that can feed outcomes back into decisions in real time. This can occur when outcomes are directly traceable to decisions and outcomes can be measured easily and routinely. By way of example, it’s now common for online advertising platforms to have a closed-loop ad-placement capability, where the outcomes measured are conversions (e.g., clicks or purchases).

As an aside, I think any growing organization contains closed-loop processes implemented by its internal rules, organizational structure and IT systems. That means I think that growing organizations are able to measure and communicate impacts back into their decision makers.

What impact do you hope to make with your work?

I’m most interested in modeling situations that cross organizational boundaries.

You also sit on your neighborhood council. What issues are most important to you locally?

I’ve spent a lot of my time on outreach, land-use, housing, historic preservation, and parks. I was fascinated/annoyed by the blight on our neighborhood boulevard and wondered what it would take to re-vitalize vacant lots and our under-utilized park. What were the blocking issues? What were the actions that could be taken? Our neighborhood is fortunate in having had some nice wins in those areas in the last few years.

How has your work doing interactive modeling informed your civic engagement?

I’d say it is the other way around. My civic engagement has made me more interested in modeling public policy issues. For example, I think it would be fascinating to develop a regional model of the individual, governmental, and institutional costs of homelessness and analyze the effectiveness of various interventions. But I haven’t looked for a funder yet.

Leave a Response

Name (required)


Email (required, but won't be published)


Website


Comment