Researchers everywhere are looking to capture the experience of life right now, but what will those studies tell us?
A virus is not the only epidemic sweeping the planet: so is opportunistic data collection. With over half of the planet under some kind of stay-at-home order, why not take advantage of that extra screen time to gather some data? Maybe a survey or a Zoom chat or Twitter poll?
I’ve noticed a big spike in surveys and data gathering initiatives in recent weeks (something my peers have confirmed anecdotally, although I suppose I could have simply done a survey). But just as my anecdotal evidence is problematic to base policy decisions on, so is much of the data gathered right now.
Context Counts…For What?
One of the principal issues associated with any data gathering is understanding the context in which it is gathered. Right now, that context is highly dynamic and exhibiting VUCA (volatility, uncertainty, complexity, and ambiguity) characteristics. I’d go further: we don’t fully understand the context we’re looking at.
What we are seeing is something that is unprecedented in its scale, scope, and reach even if much of what is happening has happened before. It’s just not happened together. It is, as my good friend and colleague Peg Lahn put to me: A Flock of Black Swans.
If we are unable to describe the context in which we are gathering data in ways that are meaningful to making sense of what is going on, how can we account for it in our data collection and sensemaking?
Is It Relevant Now?
For those looking to evaluate programs the question we must ask first: Is the program relevant now? Or consider the opposite: food banks. The relevance of food banks has been transformed as the need has spiked. In Toronto, libraries are being converted to food banks to address the need. Food banks have never been more relevant to more people in more ways than they have right now.
All one has to do is monitor the attitudes toward things such as social isolation measures, stay-at-home orders, taxes, and personal freedom over the spring and we’ll see major changes in the way people understand, appreciate, and feel about them. As we move into a new phase of the global pandemic, these will change, too.
Envision a scenario where the lockdown restrictions are lifted and freedom of movement is back again. How will people feel? What happens if — or when — COVID-19 rates increase and restrictions are imposed again? Will we experience these the same way the second time around?
What are we to do with this data once we have it? What will we see and what will we believe?
Believing to See
I’ve commented before that sometimes we need to believe something in order to see it. Our mental models can prevent us from seeing things in plain sight. Dr Anupam Jena has explored this very thing with natural experiments such as when cities like Boston shut large parts of the city down for its famous annual marathon and how it affects heart attack survival (answer: there’s a 15% mortality increase on a race day).
As he puts it:
Seeing is not the same as looking.
Dr. Anupam Jena
When we don’t understand the context or its entirely new we look for what we know. The challenge right now is that we don’t know what it is that we’re looking at. Unless our research or evaluation work is focused on the now and understanding how and what we are doing in this moment, about this moment, and for this moment we are running the risk of developing data that we look at through the lens of history, which will be another context altogether. We’ll be making sense of the past through the lens of today — something we’ve done many times before.
We will have seen, but what exactly will we believe?
Photo by Andrew Neel on Unsplash