As it adapts to life in the COVID-19 era, the research industry will see accelerated adoption of new technologies
A version of this article was published in the May/July edition of Research News by The Research Society (formerly AMSRS). Actually, only a passing resemblance. The earlier version focused exclusively on trends relating to AI and analytics software. This version has been substantially revised.
That Bloody Elbow Chart
If you read The Sun Also Rises by Ernest Hemingway, you’ll find the following exchange between two of the characters:
“How did you go bankrupt?” Bill asked.
“Two ways,” Mike said. “Gradually and then suddenly.”
Gradually-then-suddenly seems to be the defining framework for our times.
Look at any chart plotting the growth of the COVID-19 virus from the first reported case, and you’ll see a distinct elbow: a longish flat line that suddenly bends vertically. Exponential models often work like this: slow progression until a critical tipping point is reached – and then runaway growth.
Around the world, the response to the pandemic has left a similar elbow-shaped impression on hundreds of industries, public sector bodies and charitable organisations.
In China, Alibaba’s grocery delivery app was downloaded 100,000 times in a single day in February – about four times the average daily downloads in 2019. My own father-in-law started using his credit card online for the first time.
But the impact is also being felt in less obvious and more conservative corners of society.
In the UK, the government launched an ambitious plan to digitise justice at the start of 2016. Progress was predictably glacial, and after three years, a grand total of 200 cases a day were being heard via conference call or online video. Over the first three weeks of social distancing measures, that figure rose to nearly 2,000 daily cases.
Remote medical consultations have followed a similar pattern: slow progress for years – despite strong policy efforts – and then dramatic growth in adoption over the course of a few weeks as general practitioners deal with patients using remote video tools.
And so it has been with the research industry.
There has been an explosion in capability over thelast few years; but far less widespread adoption. But much recent innovation has been supply-driven rather than demand-led.
That’s understandable, and natural in mature industries.
The barriers to knocking out a new research platform in your bedroom are orders of magnitude lower than getting a new tool signed off in a large organisation. Ever tried getting through procurement, legal, data security, finance, CRM and brand marketing? Then have some sympathy with those insight managers whose desire to innovate gets squashed by the blob.
But this all means the change curve in our industry has looked a lot like those case numbers in the early weeks of infection. Bumping along near the bottom of the Y axis. A slight, barely discernible, incline in the curve.
And then – bang.
There’s a critical weight of momentum and an inciting incident – in this case a dramatic one – and some research technologies are off like a rocket.
But which ones are temporary artefacts of this weird interregnum between old normal and new normal? Which ones will survive, and which will revert to their prior niches?
I’ve set out seven areas below. Like online shopping, remote court hearings and doctors’ appointments via Skype, I don’t see the research industry turning the clock back.
Suppliers in these areas are living through be-careful-what-you-wish-for levels of sales and activity. They’re growing, hiring and trying to remember what weekends were like before the pandemic.
1. Automated Survey Research
As a quick aside, I really hate this label. I apologise to those platforms I’ve force-fitted into this terrible taxonomy.
What it’s intended to cover are those integrated tools that put a ‘DIY’ front end on a survey platform with integrated sampling and online reporting, analysis or dashboards. It’s cheaper, faster and more consistent than the old ways of agency outsourcing.
Why it won’t go back
For automated survey research and all the technologies below, there’s a common theme. An economic shock has cut budgets, forcing buyers to do far more with less – overnight, in some cases. It will take a long time for those budgets to return. By the time they do, the new behaviours will be entrenched.
Automated tools have big speed and cost benefits over agency models. Yeah, no shit.
But what’s really interesting is that these survey-based platforms are gaining strength in three areas that were traditionally high price / high margin for agencies: repeat testing, brand tracking and complex modelling.
First, repeat testing (ads or concepts). Big agencies developed proprietary models (Link, BASES) that were adopted by large businesses and embedded into decision-making processes. When you’ve tested thousands of ads around the world using Link, you don’t suddenly tear it up – no matter what the cost saving or quality improvement.
Next, brand tracking. More proprietary models and scale benefits for agencies who could cover dozens of markets from local operations. Expensive, slow, and of dubious value – especially now there are so many low cost brand indicators in social, search and proprietary data.
Finally, complex models. Even ten years ago, conjoint surveys had to be custom-designed and built by marketing scientists. I personally loved these people when I worked for an agency. You could charge them out at 10 times their cost and clients would still feel they had good value. But even the darkest basements are being illuminated by automation now. There’s no way we could get away with that behaviour now.
Zappi has a been kind of poster child for the automated surveys model. Their success has been at the difficult end of the market: getting large, risk averse corporations to change the way they buy and do comms testing and product testing research.
But there are plenty of other great examples.
Quantilope and Conjoint.ly specialise in automating the hard-to-automate types of survey research: price modelling, TURF, choice-based conjoint. Both are on a tear, hiring new staff through the pandemic
Attest has an automated research platform with a big focus on brand tracking. They just took another dollop of growth funding, and revealed that their volumes have been off the charts since lockdowns began – more than 2.5x the usage their platform would normally see.
OnePulse is a great example of a platform that is lowering barriers to answers. Quick ‘pulses’ (3-question surveys) get answers from hundreds of people within minutes.
2. Online Qualitative Research
Some of us have been doing online qualitative research for years – but the overall growth rate has been unremarkable. Practitioners resisted (“it’s not proper qual”), insight managers dragged their feet (“focus groups are tried and trusted”) and marketing people were too wedded to their free beer and snacks behind the one-way mirror.
And then the virus put paid to all that. Online qual has exploded in the weeks since lockdowns began. The software platforms that support it have never been so busy. It’s a similar story for those agencies and freelancers who run communities, forums and remote video groups.
Insight Platforms recently hosted Mastering Online Qualitative Research, a training course by Tom Woodnutt of Feeling Mutual. It was sold out, and we now need to re-run it for all those people who missed out.
Why it won’t go back
Yes, all the same reasons of cost and speed. But they’re just the triggers for widespread trial while the old methods aren’t available.
Why they’ll stick is that for many research briefs, online methods offer so much more.
I’m not talking about laggy webcam focus groups. They’re pretty shit, and a poor imitation of the real thing. These will be ditched for stuffy rooms in Slough as soon as we’re all allowed back out.
But if you use the native onlinetools and play to the strengths of the medium, you get to places that are really hard to go IRL. Mobile ethnography. Diaries. Scrapbooks. Discussions that last a few days or weeks, so you can keep digging deeper and iterate. Much more honest, emotional disclosures than you get in a viewing room – or even in someone’s home.
People are trying these methods because they cheaper, faster, and – let’s face it – the only option in lockdown. But they’ll stick with them because of the new possibilities and insights they open up.
And Qualie is one of a new breed of tools that straddles the quant/qual divide by combining automated surveys with video feedback and community voting to find the most insightful consumer expressions. I did say you could do more interesting qual work online.
3. ‘Qual at Scale’
This is red-rag-to-a-bull stuff for many qualitative researchers. Qual at Scale is oxymoronic, and it’s not really qualitative research. What this label really refers to is quantitative analysis of unstructured text data. But nobody’s going to call it that.
Why will it grow now?
Because there are so many new consumer behaviours and attitudes we need to understand. And – crucially when budgets are lean – it’s cost effective.
Qual at Scale tools can ask open questions to hundreds of people simultaneously and process the answers on-the-fly.
Remesh, for example, can handle groups of up to 1,000 people in the same open ended discussion. Beyond commercial research, it is being developed by the United Nations as way to engage people in warzones in real-time dialogue.
CRIS (Conversational Research Insight System), part of the Delvinia Research group, is a virtual moderator that uses AI to run qualitative discussions through a chat interface – analysing topics, themes and sentiment on the fly – but can also collect structured survey responses alongside open-ended questions.
4. Trend Forecasting
Until recently, trend prediction was more about cool hunting and glossy presentation than it was about robust data. But advances in Natural Language Processing – the tools behind text analytics, social listening and review monitoring – mean we can now get better forecasts of consumer behaviour using much broader evidence.
Why will it grow now?
We have better signals (more people posting more content across more platforms), better AI models, and increased pressure to provide insight at lower cost.
Tastewise is a software platform for analysing food and beverage trends. The systems scrapes data from social media posts, online restaurant menus and recipe websites. Both the language and the images are analysed with machine learning, and some of the results are even available for free in an online dashboard.
Machine learning algorithms find trends and understand consumer needs from social media, blog posts, reviews and other public sources. Those needs can then feed another algorithm that drafts new product concepts using Natural Language Generation (see below) or the system can score existing concepts based on how well they match needs that have been identified.
5. Self-Writing Reports
Natural Language Generation (NLG) is the process by which software transforms input data into a coherent narrative. NLG models have been used to write product descriptions on websites, poems and even scripts for Hollywood movies.
The Associated Press used to employ writers to summarise the quarterly financial reports of public companies. Today, NLG software creates those stories automatically, populating templates using the underlying data. The results are indistinguishable from those created by journalists.
Researchers have been writing summaries of concept tests, brand trackers and audience panels for years. Much of this work can now be automated using the same tools adopted by the Associated Press.
Why will it grow now?
Because the technology is good enough and affordable; and there will be extreme cost pressure on easily automatable agency workstreams.
Narrative Science and Wordsmith by Automated Insights are software tools that generate narratives from data. They integrate with analytics and business intelligence tools like Microsoft Power BI, Tableau and Qlik to write headlines, summaries or full stories.
Zappi uses NLG to write commentary on its product development and communication testing tools.
Even qualitative research will benefit from these advances. 20|20 Research’s Qualboard 4.0 uses Smart Reply features for qualitative moderation. Like messaging apps, it interprets the content of a post and suggests follow-ups and probes to save the moderator time when replying.
We can also expect to see more automatic summarisation tools like Agolo being used to condense transcripts from qualitative interviews.
6. Visual Insights
Computer Vision artificial intelligence tools are used to recognise scenes, objects and gestures in images and video.
There are many potential research applications: analysing logos in Instagram posts as a proxy brand metric; mining respondents’ in-home videos to understand how products are actually used; even using the content of an anonymised photo library to classify a consumer into a segment.
Why will it grow now?
Lockdowns have seen growth in remote video research and smartphone ethnography. Many countries will continue with on-off social distancing measures, even after strict lockdowns come to an end – so remote video research is here to stay. Compute vision tools will be key to analysing all that video content cost-effectively.
Pulsar is a social listening and audience intelligence platform that uses computer vision for logo recognition and understanding the content and context of images.
QualSights is a platform for remote video observation, interviews and focus groups. The software uses machine learning to transcribe audio; generate keywords and topics; and apply sentiment and emotion analysis. It also uses computer vision to recognise objects and scenes in videos.
7. Behavioural Analytics
This is one is very broad. It includes geo-location data to understand path-to-purchase activity; app and website activity to inform UX design; journey analytics that track customer interactions across channels over time; and dozens of other big data applications.
Why will it grow now?
A bigger share of economically interesting behaviour now takes place online. Even after the new normal kicks in, we will have higher levels of streaming, socialising and buying online than we did pre-virus. That’s a lot more data that brands, retailers and media firms will want to understand.
And – helpfully for marketers – we may see less privacy-based resistance to sharing personal data. Symptom tracking and contact tracing apps will be widely adopted, shaping new norms about what personal can and can’t be shared.
Ogury provides smartphone behavioural insights from over 400 million mobile users in 120 countries. The passive data it collects includes websites browsed and usage of apps. Analytics help publishers and brands better understand their audiences.
Pointillist is a customer journey analytics platform that uses machine learning and predictive analytics to optimise marketing campaigns.
Albert.ai carries out attribution analytics autonomously and then automatically tweaks targeting, media buying and digital execution.
What this means for researchers
Beyond the tragic health impacts, this pandemic has already inflicted some brutal economic damage on the research industry. We’re facing a lengthy recession and slow recovery. Some agencies won’t make it to the other side.
Those who do will have embraced change, adapted their offer and begun using more of these AI and analytics tools.
Here are three things researchers can do right now to prepare for post-pandemic new normal.
- Get learning. There are some fantastic resources online to help understand this new world. Low cost and free e-learning courses. Blogs, white papers and webinars. Software demos and free trials. Soak it all up while business is slower.
- Focus your offer. The days of generic research agencies are numbered. Nobody wants to pay good money for agencies that just manage surveys or run standard groups. Specialise to premiumise. Find the technologies and data sources that will enable you to charge more for your people. And make sure they’re doing the things that AI can’t.
- Clean up your model. Use these new tools to sort out the back office. Process automation, productivity tools, remote working, off-payroll team members – now is the time to make these changes.
Right now, we’re right in the elbow of the curve – the tipping point between gradually and suddenly.
But when the research industry recovers – which it will – the adoption curve for all these new technologies will be near vertical. Preparing for that future now will help you make the most of it when it arrives.
Disclosure: about half of the companies mentioned in this article are Insight Platforms sponsors and partners. You can see the full list here.