By Konstantina Slaveykova, DeLange Analytics Collaborator, Analytics Professional & STEM researcher
Social listening and communications analytics rely heavily on automation. Quality tools are а necessity to navigate through a sea of thousands or even millions of online data points. More often than not, the real challenge is not the lack of data but too much of it.
Choosing the right analytics provider can make all the difference. Bridging analytics and consultancy is the key to adding value. Great tech tools are always an advantage, but an experienced team of experts is what can add the most value at every step of the way: from choosing a platform to extracting qualitative insights and spotting data discrepancies.
The crucial difference here would be between reporting and analysis. As Brent Dykes from Adobe points out: “reporting” and “analysis” are not interchangeable terms. The key difference is about depth. Reports summarise information (What happened?), analyses go beyond the data, drawing connections, uncovering insights (Why? How? What can be done?).
Analytics projects are only as good as the data they are based on. In computer science, this concept is also known under the sobering acronym GIGO: “Garbage in, garbage out”. The initial choice of search parameters and the analytics tool you use can dramatically affect conversation volumes, share of voice between competitors and just about any of your key metrics.
Еxperienced analysts are aware that each analytics platform has pros and cons in terms of scope, depth and data quality. Replicability is an issue: running the same search in different platforms (I will call them A and B as my point is to look at the broader issue, not the specific tools) can bring majorly different results. For example, if I run a straightforward search for (Netflix AND service) in the last 30 days in Australia, Platform A shows 1K mentions (9.6% positive vs 17.4% negative), and Platform B detects 2.3K mentions (11% positive vs 26% negative). In other words, Platform B captures more than twice as many mentions! But is this necessarily a good thing? Platform A could be showing fewer mentions because it is better at filtering out low-quality spam content, not because it covers fewer outlets. Adding value to the client would mean investigating such questions and making transparent data choices.
Apart from volumes, there is also a discrepancy in sentiment ratios. In fact, this is to be expected. Each major analytics platform has its own team of natural language processing (NLP) researchers and in-house solutions for semantic analysis. When platforms create their own mix of machine learning (ML) techniques, mathematical models and manually compiled rules, they are not striving to produce similar sentiment results. They are striving to gain a sizable advantage. Differences between sentiment ratios across platforms are not a fluke: they are a crucial part of the edge each player seeks over the competition. There has been some improvement over the years, but real projects rarely reach the accuracy levels of the controlled experiments used to assess performance.
An experienced analytics team can add value in two ways. The first one is through consulting the client on which platform to choose. Savvy consultants not only know the pros and cons of the major players on the market: they are also in direct contact with the representatives of these online tools. They can negotiate better pricing and offer cross-tool comparisons based on their experience.
Another way to add value (especially for long-term projects in which quantitative accuracy and dashboards are essential) is having a data scientist on your team who is savvy in topic modelling, text mining or similar fields. Salespeople always try to dazzle, but a person with hard skills in this field will know how to ask the right questions and evaluate whether a tool can deliver on what it has promised. Last but not least, if you are working with datasets rather than analytics tools, an in-house data scientist can also work on a customised topic modelling solution for you. Often industry tools miss the mark because, by definition, they cast a wide net with their settings. And they keep some of these settings a trade secret, so you work with a black box: the inner workings are not transparent or available for modification. In-house topic modelling could let you fine-tune the settings for semantic analysis and achieve better results.
It is a bit of a false dichotomy when we pit technology vs human analytics. Technological solutions are, after all, engineered by humans and based on human decisions on how to model, organise and visualise data. The main issue with such solutions is that they can never anticipate all the subtle nuances inherent to your client’s specific project.
That is why human input is indispensable. However, as was mentioned earlier, this input can vary in complexity. A human analyst could read through the data and summarise the main topics discussed within a specific market and still offer a report, not analysis. The client does not need to pay for an analyst to tell them that “Conversations about Netlflix focused on TV series”. That is a summary statement. Adding a sentiment evaluation like “Most of these conversations were positive” is still a report, not analysis. Assigning sentiment is a form of categorisation, but analysis implies an even deeper level. Soon, after a few more tweaks to topic modelling settings, AI will be able to generate such summaries with pretty decent accuracy.
Adding value through asking further questions is what turns a report into an analysis. Did German audiences simply discuss TV series on Netflix or did they value the quality of local productions on the platform higher than series on regular TV? Was this also reflected in other local data: e.g. ratings, critics reviews, etc.? Was it a general trend, or did the local show Dark shift interest and boost the streaming of German-language vs global content? These are much more interesting questions. It is also vital to have analysts who specialise in specific markets. This does not refer only to language skills but also to following pop culture, general trends, local stories.
Context is a crucial contribution to adding value. Without domain knowledge of the specific topic and cultural understanding of the market, even an analyst with excellent language skills can miss essential details. This is especially valid for projects in highly specialised industries like finance, medicine, engineering, where domain knowledge is crucial.
You can also add value to the in-house team documentation by keeping an analytics log in which experienced team members can store valuable context details about a market or an industry. This can help new team members can be up to speed and can make team communcation easier and more structured. Endless chat threads with the same or similar questions with every new project are not an optimal use of time: especially in large teams with many analysts. Having a centralised document with details on a particular industry built through projects can help both old and new analysts understand a field in more depth and notice valuable insights more quickly.
The critical evaluation of data sources is another way to add value to an analysis. When an experienced analyst spots a blog with an archive showing 344 posts just for one month, they can immediately flag it as a spam blog created solely for the purpose of passing link authority to target sites (also called spamdexing).
Quickly spotting irrelevant sources like this can save hours of manual coding work (if working with a dataset) or help platforms weed out low-quality sources (if you have an account there and give them feedback on what sources to remove).
Even if the source is reliable, an analyst can add an important layer of information by addressing the biases and affiliations of the media sources influencing the conversation. A cluster of right-wing social media accounts would naturally have one sort of agenda in syndicating information compared to a left-leaning cluster. A respected industry media whose bottom line would be damaged by a competing new technology would be more likely to post critical opinions. Is the negative conversation about your client stemming entirely from small media outlets owned by this industry leader? Insights like these can be crucial.
Another way to add value to the client’s project is to have a holistic approach to selecting opinion leaders and influencers. 2021 is not 2012: there is substantial influencer fatigue among audiences across the globe. According to Goodhart’s Law, “When a measure becomes a target, it ceases to be a good measure.”. In simple terms: people know that brands pay top dollar for influencers with many followers so they trust them less. And many influencers buy followers to mimic social influence and work with brands.
Is the influencer followed by 10K people suggested by your automated tool really followed by that many people? A human analyst can use a tool like Sparktoro to check for fake followers and perform an account audit. Auditing a social media account takes a critical look at the content, the types of followers and overall online behaviour of the author. I recently rejected such an account, apparently followed by thousands of people, most of which turned out to be bot accounts sharing the same links to the same homeopathic product. It is not just a matter of “their audience is actually much smaller (if any)”. It is a matter of recommending authors with integrity who can be trusted, quality ambassadors for your client.
That being said, not having that many followers should not be a dealbreaker if you know your client’s goals and needs. In some specialised fields, especially B2B businesses, industry opinion-makers could have substantial influence offline and just a modest online following. Is your client explicitly looking for high online reach? Or would they benefit more from contacting a representative of an industry organisation who has only 300 followers on Twitter but organises industry events for thousands in the target segment?
Context and human analysis are crucial to making sense of data. However, overreliance on qualitative insights can be risky in a team without trained quantitative analysts. We know from the Nobel-prize winning work of psychologist Daniel Kahneman and his colleague Amos Tversky that the human mind is subject to a wide range of biases and mental shortcuts (heuristics). And you know what is the problem when a human analyst has to comb through thousands of mentions to extract an insight? We don’t just resort to biases: we are almost entirely reliant on them.
We assume that if something “sticks out” in the conversation, this is an indication that it is important. But does it objectively stick out (a peak of 3K mentions on a specific topic), or does it subjectively seem so? This is an especially relevant point in desktop analysis supplementing social findings. Even if the researchers take care to use incognito browser windows, the results they get and the takeaway findings can be insightful but need to be backed up by further quantitative information to hold their ground. Some conversations could be outliers (rare, differing from the general conversation), but if they sound incredibly engaging and insightful, a qualitative analyst could prioritise these findings over much more salient but less memorable trends. This is ok if the outliers supplement the actual trends, but it could be a problem if an analyst makes it seem like this is a much bigger trend than it is (imagine paying for a campaign based on such data!). That’s why we need to combine quantitative and qualitative approaches.
It is not humanly possible to read through tens of thousands of social media posts. So quantitative help is needed. But without special training, there are always risks of misconceptions.
For example, one of the most widely spread ones is conflating a large sample with a representative sample. A sample of 10 000 can still fail to be representative if it does not accurately reflect the characteristics of the sampled population. Understanding how biased sampling affects the representativeness of your data and the conclusions you make is also important. Analysts should avoid sweeping generalisations and the phrase “this proves” at all costs.
Also, apart from Google, few analytics tools normalise their data. Comparisons between non-normalised datasets is an issue: are German users really talking less about this topic than French users? Or does the tool simply harvest data from fewer German source? You should be very careful with making major statements if the tool you are using does not normalise data and use the relative proportion of the captured conversation per country. Quantitative training is essential for understanding such caveats.
Data is beautiful. And complicated. Adding value requires a holistic approach to analysis. It is a mix of understanding your client’s needs, selecting the best tools for your purpose, offering a deeper context through qualitative analysis and never disconnecting from the data.
Dedicated to the 25th annyversary of AMEC in 2021