Noticed a blog post by Content Analytics on Why analytics companies should stop focusing on "accuracy" in automated sentiment analysis saying we can't really measure sentiment in text - we can only evaluate how someone is reacting - and their emotional state is affected by what was seen / read / experienced just before writing, and that can rarely be measured.
I'm quoted in the post (my post, that is) which I bolded, below.
Discussions on automated sentiment analysis "accuracy" are starting to border on the bizarre. In the past couple of weeks, I've read claims that SAS's new tool can identify sentiment "better than most humans". Just a few days later, I read a post this week claiming that "sentiment analysis [is] best done by humans".
At the heart of this ongoing debate (and confusion) surrounding automated sentiment analysis is the issue of "accuracy"- the degree to which software can correctly extract positive, negative, or neutral tone from text. Using "accuracy" as a criterion for useful sentiment analysis demonstrates a fundamental misunderstanding of what sentiment really is and what "accuracy" really means. Unfortunately, this misunderstanding has led media researchers and software programmers to search for "100% sentiment analysis accuracy", and distracted our industry from what its real focus should be- understanding how the media influences human behavior.
Automated sentiment analysis will never be accurate. Not 1% accurate, 50% accurate, or 100% accurate. To say that an algorithm or statistical model has "accurately" identified a piece of text as positive, negative or neutral requires that sentiment is a real thing in the text that can be correctly identified, like a person's name or a product. The problem is that positive and negative don't really exist on paper or on a computer monitor. The scientists and philosophers who study sentiment all agree that it only exists as property of the animal nervous system. "Positive" and "negative" are neurological states that evolved to helps organisms avoid stuff that can harm them or to promote behavior that's likely to nourish and help them propagate. Sentiment is absolutely not something that exists "out there" in the world; it only exists in our perceptions of the world.
I have noted this several times in an entirely different context - Art - where so much is "subjective" anyway, and wrote a comment the post at Content Analytics (which I assume will be published soon) - here it is, anyway.
Great points and thanks for quoting my post on Sentiment Analysis best done by Humans - which I agree is the best way. What you observed applies in many spheres, including Art.For example, I often note my "mental" and "emotional" state affects my ability to enjoy a museum or art gallery opening.Sometimes, I'm in the mood (or, as you put it - I saw some stuff or felt some things before entering that were supportive and put me in a good state) and I get something tangible out of the experience.Other times, I simply can't focus - my mind is distracted by whatever else is going on in my life or people around me, and I can't contain what I'm looking at - and I don't enjoy the experience.At such a time I might be temped to write a negative account or a say something "negative" about what I'm looking at, that under other circumstances, in another mood, might say something entirely different. [note: if we were to randomly measure response - it would be a "toss" where anyone is in their emotional and mental state at a particular time a comment or blog post was written].So I think we come to the difference between sentiment and opinions that can be "swayed" by momentary considerations - and those that are more ingrained - core beliefs that are unlikely to change regardless of what my mood is or what I just read.
I came to an awareness that our "core beliefs" are less likely to be swayed by what we just saw or felt - and perhaps, what we need to add, if we're doing Sentiment Analysis at all, is some measure (which we probably can't get to without panel based data) of what the core disposition or beliefs are of the those stating an opinion online, and then how far aligned what they are saying is with that belief(s).
Just a thought on a sunny Saturday.
Link to original post