MIT Can Look Into the Twitter Trending Future

Posted on November 13th 2012

MIT Can Look Into the Twitter Trending Future

Trending topics on Twitter are a powerful tool to expand your social base and build a more consistent group of followers. These topics are usually words identified with hashtags in front of them, created organically by the ingenious Twitter world, or just simply by phrases and words. This is particularly useful to brands and social media agencies who try to cut out niches in the market to connect their product or service more effectively. But what if you could predict what will be the next popular trend before it happens? If your friends with Professor Devarat Shah from MIT you might have a chance.

According to an article from Wired, Professor Devarat and a fellow student from MIT have developed an algorithm that is trained to predict those juicy hashtags up to an hour ahead on average. In some cases, it can even predict 4 to 5 hours ahead of the trend phenomena.

There are other types of programs that could in theory try to perform a similar task. Wired continues to explain that, “a predictive program like this would be looking through Twitter traffic and trying to match up what it sees to a certain model. You might program it to look for a certain "step", as one topic becomes more prominent against the general background chatter.” Shah points out that these models are “simplistic” and it is unknown if trends have a step function that would lead to a correct prediction or any prediction at all. To put it in laments terms, a step function is a mathematical function with only a finite amount of pieces.

So what makes Shah’s algorithm special? Wired states:

“The algorithm was trained by Shah and Nikolov using a training set that contained 200 topics that did trend on Twitter and 200 which didn't. They let the algorithm get to work, and it managed to pull out the successfully trending topics from the unsuccessful ones with 95 percent accuracy, with just a four percent false-positive rate -- i.e. topics that were predicted to trend that then didn't.”

To further explain what this means, Shah’s algorithm compares new topics to the training set and compares the traffic over time between them. If this new topic develops similar patterns to one of the successful trends in the training set, it is assigned a weighted number on the possibility of its future success. The more a possible trend compares to previous successful trend history, the greater the chance it will be popular. Don’t you just love the wonders of math?

At this point, the connection between powerful algorithms like this and a social media agency or company becomes apparent. Who wouldn’t want to be first to the party and dominate the consumer market dance floor? Well, we will just have to wait and find out where it leads because I am sure there are many eyes focused in on this powerful tool.

What are other ways this algorithm could be beneficial to social media?

Eric Gruenfelder

Eric Gruenfelder

Intern, fishbat

See Full Profile >


Hi, I enjoyed the article. Just one thing - "But what if you could predict what will be the next popular trend before it happens?" - in this sentence, "popular trend" strikes me as a pleonasm.

Yes, definitely. But, when I mean popular trend, I am thinking more of the top trends that show up listed. Thanks for the reply!