Aggregator App

[vantubbe]

shortsnip

Building an app to quickly show what's happening online. It takes twitter and news data and performs summarization, topic modeling, and similarity. It's very much a work in progress but hopefully the idea makes sense.

This snip was particularly interesting. If you scroll down to the earliest source you'll see someone predicted Andrew Tate's arrest two months before it happened. Though now the account is suspended.

[MaartenGr]

Interesting! Are "related" and "sources" a result of the topic modeling?

[vantubbe]

Related are other "snips" within the topic, sources are original docs within the topic whose summarizations are above a syntactic similarity threshold. A snip is meant to be a quick summary of all of it's sources.

The snips are created from a 3 step process. Starting with a batch of 10k tweets, news headlines, etc...

1. Summarize the whole batch - Even though tweets are already pretty brief, they usually have a lot of nonsense. The summarization tends to make them more clear and better candidates for topic modelling. News headlines tend not to change much from summarization which is expected.
2. Topic Model - partial_fit the batch. This is the core part of the process and where I've had the most challenges.
3. Similarity - Run syntactic similarity on all the summaries within each topic. If it finds summaries that are similar enough, create a snip or add to an existing snip.

I really like it when news headlines and tweets end up in the same snip so you can see how people are talking about events. Like this one. Adding reddit post titles would be an interesting addition. Seeing the timeline of sources is really cool.

Biggest problem right now is producing cohesive topics. Clicking on related you'll likely notice snips that don't "relate" much to each other. That's where my recent question about outliers came from.

Thank you so much for taking the time to look at it!

[MaartenGr]

Thanks for sharing the use case and the extensive description! Interesting process. If you ever have any questions regarding the topic modeling, feel free to reach out!