Hi David,
You can look into the ROUGE package which does n-gram based matches. https://huggingface.co/spaces/evaluate-metric/rouge
You will find actual n-gram computation code here https://github.com/google-research/google-research/blob/master/rouge/rouge_s...
Also, Fuzzy string matching is another efficient approach to do substring matching https://github.com/seatgeek/thefuzz
Thanks, Mousumi
On Fri, Jun 23, 2023 at 9:39 AM David Beauchamp via Corpora < corpora@list.elra.info> wrote:
Hi all, I'm doing analysis on a corpus on tweets from institutions. Regarding analysis of n-grams, it is quite unusual in that there are many repeated exact tweets, or very similar tweets, leading to long super strings of often 9 or 10 or more words together. Naturally this makes accurate counting and classifying difficult due to the overlapping substrings. Does anyone know of any approaches or software which can count and classify n-grams in such circumstances? I am aware of approaches outlined by Buerki (2017) and O'Donnell (2011), but these do not seem practical due to the excessive length of the n-grams in the corpus. Does anyone know of any accessible methods or packages?
Any input much appreciated. _______________________________________________ Corpora mailing list -- corpora@list.elra.info https://list.elra.info/mailman3/postorius/lists/corpora.list.elra.info/ To unsubscribe send an email to corpora-leave@list.elra.info