Dear David,
you might have solved it meanwhile, but if not:
- if the task is to deduplicate, have a look at onion https://corpus.tools/wiki/Onion - if you need to count only, you can make a corpus in Sketch Engine to calculate, we use a suffix array to calculate ngrams up to the length of 20 by default, following:
Yamamoto and Church: Using Suffix Arrays to Compute Term Frequency and Document Frequency for All Substrings in a Corpus Computational Linguistics, Volume 27 Issue 1, March 2001, pp 1-30 http://www.aclweb.org/anthology/J01-1001
The interface displays n-grams up to the length 6 (though computed is 20), let me know if you need to display longer ones too.
Best regards, Milos Jakubicek
CEO, Lexical Computing Brno, CZ | Brighton, UK http://www.lexicalcomputing.com http://www.sketchengine.eu
On Tue, 27 Jun 2023 at 19:20, Christian Wartena via Corpora < corpora@list.elra.info> wrote:
Hello, we have used the Apriori-Algorithm to detect long identical text passages ( https://link.springer.com/chapter/10.1007/978-3-030-86159-9_34). That works quite well. I am not sure whether Frieda Jsi published the code, but it is quite easy to implement or I can send you the code.
Best Christian _______________________________________________ Corpora mailing list -- corpora@list.elra.info https://list.elra.info/mailman3/postorius/lists/corpora.list.elra.info/ To unsubscribe send an email to corpora-leave@list.elra.info