Dear community,

I am running a clustering experiment for my project using various metrics to cluster fine art auction catalogue entries. Using domain knowledge, I have extracted certain features from the text to cluster. Alongside these specialist features, I would like to input some form of Tfidf metric that measures the semantics and vocabulary used relative to the rest of the vocabulary used in other entries as a corpus of words.

The step I have taken so far:

Text preprocessing and cleaning Tokenization TFIDF vector processing with scikit From TFIDF scores of individual words calculated a mean TFIDF score across a document (auction entry)

My feeling is that this might have some weaknesses (skew impact from longer documents in the corpus / or impact of high TDIDF scores from shorter documents) but could capture some of the rarer choices of vocabulary in a sentence relative to other entries.

The idea behind the average TFIDF score is to get a document level capture of how unusual the vocabulary - in a specific document - is is relative to the entire corpus of auction entries. I would be interested to hear if anyone has had experience with such a methodology or even any feedback on it.

Does anyone in the group have any experience or thoughts on this as a sentence/document level information capture?

Other word and sentence embeddings are clear alternatives but I favour TFIDF as it has the advantage of being specific to the vocabulary used in my domain specialist data set.

Best wishes, Mathew