Best LLMs for document classification

[oreju]

Hello everyone,

I’m really excited about this repository! I was wondering if anyone could share which LLM solution—or combination of LLMs—they’ve tried with ExtractThinker that produced the best results.

Thanks!

[enoch3712]

Hello @oreju

You can look at the tests:

https://github.com/enoch3712/ExtractThinker/blob/main/tests/test_classify.py

You have several examples, and personally i use small models followed by bigger models, like gpt4o-mini followed by gpt4o if is not over the treshold.

This example is clear in the test_mom_classification_layers test

[oreju]

Hello @enoch3712 ,

Appreciate your response, and apologies for the delayed reply. I’ve analyzed your code, and everything is clear to me.

I wanted to ask—what would you consider the go-to models for open-source small LLMs? I'm thinking of self-hosting due to client privacy concerns.

Thanks!

[enoch3712]

Hello @oreju

Yes, 100%. That's literally what im doing now. To give you an idea, for now using deepseek 1.5, 7b

But should not be in useful time for you. Advice you to use DeepSeek smaller models like im doing now. I did this for a major bank in the EU.

[oreju]

@enoch3712 ,

so as I understand you are masking before calling ChatGPTs models for classification?

I was thinking on using deepseek-r1 7b,14b hosted in my cloud and running classification tasks without masking for the moment.

[enoch3712]

@oreju

This part of the PII part. Its been a PR for a while, i have been checking what to add. Its pretty much done at this point.

The problem is, imagine you are a bank and you want to use SOTA models, you need to PII everything before going.

Its not for classification per say, before eveything else in this pipeline.

[oreju]

Ok, it's clear now. Thank you for your response and for your work! Really appreciate it!