In our approach, we're just zero shot asking for markdown from the image. Vs this approach of passing in the Tesseract result + image context and asking for correction. I'm curious if there is a meaningful accuracy difference.
My first thought it that the tesseract result may decrease accuracy, especially with tables or multi column pdfs. The tesseract model has a tendency to take everything from a table and throw it into one text blob. So while it's added context to the model, it's often not helpful.
Intuitively I would be surprised if the tesseract-way was better to be honest.
I'm not the most experienced with OCR but I have tried Tesseract in the past and had difficulties. Like others mentioned, it would mix up things like 4 vs A etc. and had massive issues with decimal numbers.
I have had zero such issues with zerox.
Someone on HN also recently mentioned that tesseract is OCR of the past and basically outlived by a long shot already, I feel like that might be true.
Btw I feel extremely lucky to have found Zerox because we needed a solution like that at work and so I introduced it and spun up a prototype to solve our problem and it works so well (combined with the brand new "strict json" feature by openai), so really great job making it!
Given both libraries are primarily using GPT-4o-mini, I'd expect pretty similar performance. The author here seems to have some better prompts than we do: https://github.com/Dicklesworthstone/llm_aided_ocr/blob/main...
In our approach, we're just zero shot asking for markdown from the image. Vs this approach of passing in the Tesseract result + image context and asking for correction. I'm curious if there is a meaningful accuracy difference.
My first thought it that the tesseract result may decrease accuracy, especially with tables or multi column pdfs. The tesseract model has a tendency to take everything from a table and throw it into one text blob. So while it's added context to the model, it's often not helpful.