Parsing enhancements

[ajc9988]

Although alright, I moved to this from docling after going to Rust. But, critical features are needed that are not offered that can enhance the performance and formatting of the parsing for later RAG ingestion:

1. DIQA pre- and post-processing with a heuristic analysis for poor quality documents. By implementing automatic processing for denoising and other enhancements when quality is determined to be low (like faxes, bad scans, etc.), this can prepare the material for later processing enhancing the parsing.

2. hybrid-OCR - This is difficult to pull off well, but is something I've worked on for my own use modifying Docling and now kreuzberg in local dev (vendored for a RAG). What do I mean by hybrid? First is coming up with proper layout detection. Then is using the other packages, like pdfium or pdf-oxide (need the bbox; medical records, police reports, standard gov't forms, academic papers, legal decisions, etc.). By doing so, you can get 90+% of what you need, but when certain layout features are detected, then it calls on the OCR/VLM to process those elements, like diagrams, charts, pictures, and similar that the VLM or OCR can properly extract or describe but that cannot be done through other packages. By doing this, you keep local processing time low, you do not lose what pdfium or pdf-oxide get that the OCR can sometimes miss, but you also get all the elements that those could not extract due to their structure or nature. This is the balance of speed and performance. Then you do full OCR or VLM if the heuristic determines it is a low quality PDF, which will take longer, but allows for the heavy processing route when needed. So you have pdfium or pdf-oxide if simple extraction, hybrid-OCR for structured extraction when detected on good quality inputs, then OCR/VLM full extraction for bad quality source material.

3. integration of GROBID as a plugin. By setting it up for a local docker GROBID instance to be usable (or a collaboration with them), the footnote/endnote/bibliography extraction and linking becomes much easier and more predictable.

4. hierarchical preservation - most documents, from academic work, caselaw, contracts, etc., rely on hierarchical organization. Being able to accurately preserve and maintain the hierarchy would allow for proper summaries for RAG systems, and interesting storage options in the databases.

And maybe I just screwed up my implementation then had to work around these things to link footnotes to chunks, properly maintain hierarchies, or run it efficiently on an AMD Ryzen AI Max+ 395 while getting the performance I wanted. But some thoughts.

Edit: To be clear, I could not use your chunking because of a novel mix needed for chunking with the hierarchy tracked. So I was not using it as intended, but was using the other aspects of parsing to build off of. If you want to discuss the novel chunking requirements, we can figure out a time to discuss.

[Goldziher]

Hi @ajc9988 -

1. This is complex to do correctly, but possible I think. I need to see whats available in the Rust ecosystem that can serve as a basis. I would say this would be a great issue.
2. Hybrid OCR is certainly a next step. Also an issue.
3. GROBID is very heavy. I dont think this is a viable path. I would say the issue you are aiming for is better citations extraction / footnotes + bibliography. I would like to understand better and understand whats not working currently.
4. I spent quite a bit of effort over the past 2 releases to work on structure extraction. It should give you pretty good outputs. I am interested in knowing if it doesnt, or what issues you encountered. Also what you find to have non-preserved hierarchy - either in outputs, or in the structure extraction. These are issues that should be addressed immediately as bugs. Any gaps or delta -- i would also be keen on knowing.

I am always glad to speak with users. If you want to tell me of your use case, we can certainly have a call. Ping me on discord?

[ajc9988]

I have sent an email from a university email address. I do not use discord.

[Goldziher]

I have sent an email from a university email address. I do not use discord.

Thanks. I read it.