Unstructured unnecessarily "repairs" then falls back to OCR on extremely large documents

[dhdaines]

TL;DR this is because Unstructured.io applies a "fix" to pdfminer.six which actually introduces another bug.

The fix (which I have tested here) is, until the bug is fixed in pdfminer.six, to not apply the patch, and to simply avoid the following versions of pdfminer.six which contain the bug: 20231228 and 20240706

The source of the problem is this change to pdfminer.six (which I think was submitted by unstructured?): pdfminer/pdfminer.six#885

This broke the parsing of long content streams, because due to the complex and not particularly useful buffering done by pdfminer.six's "parser" (actually a lexer), it is frequently the case that keywords are split across a buffer boundary. In my particular testcase (cannot share, sorry) this results in the "Detected invalid dictionary construct for PDFminer" error reported by Unstructured.io, because a dictionary ends up with an odd number of items as false is getting split into fals and e.

"Repairing" the PDF will not help this problem because there isn't anything wrong with the PDF, it just has some long content streams. So what happens is that (on my document of 1500+pages) unstructured falls back to OCR, which as you can imagine is ... suboptimal.

[dhdaines]

If you must patch pdfminer.six, then please apply this fix, which fixes both the original problem and the problem that was introduced by patching it: pdfminer/pdfminer.six#1030

I can supply a PR. (note that it is not as simple as patching a single method, unfortunately, but should be doable)