Document parsing is top of my mind lately because in some of the areas we work on the bottleneck is starting to become being able to query documents the same way one queries an api.
I keep thinking the most obvious analogue is we need some way to represent documents the same way we can represent structured data in parquet. Parquet allows easy range bases queries and there is so much tooling built around Arrow.
But for documents I keep hitting a wall to figure out what the right abstractions are. Parquet allows filterable metadata. But what such metadata is there for documents. Then there is the arbitrrariness of chunking, vectorization.
If we could just do this in a 2 step process where every document to process can be represented in a parquet like data format then I think we will atleast have the semblance of a solution.
It’s much more limited in scope but fully open source and highly customisable. In fact it’s made for people to build their own pipelines on top of, providing the scaffolding needed to do so in a reliable way.
During development I’ve found it to be hard to truly generalise agent/llm-based data extraction, especially around the unlimited number of input types without task specific instructions (many files of the same kind, single large files, mixed kinds, bad quality files, docx/pdf/png/… the list goes on). Users sadly wanna upload all of these, and developers want a „one size fits all“ solution.
I am interested in how your solution deals with this. I came up with a strategy based approach so every task can be customised if needed, but I’d be delighted to see a technical writeup of how you deal with this endless variety of input + extraction task combos! :)
Fully agree, that's why we quite like the Databricks OfficeQA benchmark.. it made us experts on historical US treasuries haha
Some screenshots in here: https://www.parsewise.ai/officeqa-sota
I'm surprised at the low rate every model manages considering the (apparent) ease of the benchmarked document. Can your pipeline produce ground truth as a byproduct ? How do you think open-weight ocr models compare to the one showcased ? I've had good results with glm-ocr on complex documents (complex by their handwriting, pretty easy layouts).
What I like about your solution is the traceability of the information. A scruffy pipeline I used was gemini-flash 3.0 to pdf to notebook-lm (really amateurish work i know), but it yielded tremendeous time gains to extract info from documents (that could be borderline impossible to read for me). However, to trace back the info was obviously very tedious. But from my experience, notebooklm can now manage ocr/htr without a third party. I wonder how competitive your solution might be compared to messy workflows that work -- albeit with efforts -- but let's the researcher be "in contact" with the material.
What I really want is obviously an easy to setup local rag system, with the (very) light model that goes with it ... sweet dream.
How portable are your agent definitions? If I build one for insurance documents, how much work is needed to adapt it to a completely different domain like legal contracts or healthcare?
In practice we find that each domain (and even each organisation) ends up having highly customized definitions.
At first, fairly generic templated definitions sort of work, but what we've seen is that over time data comes up that is out of distribution, and there was no explicit instruction on how to deal with it. In such cases we tend to flag this and offer suggestions to the users on how they can improve the specificity of agents.
Another structure we have seen play out is having a manager review ratings and feedback comments from their team and updating the definitions accordingly over time (where we offer them the capability to see results of before and after side -by-side for all existing data as well, so they are more confident in the change before committing).
The amount of work is dependent on how good the initial definitions are and how complex the use case is (and how much it evolves - new data sources etc). A bit of an unsatisfying answer but it can be anywhere between a few hours one off or a couple of minutes per day on an ongoing basis.
I worked recently on an internal tool to achieve this kind of things, mostly plugging mistral OCR to gemini to extract structured data from documents. We then perform automated diffs too.
There seems to be an insane amount of competition in the "Intelligent Document Processing" market, like for instance parseur, whose founder is often on HN himself.
What do you think sets you apart from competition like :
1) Mistral document AI : depending on the model, it looks way cheaper than yours, OCR model pricing ranges from 0.001 to 0.004 EUR / page and they have structured output wired in the OCR API if needed (things then get fed to one of their LLMs) + EU-based and GDPR ready
2) parseur / rossum / docsumo / nanonets (which is YC 2017) ?
I understand what they are trying to do, but to me it feels like the moment when MongoDB entered the database space, with semi-structured, "flexible" storage format. It has its uses, for prototyping mostly.
But in high-volume, production workloads, giving a structure to the data you extract (what Parseur does through defining the Fields in your Mailbox, basically giving your output data a schema) adds a ton of value, and the larger the dataset, the truer it is.
Usually, you start by defining where you want your data to go, and which structure it should have, before working backwards from here and starting to extract the data. This is the key to automating your document workflow.
Hey, good point about structure for integrated workflows:)
Fully agree, for enterprises we need to guarantee types, flag discrepancies and provide underlying sources so they can integrate it downstream (whether that's Databricks, n8n etc.)
1. We are working with the assumption that OCR is (or soon will be) solved at super low prices.
So if we have the extracted data, what can we do with it?
Where we see Parsewise making a difference is for use cases that span across documents.
I.e. if you are extracting the same 5 fields from every invoice, there are lots of solutions as you listed (+ reducto etc). However, once you have a set of documents (e.g. an entire mortgage application package) and you are trying to get a structured response out, then your option is either an LLM API (if things fit into context and you are okay with limited citations), or building a pipeline with LLMs. I posted it in another comment but an example of trawling through 90k pages is here: https://www.parsewise.ai/officeqa-sota
2. While we rely on LLMs, the outcomes will be non-deterministic, so the bottleneck is and will remain the human verification (that is for somewhat complex use cases). The architecture that we have built is optimizing for the human reviewer to provide as granular values and citations as possible. This is either through our platform, or API clients.
I say this with a lot of love: The vibecoded applications in your demo reek of AI slop design.
This isn't a critique of your product. It's just that the a beige-orange theme, the pill components, and the left-border highlight give me that visceral reaction as reading a paragraph littered with em dashes and "not X but Y." It makes me take you less seriously.
Haha no appreciate it! That's on me for not calling it out explicitly (was trying to make the video as short as possible), but the demo UIs were literally vibe coded to show the ease of integration https://youtu.be/F1cSuZal03s?si=1H4zTcO-8cosLbVr&t=70
Similar to my other comment, we assume that llamaparse and others can provide the individual page OCR. But once you have that the way that you can integrate it into your workflows often requires additional complexity around combining results from different sources. Here is a deeper dive I wrote on the complexities of building extraction pipelines: https://www.parsewise.ai/doc-processing-pipelines
I learnt a lot at Palantir, though always worked in commercial so no ties to security state (for the better or worse).
(Also side-note, we are working towards enabling frontier performance with smaller open models that allows our customers to protect their data. https://www.parsewise.ai/officeqa-sota )
And I do get genuine joy from helping our users, so love it is:)
Noted — and I did wish the founder success. I have no personal ill will towards them. But what I'd ask HN to consider is this: our world, and the technology we introduce into it, isn't apolitical or free of normative stakes and real, harmful implications for people. Treating where you've worked and what technology you've stewarded into being as an ethically neutral fact isn't neutral at all. What concerns me is that there's an increasing firewall against calling out things that ACTUALLLY harm people — while an objection gets reframed as a personal attack on someone willingly able to propagate problematic things. But this seems to be where the corporate tech world is moving as it cozies up to the authoritarians.
Planning to serve good things for sure, and appreciate your note.
Ofc I didn't agree with everything Palantir was doing (also to the extent that we even knew about them at the time). I was working on vaccine distribution and cancer research as well, so definitely felt like helping.
Document parsing is top of my mind lately because in some of the areas we work on the bottleneck is starting to become being able to query documents the same way one queries an api.
I keep thinking the most obvious analogue is we need some way to represent documents the same way we can represent structured data in parquet. Parquet allows easy range bases queries and there is so much tooling built around Arrow.
But for documents I keep hitting a wall to figure out what the right abstractions are. Parquet allows filterable metadata. But what such metadata is there for documents. Then there is the arbitrrariness of chunking, vectorization.
If we could just do this in a 2 step process where every document to process can be represented in a parquet like data format then I think we will atleast have the semblance of a solution.
I built a similar tool some time ago called Struktur (https://struktur.sh).
It’s much more limited in scope but fully open source and highly customisable. In fact it’s made for people to build their own pipelines on top of, providing the scaffolding needed to do so in a reliable way.
During development I’ve found it to be hard to truly generalise agent/llm-based data extraction, especially around the unlimited number of input types without task specific instructions (many files of the same kind, single large files, mixed kinds, bad quality files, docx/pdf/png/… the list goes on). Users sadly wanna upload all of these, and developers want a „one size fits all“ solution.
I am interested in how your solution deals with this. I came up with a strategy based approach so every task can be customised if needed, but I’d be delighted to see a technical writeup of how you deal with this endless variety of input + extraction task combos! :)
This looks great for digital humanities, specifically archival work. Would love to try it.
Fully agree, that's why we quite like the Databricks OfficeQA benchmark.. it made us experts on historical US treasuries haha Some screenshots in here: https://www.parsewise.ai/officeqa-sota
I'm surprised at the low rate every model manages considering the (apparent) ease of the benchmarked document. Can your pipeline produce ground truth as a byproduct ? How do you think open-weight ocr models compare to the one showcased ? I've had good results with glm-ocr on complex documents (complex by their handwriting, pretty easy layouts).
What I like about your solution is the traceability of the information. A scruffy pipeline I used was gemini-flash 3.0 to pdf to notebook-lm (really amateurish work i know), but it yielded tremendeous time gains to extract info from documents (that could be borderline impossible to read for me). However, to trace back the info was obviously very tedious. But from my experience, notebooklm can now manage ocr/htr without a third party. I wonder how competitive your solution might be compared to messy workflows that work -- albeit with efforts -- but let's the researcher be "in contact" with the material.
What I really want is obviously an easy to setup local rag system, with the (very) light model that goes with it ... sweet dream.
How portable are your agent definitions? If I build one for insurance documents, how much work is needed to adapt it to a completely different domain like legal contracts or healthcare?
In practice we find that each domain (and even each organisation) ends up having highly customized definitions.
At first, fairly generic templated definitions sort of work, but what we've seen is that over time data comes up that is out of distribution, and there was no explicit instruction on how to deal with it. In such cases we tend to flag this and offer suggestions to the users on how they can improve the specificity of agents.
Another structure we have seen play out is having a manager review ratings and feedback comments from their team and updating the definitions accordingly over time (where we offer them the capability to see results of before and after side -by-side for all existing data as well, so they are more confident in the change before committing).
The amount of work is dependent on how good the initial definitions are and how complex the use case is (and how much it evolves - new data sources etc). A bit of an unsatisfying answer but it can be anywhere between a few hours one off or a couple of minutes per day on an ongoing basis.
Hey ! Is this kind of like structured output over a large scale document corpora ?
Hey, that's exactly it!
I worked recently on an internal tool to achieve this kind of things, mostly plugging mistral OCR to gemini to extract structured data from documents. We then perform automated diffs too.
There seems to be an insane amount of competition in the "Intelligent Document Processing" market, like for instance parseur, whose founder is often on HN himself.
What do you think sets you apart from competition like : 1) Mistral document AI : depending on the model, it looks way cheaper than yours, OCR model pricing ranges from 0.001 to 0.004 EUR / page and they have structured output wired in the OCR API if needed (things then get fed to one of their LLMs) + EU-based and GDPR ready 2) parseur / rossum / docsumo / nanonets (which is YC 2017) ?
Hi, Parseur founder here :D
I understand what they are trying to do, but to me it feels like the moment when MongoDB entered the database space, with semi-structured, "flexible" storage format. It has its uses, for prototyping mostly.
But in high-volume, production workloads, giving a structure to the data you extract (what Parseur does through defining the Fields in your Mailbox, basically giving your output data a schema) adds a ton of value, and the larger the dataset, the truer it is.
Usually, you start by defining where you want your data to go, and which structure it should have, before working backwards from here and starting to extract the data. This is the key to automating your document workflow.
Hey, good point about structure for integrated workflows:)
Fully agree, for enterprises we need to guarantee types, flag discrepancies and provide underlying sources so they can integrate it downstream (whether that's Databricks, n8n etc.)
Here is our documentation for working with a fixed JSON schema: https://docs.parsewise.ai/api#schema-driven-extract-convenie...
Great question!
1. We are working with the assumption that OCR is (or soon will be) solved at super low prices.
So if we have the extracted data, what can we do with it? Where we see Parsewise making a difference is for use cases that span across documents. I.e. if you are extracting the same 5 fields from every invoice, there are lots of solutions as you listed (+ reducto etc). However, once you have a set of documents (e.g. an entire mortgage application package) and you are trying to get a structured response out, then your option is either an LLM API (if things fit into context and you are okay with limited citations), or building a pipeline with LLMs. I posted it in another comment but an example of trawling through 90k pages is here: https://www.parsewise.ai/officeqa-sota
2. While we rely on LLMs, the outcomes will be non-deterministic, so the bottleneck is and will remain the human verification (that is for somewhat complex use cases). The architecture that we have built is optimizing for the human reviewer to provide as granular values and citations as possible. This is either through our platform, or API clients.
I say this with a lot of love: The vibecoded applications in your demo reek of AI slop design.
This isn't a critique of your product. It's just that the a beige-orange theme, the pill components, and the left-border highlight give me that visceral reaction as reading a paragraph littered with em dashes and "not X but Y." It makes me take you less seriously.
Cool demo otherwise.
Haha no appreciate it! That's on me for not calling it out explicitly (was trying to make the video as short as possible), but the demo UIs were literally vibe coded to show the ease of integration https://youtu.be/F1cSuZal03s?si=1H4zTcO-8cosLbVr&t=70
llamaparse also do it, what is different here?
Similar to my other comment, we assume that llamaparse and others can provide the individual page OCR. But once you have that the way that you can integrate it into your workflows often requires additional complexity around combining results from different sources. Here is a deeper dive I wrote on the complexities of building extraction pipelines: https://www.parsewise.ai/doc-processing-pipelines
Mostly cross-doc reasoning at scale (e.g., 90k-page corpora) as opposed to doc-to-markdown conversions.
Ah probably should add a link to our website: https://www.parsewise.ai/api
Added above :)
"retaining lineage"
"That is a great catch!"
[flagged]
A launch post is not a place to attack other users personally. Neither is any other HN thread for that matter, so please don't do it here.
https://news.ycombinator.com/newsguidelines.html
I learnt a lot at Palantir, though always worked in commercial so no ties to security state (for the better or worse). (Also side-note, we are working towards enabling frontier performance with smaller open models that allows our customers to protect their data. https://www.parsewise.ai/officeqa-sota )
And I do get genuine joy from helping our users, so love it is:)
[flagged]
A launch post is not a place to attack other users personally. Neither is any other HN thread for that matter, so please don't do it here.
https://news.ycombinator.com/newsguidelines.html
Noted — and I did wish the founder success. I have no personal ill will towards them. But what I'd ask HN to consider is this: our world, and the technology we introduce into it, isn't apolitical or free of normative stakes and real, harmful implications for people. Treating where you've worked and what technology you've stewarded into being as an ethically neutral fact isn't neutral at all. What concerns me is that there's an increasing firewall against calling out things that ACTUALLLY harm people — while an objection gets reframed as a personal attack on someone willingly able to propagate problematic things. But this seems to be where the corporate tech world is moving as it cozies up to the authoritarians.
Planning to serve good things for sure, and appreciate your note. Ofc I didn't agree with everything Palantir was doing (also to the extent that we even knew about them at the time). I was working on vaccine distribution and cancer research as well, so definitely felt like helping.