Comparing Rule-Based and LLM-Based B2B Document Extraction: Which Approach Performs Better?

This article compares two methods for extracting structured data from B2B order documents: a traditional rule-based system using pytesseract OCR and a modern LLM approach using Ollama with LLaMA 3. The comparison is grounded in a realistic scenario of processing B2B order forms. Below, we answer key questions about the implementation, performance, and trade-offs of each method.

1. What was the goal of building the same B2B document extractor twice?

The primary goal was to evaluate the effectiveness and practicality of two contrasting approaches for extracting structured information (e.g., order numbers, line items, dates, totals) from B2B documents. Documents varied in layout, font, and quality—typical in real-world supply chain scenarios. By building the extractor twice with the same test set, the creator aimed to compare accuracy, development time, maintainability, and robustness. The rule-based method relied on hardcoded heuristics and OCR positions, while the LLM method used natural language understanding to parse document content directly.

2. How does the rule-based approach using pytesseract work?

The rule-based system starts with pytesseract, a Python wrapper for Google's Tesseract OCR engine, to extract raw text and bounding boxes from scanned PDFs. Custom scripts then parse this output by identifying predefined fields (e.g., "Order ID", "Total Amount") using keyword matching, regular expressions, and coordinate-based rules. For example, the system might look for text containing "Order No." and then extract the value immediately to the right or below. This approach works well when document layouts are consistent but struggles with variations in positioning, font or minor OCR errors. Tuning required manual inspection of many sample documents to define robust rules.

3. What are the main steps in the LLM-based approach using Ollama and LLaMA 3?

The LLM pipeline uses Ollama to run the LLaMA 3 model locally, avoiding cloud costs and latency. First, the document is preprocessed using OCR (again with pytesseract) to convert the PDF to plain text—though in some cases the LLM can accept image inputs directly. The extracted text is then fed into the LLM with a carefully crafted prompt asking it to return a JSON object containing the desired fields (e.g., order number, due date, line items). The model uses its training knowledge to locate and interpret data even when field names are missing or the layout is non-standard. This approach can handle typos and ambiguous formatting much more flexibly than rules.

4. What were the key differences in accuracy between the two methods?

In the documented test, the LLM-based extractor achieved significantly higher accuracy across a diverse set of B2B order documents—approximately 94% field-level accuracy compared to 78% for the rule-based system. The rule-based method frequently failed when documents had slight shifts in alignment, missing field labels, or unexpected abbreviations. The LLM, by contrast, used contextual clues to infer values: for instance, if the label "Total" was missing, it could deduce the total from context like "Amount Due". However, the LLM occasionally hallucinated fields or misinterpreted numbers (e.g., confusing order number with invoice number), especially on very poor OCR output.

5. How did the two approaches handle variations in document formatting?

Document formatting variation is a major challenge in B2B contexts—different vendors use distinct layouts, fonts, and terminology. The rule-based approach required a new set of heuristics for each major layout, making it brittle. Even minor changes (e.g., a different font causing OCR to misread characters) broke the extraction. The LLM-based approach showed strong resilience: it could interpret data from tables, blocks, or single-line fields without explicit positional rules. It also handled variations in field names (e.g., "PO#" vs. "Purchase Order Number") by leveraging the model's language understanding. However, the LLM performance degraded when the OCR output had many garbled words, requiring a preprocessing step to clean the text.

6. What are the trade-offs in terms of development effort and maintenance?

Developing the rule-based system was initially time-consuming but straightforward: several days to write and test regular expressions and coordinate-based logic. Maintenance, however, was high—every new client layout required rule updates and regression testing. The LLM approach took less than a day to set up the prompt and integrate Ollama, but required careful prompt engineering and occasional fine-tuning of the prompt to reduce hallucinations. Ongoing maintenance is minimal because the LLM can adapt to new layouts without code changes. The trade-off is computational cost: running LLaMA 3 locally demands substantial CPU/GPU resources, while the rule-based system runs on any machine with minimal overhead.

7. Which approach is more suitable for production B2B scenarios and why?

For production B2B document extraction, the LLM-based approach is generally more suitable when the document variety is high and accuracy requirements are moderate. Its flexibility reduces the need for constant rule updates and can handle edge cases that would break a fixed system. However, for scenarios with strict latency or resource constraints (e.g., embedded devices), rule-based extraction may be preferable due to its speed and low computational cost. A hybrid approach is also viable: use rules for well-known templates and fall back to an LLM for outliers. In the tested scenario, the author leaned toward the LLM method for its superior adaptability, recommending prompt validation and output formatting layers to mitigate hallucinations.