How receipt scanning actually works

Quick answer

Receipt scanning turns a photo into structured data in four stages: the image is cleaned up and de-skewed, OCR converts pixels into text with positions, a model interprets that text to identify which numbers are the total, the tax, and the line items, and finally the merchant is matched to a spending category. Modern apps do all four in a couple of seconds, which is why a photo is now faster than typing.

Stage 1 — Image preparation

The photo you take is rarely ideal: shot at an angle, lit by one overhead bulb, with the receipt curling at both ends. Before any text is read, the image is corrected.

• Detection and cropping — finding the receipt's edges and discarding the table around it
• Perspective correction — flattening an angled shot into a rectangle
• Contrast and noise handling — recovering faint thermal print and removing shadows
• Rotation — straightening so text lines run horizontally

Most scanning failures that look like "the OCR is bad" actually start here. A photo taken in poor light with the receipt half-folded gives every later stage worse input.

Stage 2 — Optical character recognition

OCR converts the cleaned image into characters. The important part is that good OCR returns more than a wall of text — it returns each piece of text with its position on the page.

That spatial information is what makes the next stage possible. "12.40" alone is meaningless. "12.40, right-aligned, on the same line as the word Coffee, above a line labelled TOTAL" is a line item.

Receipts are genuinely hard for OCR: narrow thermal paper, condensed fonts, low-contrast print, and heavy abbreviation. This is why general-purpose text recognition often underperforms on receipts compared with models trained on them specifically.

Stage 3 — Understanding the layout

Now the app has to answer: which of these numbers is the total?

A receipt typically contains a dozen numbers — item prices, quantities, subtotal, tax, discounts, tip, total, change given, card last four digits, a transaction ID. Picking the right one is an interpretation problem, not a reading problem.

The model uses patterns learned across many receipt formats:

• Merchant — usually the largest text at the top, often matched against known merchants
• Date — recognized across formats, including DD/MM versus MM/DD ambiguity
• Total — near a keyword like TOTAL, AMOUNT DUE, or BALANCE, typically the largest value and positioned below the subtotal and tax
• Tax — labelled and reconciling against subtotal and total
• Line items — description on the left, price right-aligned, repeated in a block

The strongest signal is arithmetic: if subtotal plus tax equals a candidate total, that candidate is almost certainly the total. Consistency checks like this are how good extraction catches its own mistakes.

Stage 4 — Categorization

With the merchant known, the expense is assigned a category — groceries, dining, travel, fuel, software. This comes from merchant recognition, keywords in the line items, and patterns in how similar purchases are usually classified. Corrections you make teach the system your preferences, which is why an app gets noticeably better after a few weeks.

Why extraction still occasionally gets it wrong

• Physically damaged receipts — a crease through the total line removes information no model can recover
• Faded thermal paper — if a human cannot read it, neither can OCR
• Unusual layouts — some point-of-sale systems print in formats that match nothing common
• Handwritten amounts — much harder than printed text
• Multiple receipts in one photo — photograph them one at a time
• Ambiguous dates — 03/04 is genuinely undecidable without regional context

Practical fixes: photograph on a flat, contrasting surface in even light, capture all four edges, take the photo while the receipt is fresh, and one receipt per shot.

What it means for you

The reason a receipt scanner saves time is not that it types faster than you. It is that stages three and four remove the decisions — you never have to look at a receipt and think about which category it belongs to.

Receipt Scannr runs this whole pipeline on a single photo and returns merchant, total, date, and line items, categorized and ready to appear in your spending charts. Free to download on iOS and Android.