TailSpend Docs

Receipt scanning & AI expense prefill

This document specifies the scan-a-receipt feature: a user photographs or uploads a receipt, TailSpend runs OCR and an LLM over it, and the result prefills the normal create-expense form. The feature is metered with credits that users earn by watching rewarded video ads.

This describes the implementation, including the asynchronous scan-job flow, content deduplication, credit handling, and temporary retention. The issue #694 OCR/model changes below are implemented in source. Deployment was authorized on 22 October 2026 through the normal dev → beta → production pipeline; the GitHub Actions deployment run is the source of truth for rollout status.

For the historical baseline and alternatives evaluated in issue #694, see the receipt OCR evaluation. It records research and a bounded live comparison, including extraction failures in both alternatives and the previous Gemma baseline. The replacement uses PP-OCRv6 small for images and rendered PDF pages, with Ministral 3 8B for structuring. Textract is removed from the serving path. Four development receipts pass exact totals and item-cost lists with the new configuration; representative accuracy and deployed Lambda performance remain unmeasured.

Goals

Non-goals (v1)

Decisions (locked)

These were settled before design; recorded here so the plan is unambiguous.

Decision Choice Consequence
LLM hosting Ministral 3 8B Instruct on Bedrock (mistral.ministral-3-8b-instruct) Serverless InvokeModel with native JSON-schema output; OCR text only. USD $0.15 per million input and output tokens at standard us-east-1 rates.
OCR engine PP-OCRv6 small for JPEG/PNG and rendered PDF pages A dedicated 2 GiB x86 Python container Lambda bundles the evaluated ONNX weights and PDFium. PDFs render sequentially with explicit page boundaries; no Textract calls.
Earning surface Mobile only (AdMob rewarded video) Balance is shared; web spends, mobile earns. Web shows "earn on mobile" messaging when short.
Ad reward trust AdMob Server-Side Verification (SSV) Credits are granted by a Google→backend signed callback, never by the client. Replay-protected by transaction_id.
Region us-east-1 Receipt OCR Lambda and Bedrock remain in the deployment region.

Where this sits in the system

The feature is additive and respects every existing contract discipline:

Receipt scanning: upload, asynchronous processing, recovery and credits

User experience

Entry point — dashboard FAB

A floating + button sits bottom-right on both dashboards (web FabMenu, mobile fab-menu). Tapping it opens a "Add an expense" menu: cards for the most recently-used groups (tap one to jump into its New Expense screen, where the scan buttons live), plus New group and — when there are more groups than shown — All groups. The menu no longer routes to a standalone scan screen; scanning happens on New Expense.

Scan is part of the New Expense screen

There is no separate scan screen. Scanning is a "Fill from a receipt" bar at the top of the New Expense screen on both clients — web ReceiptScanBar inside CreateExpensePage, mobile ReceiptScanBar inside groups/[groupId]/expenses/new.tsx. The FAB's recent-group cards route straight here.

Flow in the bar:

  1. Credit balance + per-scan cost. If balance < cost:
    • Mobile: a "Watch a video to earn credits" button opens an AdMob rewarded video.
    • Web: copy pointing the user to the mobile app to earn credits.
  2. Capture — Take photo (touch only) or Upload image or PDF:
    • Mobile: expo-image-picker for photo/image (then expo-image-manipulator downscales + compresses to JPEG, long edge ≤ 1600px), and expo-document-picker (type: 'application/pdf') for PDFs (uploaded as-is — re-encoding would destroy the document). These stay separate buttons because they're distinct native pickers.
    • Web: two hidden <input type="file">s — one capture="environment" for the camera, and one combined picker that accepts images and PDF in a single OS dialog (accept="image/*,application/pdf"). They can't merge into one input because the capture flag would also hijack plain file browsing. Images are canvas-downscaled to JPEG; a PDF is uploaded as-is.
  3. Hash + deduplicated upload — calculate SHA-256 over the exact prepared bytes. The upload route returns the existing user-scoped object when those bytes are already present; otherwise S3 verifies the checksum during PUT.
  4. Dispatch + poll — asynchronously invoke a durable scan job, then poll its persisted state: QUEUED → READING_TEXT → ANALYSING → COMPLETE/FAILED. One progress bar spans preparation, upload, OCR, and analysis. A completed identical job is reused when the user, group, currency, language and participant context still match. Reusing the content result avoids repeated OCR/model cost, but the new user attempt still costs one credit. A server-issued attemptId makes transport retries idempotent without making a later selection of the same file free. The shared client retries a transport failure once with the same request and attempt ID. API rejections, returned failed jobs, and requests without an attempt ID are not automatically retried.
  5. Prefill in place — on success the New Expense form re-mounts prefilled from the draft (see Prefill); the user edits and saves through the existing, unchanged create path. No navigation.

PDF support. The same PP-OCRv6 container handles images and PDFs. PDFium renders unencrypted PDFs of one to three pages, up to 10 MB. Pages are OCRed sequentially and joined using [Page N of M] markers before one Ministral request. Rendering targets 200 DPI with a 2,000-pixel longest-edge cap; it does not allocate full-resolution images for all pages together. Corrupt, encrypted, over-limit or partly unreadable documents remain charged and map to 422. Textract is no longer called. The accepted content types and upload flow are unchanged; see ReceiptContentTypeSchema and receipts.storage.ts.

Prefill into the expense form

ExpenseForm (apps/mobile/src/components/expense/ExpenseForm.tsx and the web equivalent) already accepts initial values for edit mode. Extend it to accept an optional initialDraft and seed:

The draft is returned from a temporary scan-job row and passed directly into the form. That row expires after one day and is not an expense until the user reviews and saves it.

Data model

Credits — folded into a UserData single table

The existing Users table is keyed by UserId with no sort key, so it cannot hold a ledger. Rather than add a third user-scoped table, reshape Users into a UserData single table (DataStack, on-demand, RETAIN, tailspend-{stage}-user-data) partitioned by UserId with a sort key — mirroring the GroupData pattern. Profile and credits then share a partition, so a screen can read both in one Query:

SK Purpose
PROFILE The user head row: DisplayName, Email, PreferredCurrency, IsTestAccount, timestamps (migrated from the old Users row).
BALANCE Credit head row. Balance (integer credits) maintained with atomic ADD.
TXN#<createdAt>#<txnId> Append-only ledger entry: Type (EARN_AD / SPEND_SCAN / REFUND), Delta, BalanceAfter, metadata.
ADTXN#<transactionId> Idempotency marker for an AdMob SSV callback. Written with attribute_not_exists so a replayed transaction_id is a no-op.

Identities stays its own table. It is keyed by IdentityKey (<provider>#<subject>), not UserId, and sits on the auth hot path — every request resolves (provider, sub) → UserId via a strongly-consistent GetItem. That access pattern is already optimal and does not share UserData's partition key, so folding it in would only force a generic-PK redesign of a hot-path table for no read-efficiency gain. It is left untouched.

Scope of the Users → UserData change (a one-time, separable migration):

Balance mutations are atomic:

Receipt images — S3

A new bucket tailspend-{stage}-receipts in a new MediaStack:

Scan jobs

SCANJOB#<id> rows live in the caller's UserData partition. The deterministic id hashes the receipt and group context. Rows carry the current stage, progress, the worker context, and finally the draft or a safe error. DynamoDB TTL removes them after one day; this is temporary retry/deduplication state, not archival. The same partition holds SCANCOUNT#<yyyy-mm-dd> daily-limit rows, preserving the UserData single-table design and allowing an atomic counter+credit spend.

API surface

Receipt scanning uses upload tickets and background jobs. The synchronous scan endpoint has been removed; installed clients that used it must update.

New Zod schemas in @repo/api-schema, registered in routes.ts:

Routes (add to all three registries + ApiRoutes/routeSchemas):

Route Auth Grants Notes
POST /scans/uploads JWT S3 get/put (receipts, own prefix) Reuses an existing hash key or returns a checksum-bound presigned PUT.
POST /groups/{groupId}/scan-jobs JWT GroupData: read, UserData: read-write, S3 get, Lambda invoke Creates/reuses a job and enqueues new work. Same expense-write authorization as create-expense.
GET /groups/{groupId}/scan-jobs/{scanJobId} JWT GroupData: read, UserData: read Polls the caller's temporary job status/result.
GET /users/me/credits JWT UserData: read Balance + recent ledger.
GET /ads/admob/ssv NONE UserData: read-write Google's signed callback. Verifies the AdMob SSV signature against Google's public keys, maps custom_data → userId, dedups on transaction_id, credits atomically. This is the second NONE route family alongside the public-invite reads; document it next to them in route-auth.ts.

GET /users/me/credits placement note: it follows the existing /users/me/* convention (get-me, update-settings, list-my-groups).

Processing pipeline

POST /groups/{groupId}/scan-jobs returns quickly after persisting and dispatching it through Lambda asynchronous invocation. The worker runs the paid pipeline with a 15-minute function budget, two async retries and a 16-minute maximum event age. Its live 14-day SQS dead-letter queue retains failed raw ScanJobMessage bodies for replay; the obsolete scan-jobs SQS queue and disabled polling event source are removed. Replay invokes the worker with the original message body, not an SQS Records wrapper. Images and PDFs synchronously invoke the PP-OCRv6 helper Lambda, including PDF rendering. There is no remote OCR polling loop. Clients poll the GET route instead of holding an HTTP request open. The handler follows the existing with-handler + resolveAuthContext + requireAction pattern used by create-expense.ts.

  1. Authorise — resolve (userId, groupId) → participant + role; require WRITE_OWN_EXPENSE.
  2. Charge + limit — atomically increment the UTC-day counter (maximum 50), decrement BALANCE, write the attempt-id marker, and append the ledger row. Stop with 429 at the daily limit or 422 when credits are insufficient.
  3. Fetch document — pass the validated, user-scoped S3 object reference to the receipt OCR helper. The helper reads only its stage's receipts bucket under u/*; client input never selects arbitrary buckets.
  4. OCR — JPEG/PNG and rendered PDF pages use PP-OCRv6 small with classification disabled, one inference thread, and warm model reuse. Its CPU container has 2,048 MiB, x86_64 architecture, a 60-second timeout, and ten reserved executions. The container bundles and checksums both evaluated ONNX weights; runtime model downloads are unnecessary. It bounds input to 10 MiB and 20 million pixels, applies EXIF orientation, and resizes the longest side to at most 2,000 pixels. PDFium 5.14.0 renders one PDF page at a time, with a target 200 DPI and the same 2,000-pixel cap. Native page/bitmap/image resources close between pages. Existing PDF limits remain three pages, unencrypted, 10 MB. Page markers preserve ordering; partial documents and excessive aggregate text are rejected.
  5. Structure — call mistral.ministral-3-8b-instruct using InvokeModel with strict response_format: json_schema, temperature zero, and a 2,048-token output cap. System instructions are separate from JSON-encoded OCR/context. Every schema field is required but unknown values may be null. Quantities are integers; prices are major-unit decimal strings. The prompt distinguishes final totals from tender/subtotals, retains signed discounts and nonzero rounding, excludes payment rows, and forbids invented balancing amounts. Documents above 24,000 OCR characters return a minimal draft rather than silently clipping off the final total. No automatic model retry is made.
  6. Validate + normalise — reject non-stop completions, invalid JSON, prose quantities and invalid field shapes. Validate the public draft after exact decimal-to-minor-unit conversion; never silently round excess precision. Ignore SDN BHD as currency evidence; accept explicit amount-adjacent ISO codes or RM/MYR and otherwise use the group currency. Remove tender and summary items, split explicit quantities deterministically, and preserve negative discounts/rounding. Shape validation does not prove a correct total or complete item list. An incomplete draft is still charged and user-reviewed. Included VAT/GST disclosures are removed only when the OCR explicitly marks tax as included, every amount is known, and the remaining non-tax items already sum exactly to the grand total. Additional tax needed to reach that total is preserved, as are unresolved mismatches; no balancing item is added.
  7. Respond — return the draft + remainingCredits. Failures remain charged; the paid attempt covers the abuse/cost risk of arbitrary uploaded content.

The image is left to the S3 lifecycle rule to expire; no explicit delete is required (a best-effort delete-after-success is an optional enhancement).

Receipt OCR packaging

apps/receipt-ocr contains the CPU Lambda image, checksum verification, and offline smoke instructions. Separate receipt OCR keeps its native dependencies and 2 GiB allocation out of the Node job worker. The worker still bills its 1 GiB while waiting for OCR and Bedrock; this change does not implement Step Functions. Images and PDFs share the helper; there is no automatic Textract fallback or paid retry on OCR failure. Embedded PDF text is not extracted separately in this phase; visible pages are always rendered.

Grants and runtime configuration

Only the job worker receives SCAN_OCR_FUNCTION_NAME, permission to invoke that helper, SCAN_MODEL_ID, and permission to invoke its configured Ministral foundation-model ARN. The helper receives RECEIPTS_BUCKET_NAME and S3 GetObject on tailspend-{stage}-receipts/u/*; it receives no Bedrock, Textract, DynamoDB, Scheduler or credit access. The worker retains its existing receipt validation permissions. All Textract start/get/detect permissions are removed, along with the serving Textract adapter. The Textract SDK is a development-only dependency for the historical evaluation runner. The Node New Relic layer continues to wrap the job worker; the Python helper uses CloudWatch logs.

Credit economics

Costs are tiny and configured as constants (CREDIT_COSTS), tunable without a schema change:

Security & privacy

Testing

Rollout

Original v1 implementation plan

The historical tasks below describe the initial Textract/Gemma version. Issue #694 changes the image OCR and structurer as described in the pipeline above.

Phased so each task is an independently reviewable PR. IDs (OCR-n) are used on the GitHub Projects board.

Phase 0 — Contracts & shared constants

Phase 1 — Infrastructure

Phase 2 — Backend services & handlers

Phase 3 — Shared client hooks

Phase 4 — Mobile

Phase 5 — Web

Phase 6 — Hardening & launch

Open questions

Item-specific discounts and service charges

The expense editor uses signed item amounts on web and mobile: a discount is a negative item, and Add discount opens that same editor with a negative amount and the By item totals split. Only the selected item is expanded. Add controls sit below the list and focus/scroll to the new editor. Zero-cost receipt modifiers can be retained without preventing submission.

ITEM_TOTALS.itemIndexes optionally selects zero-based positions in the submitted expense items array. When absent, the basis is all positive purchase lines before discounts. When present, the adjustment follows only those purchases and their assigned people. Other ITEM_TOTALS adjustments cannot be referenced, preventing cycles. Clients remap indexes when removing rows; deleting all referenced purchases requires choosing a new scope. Participant transfers preserve the scope. Rounding for item-total adjustments skips participants with no qualifying spend.

Receipt structuring returns optional FOOD/DRINK/OTHER categories and FOOD/ALL/UNKNOWN discount scope. The shared draft mapper resolves food scope after quantity expansion, so repeated drinks remain separately assignable and excluded from food discounts. Scan review shows the printed total, differences from the editable items, and reminders to check discount eligibility. Model inference is reviewable, not a guarantee of the restaurant's promotion rules.

For the Ikkan example, purchases are £12.95 + £3.45 + £13.95 + £3.45 = £33.80. Keep the printed discount of −£13.44 linked to the two bowls and the £3.52 service charge proportional to all purchases before discounts, including drinks. The total is £23.88. Do not infer a 50% discount and replace it with £13.45; do not recalculate the service charge from an assumed rate. Paid Amount 0.00 is tender information, not the expense total. Food eligibility in this image is inferred and flagged for review because the receipt does not explicitly print “food only”.

View Markdown source