Receipt scanning & AI expense prefill
This document specifies the scan-a-receipt feature: a user photographs or uploads a receipt, TailSpend runs OCR and an LLM over it, and the result prefills the normal create-expense form. The feature is metered with credits that users earn by watching rewarded video ads.
This describes the implementation, including the asynchronous scan-job flow, content deduplication, credit handling, and temporary retention. The issue #694 OCR/model changes below are implemented in source. Deployment was authorized on 22 October 2026 through the normal dev → beta → production pipeline; the GitHub Actions deployment run is the source of truth for rollout status.
For the historical baseline and alternatives evaluated in issue #694, see the receipt OCR evaluation. It records research and a bounded live comparison, including extraction failures in both alternatives and the previous Gemma baseline. The replacement uses PP-OCRv6 small for images and rendered PDF pages, with Ministral 3 8B for structuring. Textract is removed from the serving path. Four development receipts pass exact totals and item-cost lists with the new configuration; representative accuracy and deployed Lambda performance remain unmeasured.
Goals
- From the dashboard, one tap (a floating + action) starts a capture flow.
- Pick a group, then take a photo or choose an image (mobile and web).
- OCR images and bounded rendered PDF pages with PP-OCRv6 small, then
structure the text with Ministral 3 8B Instruct on Amazon Bedrock
(
mistral.ministral-3-8b-instruct) using structured outputs, producing a draft that maps onto the existingCreateExpenseRequest. - Hand the draft to the existing
ExpenseFormso the user reviews and edits before saving. The scan never writes an expense directly — a human always confirms. - Meter scans with credits. Users earn credits by watching AdMob rewarded videos on mobile. Web can spend a shared balance but cannot earn (no web rewarded-ad SDK; see Decisions).
- Charge every submitted scan attempt, including duplicate, empty, random, corrupt, and failed inputs, and cap each user at 50 paid attempts per UTC day.
Non-goals (v1)
- No automatic categorisation/merchant enrichment beyond what the model returns (the expense model has no category field today — out of scope).
- No paid credit top-ups / IAP. Earning is rewarded-video only in v1.
- No long-term receipt archival — uploads and scan-job drafts expire after one day.
Decisions (locked)
These were settled before design; recorded here so the plan is unambiguous.
| Decision | Choice | Consequence |
|---|---|---|
| LLM hosting | Ministral 3 8B Instruct on Bedrock (mistral.ministral-3-8b-instruct) |
Serverless InvokeModel with native JSON-schema output; OCR text only. USD $0.15 per million input and output tokens at standard us-east-1 rates. |
| OCR engine | PP-OCRv6 small for JPEG/PNG and rendered PDF pages | A dedicated 2 GiB x86 Python container Lambda bundles the evaluated ONNX weights and PDFium. PDFs render sequentially with explicit page boundaries; no Textract calls. |
| Earning surface | Mobile only (AdMob rewarded video) | Balance is shared; web spends, mobile earns. Web shows "earn on mobile" messaging when short. |
| Ad reward trust | AdMob Server-Side Verification (SSV) | Credits are granted by a Google→backend signed callback, never by the client. Replay-protected by transaction_id. |
| Region | us-east-1 | Receipt OCR Lambda and Bedrock remain in the deployment region. |
Where this sits in the system
The feature is additive and respects every existing contract discipline:
- New entities are Zod schemas in
@repo/api-schema, wired into@repo/openapi'sroutes.ts, which regeneratesopenapi.jsonand the typed@repo/api-client. Clients consume new TanStack Query hooks from@repo/api-hooks. - Every route is added to the synth-validated authorization and grant registries
(
route-auth.ts,route-grants.ts). Only routes with separate Lambda functions also appear inEXCEPTIONAL_ROUTE_HANDLERSinrest-route-handlers.ts. Exact OpenAPI/full-registry parity and exceptional-subset parity are enforced at synth. - A new CDK stack (
MediaStack) owns the S3 receipts bucket; the existingUserstable is reshaped into aUserDatasingle table that also holds the credit balance + ledger (see Data model). Scan/credit Lambdas reuse the existing per-routeServiceLambdaconstruct, extended to grant non-DynamoDB AWS permissions (S3, OCR Lambda, Bedrock) — see Grant-model change.

User experience
Entry point — dashboard FAB
A floating + button sits bottom-right on both dashboards
(web FabMenu,
mobile fab-menu). Tapping it opens
a "Add an expense" menu: cards for the most recently-used groups (tap one to
jump into its New Expense screen, where the scan buttons live), plus New
group and — when there are more groups than shown — All groups. The menu
no longer routes to a standalone scan screen; scanning happens on New Expense.
Scan is part of the New Expense screen
There is no separate scan screen. Scanning is a "Fill from a receipt" bar at
the top of the New Expense screen on both clients — web
ReceiptScanBar inside
CreateExpensePage, mobile
ReceiptScanBar
inside groups/[groupId]/expenses/new.tsx.
The FAB's recent-group cards route straight here.
Flow in the bar:
- Credit balance + per-scan cost. If balance < cost:
- Mobile: a "Watch a video to earn credits" button opens an AdMob rewarded video.
- Web: copy pointing the user to the mobile app to earn credits.
- Capture — Take photo (touch only) or Upload image or PDF:
- Mobile:
expo-image-pickerfor photo/image (thenexpo-image-manipulatordownscales + compresses to JPEG, long edge ≤ 1600px), andexpo-document-picker(type: 'application/pdf') for PDFs (uploaded as-is — re-encoding would destroy the document). These stay separate buttons because they're distinct native pickers. - Web: two hidden
<input type="file">s — onecapture="environment"for the camera, and one combined picker that accepts images and PDF in a single OS dialog (accept="image/*,application/pdf"). They can't merge into one input because the capture flag would also hijack plain file browsing. Images are canvas-downscaled to JPEG; a PDF is uploaded as-is.
- Mobile:
- Hash + deduplicated upload — calculate SHA-256 over the exact prepared bytes. The upload route returns the existing user-scoped object when those bytes are already present; otherwise S3 verifies the checksum during PUT.
- Dispatch + poll — asynchronously invoke a durable scan job, then poll its persisted state:
QUEUED → READING_TEXT → ANALYSING → COMPLETE/FAILED. One progress bar spans preparation, upload, OCR, and analysis. A completed identical job is reused when the user, group, currency, language and participant context still match. Reusing the content result avoids repeated OCR/model cost, but the new user attempt still costs one credit. A server-issuedattemptIdmakes transport retries idempotent without making a later selection of the same file free. The shared client retries a transport failure once with the same request and attempt ID. API rejections, returned failed jobs, and requests without an attempt ID are not automatically retried. - Prefill in place — on success the New Expense form re-mounts prefilled from the draft (see Prefill); the user edits and saves through the existing, unchanged create path. No navigation.
PDF support. The same PP-OCRv6 container handles images and PDFs. PDFium
renders unencrypted PDFs of one to three pages, up to 10 MB. Pages are OCRed
sequentially and joined using [Page N of M] markers before one Ministral
request. Rendering targets 200 DPI with a 2,000-pixel longest-edge cap; it does
not allocate full-resolution images for all pages together. Corrupt, encrypted,
over-limit or partly unreadable documents remain charged and map to 422.
Textract is no longer called. The accepted content types and upload flow are
unchanged; see ReceiptContentTypeSchema and
receipts.storage.ts.
Prefill into the expense form
ExpenseForm
(apps/mobile/src/components/expense/ExpenseForm.tsx
and the web equivalent) already accepts initial values for edit mode. Extend it
to accept an optional initialDraft and seed:
label← merchant / receipt title.currency← detected currency, else the group currency.items[]← one row per detected line item (label,costin minor units), each defaulting to anEQUALsplit across all participants. Quantity lines are expanded into one row per physical unit: for example,2 x Coke 6.00becomes twoCokerows at3.00each. Bare receipt quantities are handled too:2 Coke 3.00becomes twoCokerows at1.50. The model identifies the quantity and copies the printed line total; OCR evidence provides a deterministic fallback, and the server performs the division in minor units so rounding is deterministic and the rows still sum exactly to the receipt.payments[]← single payer = current user,amount= items total (the existing default).- Per-field low-confidence flags highlight cells the model was unsure about so the user knows what to check. Drafts are best-effort: every field is optional and the form must render with whatever subset arrived.
The draft is returned from a temporary scan-job row and passed directly into the form. That row expires after one day and is not an expense until the user reviews and saves it.
Data model
Credits — folded into a UserData single table
The existing Users table is keyed by UserId with no sort key, so it
cannot hold a ledger. Rather than add a third user-scoped table, reshape Users
into a UserData single table (DataStack,
on-demand, RETAIN, tailspend-{stage}-user-data) partitioned by UserId with
a sort key — mirroring the GroupData pattern. Profile and credits then share a
partition, so a screen can read both in one Query:
| SK | Purpose |
|---|---|
PROFILE |
The user head row: DisplayName, Email, PreferredCurrency, IsTestAccount, timestamps (migrated from the old Users row). |
BALANCE |
Credit head row. Balance (integer credits) maintained with atomic ADD. |
TXN#<createdAt>#<txnId> |
Append-only ledger entry: Type (EARN_AD / SPEND_SCAN / REFUND), Delta, BalanceAfter, metadata. |
ADTXN#<transactionId> |
Idempotency marker for an AdMob SSV callback. Written with attribute_not_exists so a replayed transaction_id is a no-op. |
Identities stays its own table. It is keyed by IdentityKey
(<provider>#<subject>), not UserId, and sits on the auth hot path — every
request resolves (provider, sub) → UserId via a strongly-consistent GetItem.
That access pattern is already optimal and does not share UserData's partition
key, so folding it in would only force a generic-PK redesign of a hot-path table
for no read-efficiency gain. It is left untouched.
Scope of the Users → UserData change (a one-time, separable migration):
TailSpendTablesswaps its"Users"member for"UserData"; the env-var map (USERS_TABLE_NAME→USER_DATA_TABLE_NAME) and grant ARN-suffix map (users→user-data) inservice-lambda.tsupdate to match (the construct already asserts env coverage for every table at synth).- Existing routes that grant
Users(route-grants.ts:POST /groups, the/users/*routes, join-requests) switch their grant key toUserData. The users repository reads the profile withSK = PROFILEinstead of a bareGetItem, and the POST_AUTHENTICATION Lambda writes thePROFILErow (still in its existingTransactWriteItemswith the Identities row). - DynamoDB cannot add a sort key to a table in place, so this is a new table +
backfill: create
UserData, copy eachUsersrow in as aPROFILErow, cut the handlers over, then retire the oldUserstable. Because credits are brand new, no credit data migrates — only the profile backfill. This migration is independent of the scan feature and can land first (task OCR-3).
Balance mutations are atomic:
Spend (reserve) — a single
TransactWriteItemsthat writes aSPEND#<attemptId>:<jobId>idempotency row (guarded onattribute_not_exists), incrementsSCANCOUNT#<yyyy-mm-dd>subject toCount < 50, decrements the guarded balance, and appends the ledger row. All four writes succeed or fail together. The counter uses UTC and expires via TTL after its enforcement window.Every attempt is final — empty/random images, missing objects, corrupt documents, OCR/model failures, and queue failures are not refunded. This prevents arbitrary inputs from being used to consume paid services for free.
Attempt versus content identity —
attemptIdidentifies one user action; the hash-derivedjobIdidentifies reusable work. Replaying the same pair is free and does not rerun work. A new attempt against the same content is charged but can immediately reuse the stored result.Earn — the SSV handler writes the
ADTXN#…idempotency row and the balanceADDin a singleTransactWriteItems, so a credit is granted exactly once per ad transaction.
Receipt images — S3
A new bucket tailspend-{stage}-receipts in a new MediaStack:
- Block all public access, SSE (S3-managed or KMS), TLS-only bucket policy.
- Lifecycle rule expires objects after a short window (e.g. 1 day) — images
are needed only for the duration of the scan.
RETAINis not used; these are transient and PII-bearing. - CORS allowing
PUTfrom the web origin and mobile so clients upload directly via presigned URL (no image bytes through API Gateway/Lambda). - Hash-aware keys are namespaced per user:
u/<userId>/sha256/<contentHash>.<ext>. Legacy clients still receive UUID keys. The presign handler owns every key, and S3 validates the SHA-256 checksum on new uploads.
Scan jobs
SCANJOB#<id> rows live in the caller's UserData partition. The deterministic
id hashes the receipt and group context. Rows carry the current stage, progress,
the worker context, and finally the draft or a safe error. DynamoDB TTL removes
them after one day; this is temporary retry/deduplication state, not archival.
The same partition holds SCANCOUNT#<yyyy-mm-dd> daily-limit rows, preserving
the UserData single-table design and allowing an atomic counter+credit spend.
API surface
Receipt scanning uses upload tickets and background jobs. The synchronous scan endpoint has been removed; installed clients that used it must update.
New Zod schemas in @repo/api-schema, registered in
routes.ts:
packages/api-schema/src/scans.tsCreateUploadTicketRequestSchema→{ contentType, byteSize, contentHash? }UploadTicketSchema→{ attemptId, objectKey, uploadRequired, uploadUrl?, expiresAt?, checksumSha256? }CreateScanJobRequestSchema→{ objectKey, contentHash, attemptId? }ScanJobSchema→ persisted stage/progress plus result or safe errorExpenseDraftSchema→ best-effort, all-optional mirror ofCreateExpenseRequestplus per-field confidence; the shape the form consumes.ScanResultSchema→{ draft, creditsCharged, remainingCredits, rawTextHash? }
packages/api-schema/src/credits.tsCreditBalanceSchema→{ balance, ledger: CreditLedgerEntry[] }CreditLedgerEntrySchema, and sharedCREDIT_COSTSconstants (SCAN_COST,AD_REWARD) so client and server agree.
Routes (add to all three registries + ApiRoutes/routeSchemas):
| Route | Auth | Grants | Notes |
|---|---|---|---|
POST /scans/uploads |
JWT | S3 get/put (receipts, own prefix) |
Reuses an existing hash key or returns a checksum-bound presigned PUT. |
POST /groups/{groupId}/scan-jobs |
JWT | GroupData: read, UserData: read-write, S3 get, Lambda invoke |
Creates/reuses a job and enqueues new work. Same expense-write authorization as create-expense. |
GET /groups/{groupId}/scan-jobs/{scanJobId} |
JWT | GroupData: read, UserData: read |
Polls the caller's temporary job status/result. |
GET /users/me/credits |
JWT | UserData: read |
Balance + recent ledger. |
GET /ads/admob/ssv |
NONE | UserData: read-write |
Google's signed callback. Verifies the AdMob SSV signature against Google's public keys, maps custom_data → userId, dedups on transaction_id, credits atomically. This is the second NONE route family alongside the public-invite reads; document it next to them in route-auth.ts. |
GET /users/me/credits placement note: it follows the existing /users/me/*
convention (get-me, update-settings, list-my-groups).
Processing pipeline
POST /groups/{groupId}/scan-jobs returns quickly after persisting and
dispatching it through Lambda asynchronous invocation. The worker runs the paid
pipeline with a 15-minute function budget, two async retries and a 16-minute
maximum event age. Its live 14-day SQS dead-letter queue retains failed raw
ScanJobMessage bodies for replay; the obsolete scan-jobs SQS queue and disabled
polling event source are removed. Replay invokes the worker with the original
message body, not an SQS Records wrapper. Images and PDFs synchronously invoke the
PP-OCRv6 helper Lambda, including PDF rendering. There is no remote OCR polling loop. Clients poll the GET
route instead of holding an HTTP request open.
The handler follows the existing
with-handler +
resolveAuthContext + requireAction pattern used by
create-expense.ts.
- Authorise — resolve
(userId, groupId)→ participant + role; requireWRITE_OWN_EXPENSE. - Charge + limit — atomically increment the UTC-day counter (maximum 50),
decrement
BALANCE, write the attempt-id marker, and append the ledger row. Stop with429at the daily limit or422when credits are insufficient. - Fetch document — pass the validated, user-scoped S3 object reference to
the receipt OCR helper. The helper reads only its stage's receipts
bucket under
u/*; client input never selects arbitrary buckets. - OCR — JPEG/PNG and rendered PDF pages use PP-OCRv6 small with classification disabled, one inference thread, and warm model reuse. Its CPU container has 2,048 MiB, x86_64 architecture, a 60-second timeout, and ten reserved executions. The container bundles and checksums both evaluated ONNX weights; runtime model downloads are unnecessary. It bounds input to 10 MiB and 20 million pixels, applies EXIF orientation, and resizes the longest side to at most 2,000 pixels. PDFium 5.14.0 renders one PDF page at a time, with a target 200 DPI and the same 2,000-pixel cap. Native page/bitmap/image resources close between pages. Existing PDF limits remain three pages, unencrypted, 10 MB. Page markers preserve ordering; partial documents and excessive aggregate text are rejected.
- Structure — call
mistral.ministral-3-8b-instructusingInvokeModelwith strictresponse_format: json_schema, temperature zero, and a 2,048-token output cap. System instructions are separate from JSON-encoded OCR/context. Every schema field is required but unknown values may be null. Quantities are integers; prices are major-unit decimal strings. The prompt distinguishes final totals from tender/subtotals, retains signed discounts and nonzero rounding, excludes payment rows, and forbids invented balancing amounts. Documents above 24,000 OCR characters return a minimal draft rather than silently clipping off the final total. No automatic model retry is made. - Validate + normalise — reject non-
stopcompletions, invalid JSON, prose quantities and invalid field shapes. Validate the public draft after exact decimal-to-minor-unit conversion; never silently round excess precision. IgnoreSDN BHDas currency evidence; accept explicit amount-adjacent ISO codes or RM/MYR and otherwise use the group currency. Remove tender and summary items, split explicit quantities deterministically, and preserve negative discounts/rounding. Shape validation does not prove a correct total or complete item list. An incomplete draft is still charged and user-reviewed. Included VAT/GST disclosures are removed only when the OCR explicitly marks tax as included, every amount is known, and the remaining non-tax items already sum exactly to the grand total. Additional tax needed to reach that total is preserved, as are unresolved mismatches; no balancing item is added. - Respond — return the draft +
remainingCredits. Failures remain charged; the paid attempt covers the abuse/cost risk of arbitrary uploaded content.
The image is left to the S3 lifecycle rule to expire; no explicit delete is required (a best-effort delete-after-success is an optional enhancement).
Receipt OCR packaging
apps/receipt-ocr contains the CPU Lambda image,
checksum verification, and offline smoke instructions. Separate receipt OCR keeps
its native dependencies and 2 GiB allocation out of the Node job worker. The
worker still bills its 1 GiB while waiting for OCR and Bedrock; this change does
not implement Step Functions. Images and PDFs share the helper; there is no
automatic Textract fallback or paid retry on OCR failure. Embedded PDF text is
not extracted separately in this phase; visible pages are always rendered.
Grants and runtime configuration
Only the job worker receives SCAN_OCR_FUNCTION_NAME, permission to
invoke that helper, SCAN_MODEL_ID, and permission to invoke its configured
Ministral foundation-model ARN. The helper receives RECEIPTS_BUCKET_NAME and
S3 GetObject on tailspend-{stage}-receipts/u/*; it receives no Bedrock,
Textract, DynamoDB, Scheduler or credit access. The worker retains its existing
receipt validation permissions. All Textract start/get/detect permissions are
removed, along with the serving Textract adapter. The Textract SDK is a
development-only dependency for the historical evaluation runner. The Node New Relic layer
continues to wrap the job worker; the Python helper uses CloudWatch logs.
Credit economics
Costs are tiny and configured as constants (CREDIT_COSTS), tunable without a
schema change:
- Image scans incur OCR Lambda compute plus Ministral tokens. At 1,000 input and 1,000 output tokens per receipt, model cost is $0.30 per 1,000 receipts. An illustrative two-second OCR invocation at 2 GiB x86 costs about $0.067 per 1,000, excluding request charges. The waiting Node worker, initialization, validation, storage, logging and retries add cost; this is not measured Lambda billing. PDFs incur page-rendering/OCR compute and additional model tokens, eliminating Textract's $1.50 per 1,000-page fee. Multi-page PDFs consume more compute/tokens while spending the same scan credit. PDF costs remain unmeasured on Lambda; synthetic container OCR timings are not production billing estimates. See the historical measured comparison and its implementation follow-up for the distinction between evidence and estimates.
- A rewarded video earns the app a fraction of a cent to a few cents (eCPM dependent).
- Suggested starting ratio: 1 scan = 1 credit, 1 rewarded video = N credits (pick N so one video comfortably funds several scans and stays ahead of AWS cost). New users can be seeded a few free credits to try the feature. Revisit the ratio once real ad revenue and AWS cost are observed.
Security & privacy
- No client-trusted credits. Balances move only via the conditional-write spend path and the signature-verified SSV callback. The client never asserts "I earned/spent N".
- AdMob SSV signature is verified against Google's published keys at
https://www.gstatic.com/admob/reward/verifier-keys.json(no apex redirect dependency); thetransaction_ididempotency row prevents replay/double-credit.custom_databinds the reward to the authenticated user who requested the ad. - Receipts are PII (names, card last-4, locations). Mitigations: private + SSE bucket, short lifecycle expiry, per-user key prefixes, presigned URLs (bytes never transit our Lambdas on upload), and never logging raw OCR text or model output at INFO — log a hash/length instead.
- Bedrock input/output is not used for model training (AWS contractual); keep all calls in-region (us-east-1).
- Abuse / cost ceilings: per-user daily scan cap and a CloudWatch budget alarm on OCR Lambda/Bedrock spend; rate-limit the upload + scan routes.
- DoS surface: the
NONESSV route is internet-facing — reject unsigned or stale callbacks fast and cheaply before any DynamoDB write.
Testing
- Schema/contract:
api-schemaunit tests for the new schemas; the existingvalidate-route-registriestest guards that the new routes appear in all three registries and the OpenAPI doc. - Pipeline unit tests: inject mock
ocr/bedrock/creditsservices into the scan handler factory (same DI pattern ascreateExpenseHandler); assert charge-on-failure, the atomic 50/day gate, currency/participant clamping, and minor-unit maths. - SSV tests: valid signature credits once; replayed
transaction_idis a no-op; bad signature is rejected with no write. - Client: form-prefill mapping tests (draft → form state), insufficient-credit gating, and the earn-then-scan happy path.
- Integration (
apps/api-integration-tests): end-to-end against a deployed dev stage using a fixture receipt and the integration-test user; assert a draft returns and a credit is charged.
Rollout
- Ship behind a feature flag / env gate so the FAB and routes can be dark in
prod until the pipeline is validated on
dev/beta. - AdMob needs real ad-unit ids and the SSV callback URL registered in the AdMob
console per platform; keep ids in client env (
EXPO_PUBLIC_*) and the SSV verification config in SSM, consistent with existing secret handling. - Mobile gains native deps (
react-native-google-mobile-ads,expo-image-picker,expo-image-manipulator), so a new EAS build is required — this cannot ship as an OTA update. Plan a store release.
Original v1 implementation plan
The historical tasks below describe the initial Textract/Gemma version. Issue #694 changes the image OCR and structurer as described in the pipeline above.
Phased so each task is an independently reviewable PR. IDs (OCR-n) are used on
the GitHub Projects board.
Phase 0 — Contracts & shared constants
- OCR-1 —
api-schema: addscans.ts+credits.tsschemas andCREDIT_COSTS; export fromindex.ts. Unit tests. - OCR-2 —
@repo/openapi: register the four routes + request/response schemas inroutes.ts; regenerateopenapi.jsonand the typed client.
Phase 1 — Infrastructure
- OCR-3 —
Users → UserDatasingle-table migration (separable, lands first): create theUserDatatable (PKUserId+ SK), backfill eachUsersrow as aPROFILErow, swapTailSpendTables/env-var/grant-suffix maps and theUsersgrants inroute-grants.ts, update the users repository and the POST_AUTHENTICATION Lambda to write/readSK = PROFILE, then retireUsers. Adds theBALANCE/TXN#/ADTXN#credit row kinds in the same table. Identities is left as-is. - OCR-4 —
MediaStack: S3 receipts bucket (private, SSE, lifecycle expiry, CORS); wire intomain.ts. - OCR-5 — Grant-model extension: add the
awscapability block toRouteGrants, emit S3/Textract/Bedrock policies inServiceLambda, update the registry validator + tests. Add the Bedrock model-id config module (defaultgoogle.gemma-3-4b-it).
Phase 2 — Backend services & handlers
- OCR-6 — Credits service + repository (atomic spend/daily limit/earn, ledger, ad-txn idempotency) with unit tests.
- OCR-7 —
POST /scans/uploadshandler (presigned PUT, per-user key). - OCR-8 — OCR adapter (Textract
DetectDocumentText→{ rawText }) behind a swappable interface. - OCR-9 — Bedrock/Gemma structured-output adapter + prompt
(
google.gemma-3-4b-it);ExpenseDraftparsing/normalisation. - OCR-10 —
POST /groups/{groupId}/scan-jobsauthorizes, charges and queues work; the background worker runs OCR → structure → validate. Clients pollGET /groups/{groupId}/scan-jobs/{scanJobId}for progress and the result. - OCR-11 —
GET /users/me/creditshandler. - OCR-12 —
GET /ads/admob/ssvhandler (signature verification + idempotent atomic credit). Register as aNONEroute.
Phase 3 — Shared client hooks
- OCR-13 —
@repo/api-hooks:useCreateUploadTicket,useCreateScanJob,useScanJob,useMyCredits(+ cache invalidation), and the draft→form mapping helper.
Phase 4 — Mobile
- OCR-14 — Add native deps; dashboard FAB →
scan/new; register the route. - OCR-15 — Scan screen: group picker, capture/upload (
expo-image-picker+expo-image-manipulator), upload→scan→navigate-to-prefilled-form. - OCR-16 — AdMob rewarded-video integration + credit UI (balance, earn button, insufficient-credit gating).
- OCR-17 — Extend
ExpenseFormto accept and renderinitialDraft(low-confidence highlighting).
Phase 5 — Web
- OCR-18 — Dashboard FAB →
/scan; register the route. - OCR-19 — Scan page: group picker, file-input capture + canvas downscale, upload→scan→navigate-to-prefilled-form; credit balance display with "earn on mobile" messaging.
- OCR-20 — Extend web
ExpenseFormto accept and renderinitialDraft.
Phase 6 — Hardening & launch
- OCR-21 — Per-user scan rate limit + daily cap; CloudWatch budget alarms on Textract/Bedrock spend.
- OCR-22 — Integration tests (
api-integration-tests) for the scan + credit happy path againstdev. - OCR-23 — Feature-flag wiring, AdMob ad-unit + SSV config per stage, docs/runbook updates; new EAS build + store release.
Open questions
- Credit ratio (free seed, scan cost, ad reward) — start with the suggestion
above and tune against observed ad revenue vs measured full-pipeline AWS cost.
Updated for #694: image OCR is PP-OCRv6 small; structuring uses
mistral.ministral-3-8b-instruct. PDFs render locally and use the same OCR; the serving Textract path and IAM grants are removed. Source changes are not evidence of production deployment or measured extraction-quality improvement.
Item-specific discounts and service charges
The expense editor uses signed item amounts on web and mobile: a discount is a negative item, and Add discount opens that same editor with a negative amount and the By item totals split. Only the selected item is expanded. Add controls sit below the list and focus/scroll to the new editor. Zero-cost receipt modifiers can be retained without preventing submission.
ITEM_TOTALS.itemIndexes optionally selects zero-based positions in the submitted
expense items array. When absent, the basis is all positive purchase lines before
discounts. When present, the adjustment follows only those purchases and their
assigned people. Other ITEM_TOTALS adjustments cannot be referenced, preventing
cycles. Clients remap indexes when removing rows; deleting all referenced purchases
requires choosing a new scope. Participant transfers preserve the scope. Rounding
for item-total adjustments skips participants with no qualifying spend.
Receipt structuring returns optional FOOD/DRINK/OTHER categories and FOOD/ALL/UNKNOWN discount scope. The shared draft mapper resolves food scope after quantity expansion, so repeated drinks remain separately assignable and excluded from food discounts. Scan review shows the printed total, differences from the editable items, and reminders to check discount eligibility. Model inference is reviewable, not a guarantee of the restaurant's promotion rules.
For the Ikkan example, purchases are £12.95 + £3.45 + £13.95 + £3.45 = £33.80.
Keep the printed discount of −£13.44 linked to the two bowls and the £3.52 service
charge proportional to all purchases before discounts, including drinks. The total
is £23.88. Do not infer a 50% discount and replace it with £13.45; do not recalculate
the service charge from an assumed rate. Paid Amount 0.00 is tender information,
not the expense total. Food eligibility in this image is inferred and flagged for
review because the receipt does not explicitly print “food only”.