AI menu scan (web + mobile)

Summary

Owners, staff, and fans can photograph a food-truck menu (camera or photo roll). Photos upload to Convex file storage; a Gemini vision action parses items, prices, and photo placement. The owner reviews at whole-menu and per-item levels before rows commit to menuItems. Ambiguous product photos land in a holding area for manual mapping.

Backend (shared)

  • Convex tables: menuSubmissions, menuSubmissionItems, menuPhotoHolding

  • menuStorage.generateMenuPhotoUploadUrl — Convex file upload URLs

  • menuParseAction.parseMenuSubmissionInternal — Gemini 2.0 Flash (GEMINI_API_KEY on Convex)

  • Fan/worker submissions set submitterKind: fan and require owner approval

Web (site-nextjs)

  • Manage → Menu scan (AI) (/manage/menu)

  • Fan helper: /submit-menu/[truckId] for claimed trucks

  • REST API under /api/menu-submissions/* (Bearer auth — same as mobile)

Mobile (ftn-app)

  • Home → Scan menu and Review pending submissions

  • image_picker for camera / gallery

  • MenuSubmissionService calls the same REST API

Setup

  1. Enable Generative Language API in Google Cloud and create an API key.

  2. Set GEMINI_API_KEY on the Convex deployment (Production until release).

  3. Optional: GEMINI_MENU_MODEL (default gemini-2.0-flash).

Photo enhancement (Imagen / separate crop pipeline) is scaffolded via menuPhotoHolding.enhancementStatus; full auto-enhance is follow-up work.

Not in scope yet

  • Pushing approved items to Square Catalog

  • Convex Auth on mutations (still server-gated via Next.js / bearer)