AI Services Market › Use cases › Life Admin & Home
Life Admin & HomeDone with Gemini Apr 1, 2025
Fill out forms from images with a form-filling agent
The author used Gemini to build an agent that plans the fields in a form image and matches them to bounding boxes, for a friend who wanted forms filled automatically. · Apr 1, 2025
- What They Did
- The author said a friend wanted JPEG/PNG forms filled automatically, so they reused prior work with OmniParser and built an annotated dataset from IRS and OPM forms with clearly defined fields. They used Gemini to identify what fields were present in a document and match those fields to bounding boxes, noting GPT-4o plus YOLO could have done the same job.
- How It Went
- The post presents it as a working approach for automating tedious form filling, especially in manual supply-chain workflows. It does not give accuracy numbers, speed, or failure cases in the thread text shown.
- Worth Knowing
- The main catch is that they first compiled their own annotated dataset from real forms, rather than relying on prompting alone.
Try it yourself with Gemini
Build me a small tool that takes an image of a form at [file path], detects each blank field and its label, and draws bounding boxes where answers should go. Then fill the fields using the details in [my info file], and save the completed form as a new image at [output path]. Show me the result for review before treating it as done, and stop once the filled form lines up with every field.