All Recipes

18 recipes

Extract Key Clauses from Contract Images

Have your agent automatically extract parties, amounts, dates, penalty clauses, and jurisdiction from scanned contracts and output structured JSON for approval workflows or risk systems.

Claude 3.5 Sonnet GPT-4o Gemini 1.5 Pro

Document Understanding Intermediate

Parse Medical Lab Report Images

Extract test item names, values, units, reference ranges, and abnormal flags from blood work and biochemistry report photos — structured output for health management agents.

GPT-4o Claude 3.5 Sonnet

Chart & Table Intermediate

Extract Data from Bar Charts

Have your agent pull numerical data — labels, values, series names — out of bar chart images from reports, dashboards, or slides and return structured JSON.

GPT-4o Gemini 1.5 Pro Claude 3.5 Sonnet

Multi-image Reasoning Beginner

Before and After Image Comparison

Have your agent compare two versions of an image and output a structured diff of what was added, removed, or modified.

GPT-4o Gemini 1.5 Pro Claude 3.5 Sonnet

Chart & Table Intermediate

Read Data from Dashboard Screenshots

Have your agent extract KPI values, alert states, and metric readings from Grafana, DataDog, or business analytics dashboard screenshots to trigger downstream actions.

GPT-4o Gemini 1.5 Pro

Structured Output Intermediate

Structured Output with Confidence Scores

When extracting structured data from low-quality images (blurry scans, poor lighting), have the model report a per-field confidence score so the system can automatically flag fields for human review.

GPT-4o Claude 3.5 Sonnet

Computer Use Beginner

Detect and Handle Error Dialogs

When an unexpected error dialog appears mid-task, have the agent classify it (recoverable vs fatal) and automatically dismiss, retry, or escalate to a human.

GPT-4o Claude 3.5 Sonnet

Computer Use Intermediate

Automate File Browser Navigation

Have an agent navigate Windows Explorer, macOS Finder, or a Linux file manager to locate files and perform move, rename, or delete operations based on the visual window state.

Claude 3.5 Sonnet GPT-4o

UI Grounding Intermediate

Detect and Auto-fill Form Fields

Have your agent screenshot a form, extract all field labels and types as structured data, then fill them based on provided values — works on registration, checkout, and survey forms where the structure is unknown in advance.

GPT-4o Claude 3.5 Sonnet

Structured Output Intermediate

Enforce JSON Schema Output from VLMs

Make a VLM return data that strictly conforms to a predefined JSON schema — matching specific field names, types, and required fields — for integration with typed systems like databases, APIs, and TypeScript types.

GPT-4o Claude 3.5 Sonnet Gemini 1.5 Pro

UI Grounding Beginner

Parse Web Navigation Structure

Have your agent screenshot a webpage and extract the navigation menu structure (top nav, sidebar, breadcrumbs) as a tree JSON, so the agent understands the site map and can plan subsequent navigation steps.

GPT-4o Claude 3.5 Sonnet Gemini 1.5 Pro

Structured Output Advanced

Extract Nested Structured Data from Images

Extract hierarchically structured information from images (org charts, nested tables, multi-level forms, tree diagrams) as nested JSON while preserving the full hierarchy.

GPT-4o Claude 3.5 Sonnet

Computer Use Intermediate

Understand Screen State and Decide Next Action

The core perceive-decide loop for computer-use agents — given a screenshot and action history, have a VLM determine app state, whether the last action succeeded, and what to do next.

Claude 3.5 Sonnet GPT-4o

Multi-image Reasoning Advanced

Analyze Sequential Screenshots for Action Replay

Have your agent receive a series of session screenshots and reconstruct the user's step-by-step actions as a structured action log.

GPT-4o Gemini 1.5 Pro

Multi-image Reasoning Intermediate

Visual Regression Testing with VLMs

Use a VLM in your CI/CD pipeline to review before/after screenshots of frontend deploys and automatically flag layout breaks, missing elements, and style regressions.

GPT-4o Claude 3.5 Sonnet

Chart & Table Beginner

Convert Image Tables to Structured JSON

Have your agent extract tables from screenshots, scanned documents, or presentation slides and return them as a list of dicts or CSV.

GPT-4o Claude 3.5 Sonnet Gemini 1.5 Pro

UI Grounding Intermediate

Locate and Click Web UI Elements

Have your agent screenshot a page, ask a VLM for the normalized coordinates of a target button or input, then click at those pixel coordinates with pyautogui or Playwright — reliable even when CSS selectors break.

GPT-4o Claude 3.5 Sonnet

Document Understanding Beginner

Extract Structured Data from Invoice Images

Have your agent automatically extract vendor, amount, date, and other fields from scanned invoices and output JSON. Includes real production gotchas.

GPT-4o Claude 3.5 Sonnet Gemini 1.5 Pro