Google Cloud's Gemini for Multimodal AI Agents

A support ticket that includes a screenshot, a scanned invoice PDF, and three lines of typed complaint used to mean three separate pipelines: OCR for the PDF, a vision model for the screenshot, and an LLM to tie it all together in prose no one fully trusted. Gemini on Vertex AI collapses that into one model call that reasons over all three inputs jointly, and once you've built an agent that way, going back to stitched-together pipelines feels like a regression.

Multimodal Input in a Single Request

Gemini accepts interleaved text, image, and document parts in one contents array — there's no separate "vision endpoint."

from vertexai.generative_models import GenerativeModel, Part
import vertexai

vertexai.init(project="acme-support-prod", location="us-central1")
model = GenerativeModel("gemini-2.5-pro")

response = model.generate_content(
    [
        "A customer submitted this invoice and a screenshot of the error they saw. "
        "Identify the invoice number, the charge amount, and whether the screenshot "
        "shows a payment failure or a rendering bug.",
        Part.from_data(data=invoice_pdf_bytes, mime_type="application/pdf"),
        Part.from_data(data=screenshot_bytes, mime_type="image/png"),
    ]
)

print(response.text)

The model natively handles multi-page PDFs — including scanned, non-text-layer documents — by treating each page as visual input, so there's no separate OCR pass to maintain or debug when it silently mangles a skewed scan.

Function Calling for Grounded Actions

Reasoning over the inputs is only half the agent; it needs to act. Gemini's function calling declares tools the same way, whether the triggering input was text or an image.

from vertexai.generative_models import FunctionDeclaration, Tool

lookup_invoice = FunctionDeclaration(
    name="lookup_invoice",
    description="Fetch invoice details from the billing system by invoice number",
    parameters={
        "type": "object",
        "properties": {"invoice_number": {"type": "string"}},
        "required": ["invoice_number"],
    },
)

billing_tool = Tool(function_declarations=[lookup_invoice])
model = GenerativeModel("gemini-2.5-pro", tools=[billing_tool])

chat = model.start_chat()
response = chat.send_message([image_part, "What's the status of this invoice?"])

call = response.candidates[0].content.parts[0].function_call
if call.name == "lookup_invoice":
    result = billing_api.get_invoice(call.args["invoice_number"])
    response = chat.send_message(
        Part.from_function_response(name="lookup_invoice", response={"content": result})
    )

The pattern is identical to text-only function calling — the model extracted the invoice number from the image, called the tool, and incorporated the real billing record into its final answer, without any custom image-to-text glue code in between.

Grounding with Search and Vertex AI Search

Left alone, a model will confidently invent a return policy or a product spec it never saw. Grounding tools constrain generation to retrieved evidence:

from vertexai.generative_models import Tool, grounding

search_tool = Tool.from_google_search_retrieval(
    grounding.GoogleSearchRetrieval()
)

response = model.generate_content(
    "What's Acme's current return window for opened electronics?",
    tools=[search_tool],
)

for chunk in response.candidates[0].grounding_metadata.grounding_chunks:
    print(chunk.web.uri)

For enterprise knowledge that isn't public, VertexAISearch grounds against your own indexed corpus (support docs, product catalogs, internal wikis) instead of the open web:

enterprise_tool = Tool.from_retrieval(
    grounding.Retrieval(
        source=grounding.VertexAISearch(
            datastore="projects/acme-support-prod/locations/global/collections/default_collection/dataStores/kb-2026"
        )
    )
)

Both approaches attach citation metadata to the response, which matters as much for debugging as for user trust — when the agent gets something wrong, you can see exactly which document it grounded on and fix the source rather than the prompt.

A Realistic Use Case: Scanned Document Triage

Put together, a document-triage agent for an insurance intake queue looks like this end to end:

1. Ingest: scanned claim form (PDF) + photo of damage + adjuster's typed note
2. Gemini call: extract structured fields (claim type, policy number, damage severity estimate)
3. Function call: cross-reference policy number against policy_lookup API
4. Grounding: check claim type against current coverage terms via Vertex AI Search
5. Output: structured JSON decision + human-readable summary + citations

interface TriageResult {
  policyNumber: string;
  claimType: 'auto' | 'property' | 'liability';
  estimatedSeverity: 'low' | 'medium' | 'high';
  coverageConfirmed: boolean;
  citedSources: string[];
  summary: string;
}

Running this against a real backlog of a few thousand mixed-quality scans, the biggest accuracy gains didn't come from prompt tweaking — they came from lowering the PDF page-image resolution just enough to fit more pages in context without Gemini losing legibility on small print, and from explicitly asking the model to state "insufficient information" rather than guess when a scan was too degraded to read.

Wrapping Up

The real shift with Gemini isn't that it can "see" — vision-language models have existed for years — it's that multimodal input, function calling, and grounding all live in the same conversational loop, so an agent can look at a photo, decide it needs a database lookup, and cite a policy document in one coherent turn instead of three brittle services glued together after the fact. That's the difference between a demo that works on curated examples and an agent that survives a real intake queue full of blurry scans and inconsistent phone photos.