Chapter 13 · Multimodal Agents

Learning objectives: Learn to pass multimodal inputs (Image / Audio / Video / File) to Agents, understand the implementation paths for image and audio output, and complete a practical project: "Invoice Image → Structured Data".

13.1 The Multimodal Content Object System

Agno uses a unified content object abstraction for four modalities, all imported from agno.media. The structure is as follows:

Class	Supported sources	Common metadata
Image	url, filepath, content	format, mime_type, detail
Audio	url, filepath, content	format, mime_type, sample_rate, channels
Video	url, filepath, content	format, mime_type, duration
File	url, filepath, content, external	filename, format, mime_type

The same class supports both URLs and local files or raw bytes, enabling seamless switching between "online image links" and "local file processing." Pass them via keyword arguments to run() / print_response(): images=[...], audios=[...], videos=[...], files=[...].

Model capability is everything

The framework is just a carrier — what determines whether the model can see images or hear audio is the model itself. When integrating vision models, confirm that your third-party provider's model supports the modality (e.g., DeepSeek's chat series currently does not accept image input; choose a vision-capable model from a third-party provider for visual tasks). Video input is currently supported only by Gemini and select AWS Bedrock models.

13.2 Image Understanding: URLs and Local Files

# vision_agent.py —— Image understanding (requires a vision-capable model)
import os
from agno.agent import Agent
from agno.models.openai import OpenAIChat   # or OpenAILike pointing to any vision endpoint
from agno.media import Image

agent = Agent(
    model=OpenAIChat(
        id="qwen-vl-plus",                  # Example: third-party OpenAI-compatible vision model
        api_key=os.getenv("VISION_API_KEY"),
        base_url="https://dashscope.aliyuncs.com/compatible-mode/v1",
    ),
    instructions=["Be specific: describe the main subject first, then details"],
    markdown=True,
)

# Method 1: Direct URL
agent.print_response(
    "What is in this image? Summarize in one sentence.",
    images=[Image(url="https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/640px-Cat03.jpg")],
)

# Method 2: Local file
agent.print_response(
    "Describe this image",
    images=[Image(filepath="./photo.jpg")],
)

# Method 3: Multi-image comparison
agent.print_response(
    "Compare the style differences between these two images",
    images=[Image(filepath="./a.jpg"), Image(filepath="./b.jpg")],
)

Three details worth noting:

• Multiple images = add more items to the list; order is the order the model sees them;
• Image(detail=...) controls the visual detail level (affects token consumption);
• For byte-stream scenarios (e.g., freshly downloaded, not yet written to disk), use Image(content=raw_bytes).

13.3 Document Understanding and Audio Transcription

PDFs and other documents are passed via File objects; audio via Audio objects:

# doc_audio.py —— PDF summary and speech transcription
import os
from agno.agent import Agent
from agno.models.openai import OpenAIChat
from agno.media import Audio, File

pdf_agent = Agent(
    model=OpenAIChat(
        id="deepseek-chat",                 # A text model can handle parsed PDF text
        api_key=os.getenv("DEEPSEEK_API_KEY"),
        base_url="https://api.deepseek.com/v1",
    ),
)
pdf_agent.print_response(
    "Summarize the three key conclusions of this report",
    files=[File(filepath="./annual_report.pdf")],
)


audio_agent = Agent(
    # Audio input requires selecting a model endpoint that supports audio
    model=OpenAIChat(
        id="qwen-audio-asr",
        api_key=os.getenv("AUDIO_API_KEY"),
        base_url="https://dashscope.aliyuncs.com/compatible-mode/v1",
    ),
)
with open("./meeting.wav", "rb") as f:
    audio_bytes = f.read()

audio_agent.print_response(
    "Transcribe this recording and list the action items",
    audios=[Audio(content=audio_bytes, format="wav")],
)

13.4 Image and Audio Output

Beyond "reading," Agno also supports "writing." Image generation goes through the tool path — equip the Agent with OpenAITools and specify an image model; audio output is returned directly by the model:

# gen_image.py —— Let the Agent generate an image and save it locally
import base64
import os
from agno.agent import Agent
from agno.models.openai import OpenAIChat
from agno.tools.openai import OpenAITools          # Built-in OpenAI capabilities toolkit
from agno.utils.media import save_base64_data      # Official utility for saving base64 data

agent = Agent(
    model=OpenAIChat(
        id="gpt-image-hosting-model",               # Example: the chat model hosting image generation
        api_key=os.getenv("IMAGE_API_KEY"),
        base_url="https://your-openai-compatible-endpoint/v1",
    ),
    tools=[OpenAITools(image_model="gpt-image-2")], # Specify the actual image-generation model
)

response = agent.run("Draw an illustration of a sunset over snowy mountains")

if response.images and response.images[0].content:
    image_base64 = base64.b64encode(response.images[0].content).decode("utf-8")
    save_base64_data(image_base64, "sunset.png")     # Write to disk
    print("Saved sunset.png")

# gen_audio.py —— Audio response
import os
from agno.agent import Agent
from agno.models.openai import OpenAIChat
from agno.utils.audio import write_audio_to_file

agent = Agent(
    model=OpenAIChat(
        id="gpt-audio",
        api_key=os.getenv("AUDIO_API_KEY"),
        base_url="https://your-openai-compatible-endpoint/v1",
        modalities=["text", "audio"],                # Declare output modality
        audio={"voice": "alloy", "format": "wav"},   # Voice and format
    ),
)

response = agent.run("Explain compound interest using a fable")
if response.response_audio:
    write_audio_to_file(audio=response.response_audio.content, filename="story.wav")

Where do generated media go?

Image generation results are in the response.images list (bytes in .content); audio results are in response.response_audio. By default they exist only in memory — you need to encode and write them to disk yourself. The official library provides two utility functions: save_base64_data and write_audio_to_file.

13.5 Capstone: Invoice Image Structured Extraction

Combine this chapter's content with the tools from the previous chapter and the structured output from Chapter 5 — this is one of the most frequent multimodal scenarios in enterprise settings:

# invoice_ocr.py —— Invoice image -> Structured JSON
import os
from pydantic import BaseModel, Field
from agno.agent import Agent
from agno.models.openai import OpenAIChat
from agno.media import Image


class Invoice(BaseModel):
    vendor: str = Field(..., description="Vendor name")
    amount: float = Field(..., description="Total amount including tax")
    currency: str = Field(default="CNY", description="Currency code")
    invoice_no: str = Field(..., description="Invoice number")


agent = Agent(
    model=OpenAIChat(
        id="qwen-vl-plus",                    # Vision model handles "seeing"
        api_key=os.getenv("VISION_API_KEY"),
        base_url="https://dashscope.aliyuncs.com/compatible-mode/v1",
    ),
    output_schema=Invoice,                    # Pydantic handles "shaping"
    instructions=[
        "Extract fields from the invoice image; retain two decimal places for amounts",
        "Leave unclear fields as empty strings and note them in remarks",
    ],
)

result: Invoice = agent.run(
    "Extract information from this invoice",
    images=[Image(filepath="./invoice_scan.png")],
).content

print(result.vendor, result.amount, result.invoice_no)

This pattern is highly extensible: swap output_schema and it adapts to contracts, ID cards, financial reports — any "image in, structure out" requirement.

Chapter Summary

• Four modalities map to four content objects in agno.media, each supporting url / filepath / content as sources;
• Inputs are attached to run()'s images/audios/videos/files parameters; availability depends on your chosen model's capabilities — video input is currently supported only by Gemini and Bedrock;
• Output direction: images are generated via OpenAITools (results in response.images), audio is emitted directly by the model (response.response_audio);
• Multimodal + output_schema combination is the standard solution for OCR / receipt extraction scenarios.

🧪 随堂测验

点击你认为正确的选项。答错时会展示正确答案与原因解析。

1. What is the correct way to pass a local image to an Agent?

2. An Agent is configured with Image input but the model errors that it does not support images. What is the problem?

3. After calling a generative-image Agent, where are the generated image bytes located?

4. For "scanned PDF -> extract fields to JSON", which combination is most appropriate?

🛠️ Hands-On Exercises

1. Find photos from two different scenarios and have the Agent "describe a single image" and "compare two images" respectively. Observe how the answer structure differs when the prompt is the same.
2. Adapt the invoice extraction from section 13.5 to contract key information extraction (parties, amount, term) and design your own Pydantic model.
3. Add a custom save_last_image() tool to the generative-image Agent from 13.4, so users can save the last generated image with a single command.

After completing the hands-on exercises, proceed to Chapter 14: Session Management — session state and chat history.