Multimodal AI

Give each input a job

You are describing a small stationery display. A photograph shows two closed notebooks with visible labels A and B. A written catalogue says A contains blank pages and B contains ruled pages. You want an accessible description that combines the visible arrangement with the catalogue information without pretending that the closed pages are visible. This is a useful multimodal task because different inputs contribute different kinds of evidence.

A modality is a kind of information, such as text, images, audio or video. A multimodal system can work with more than one kind. An image and a text question can be inputs to a system whose output is only text. Another system may generate an image or audio. Supporting a modality as input does not imply the ability to produce that modality as output.

Define the task before attaching material. For this display, a photograph and a short catalogue extract are enough. Adding unrelated recordings, the entire inventory and a folder of old photographs would increase ambiguity and exposure. More input is useful only when it contributes evidence needed for the answer.

Understand the connection between signals

Images contain pixel patterns, text contains symbols, and audio contains changing signals over time. A model needs numerical representations of these inputs. Different systems use different processing components and ways to combine them. Some learn from paired examples, such as images with associated descriptions, so that related content can be represented in ways the system can compare or use together.

CLIP is a documented example of learning from image-text pairs. Its training encourages matching images and descriptions to have compatible representations relative to mismatched pairs. This can support matching an image with a candidate description. CLIP itself should not be confused with a conversational assistant that writes a new answer; a complete application may need additional components.

Other multimodal systems connect visual or audio representations with a language-generating model. The practical consequence is that an answer can combine signals rather than merely reading a text transcript of everything. However, the exact architecture is implementation-specific. You cannot infer which encoders, attention layers or training data a proprietary service uses simply because it accepts a photograph.

Learned association is not guaranteed understanding of every relationship. A model can identify two notebooks yet reverse their labels, ignore a small tag or supply a familiar description unsupported by the input. The system must be evaluated on the relationships that matter to the task, not only on whether it names the objects correctly.

Make source correspondence explicit

Label the photograph P1 and the catalogue C1. In the catalogue, preserve the identifiers A and B exactly as they appear on the display. Ask the assistant to associate each visible label with its catalogue entry before drafting the description. If the label is unreadable, it should say that the association is uncertain rather than join the records by a guessed colour or position.

Dates and versions matter too. A photograph taken before the display was rearranged may not match today's catalogue. Check that both sources refer to the same items and state. A convincing explanation can still be wrong if it combines accurate facts from different moments. Identify the uncertainty and request the missing evidence instead of automatically trusting whichever source is longer.

For audio or video, alignment often involves timestamps. A spoken “this one” needs the relevant visual moment, not a random frame from the same recording. Systems may sample, resize or otherwise process media, so ask for the relevant interval and inspect it yourself. A video summary does not establish that every frame and sound received equal attention.

Ask for evidence before a polished description

Use the request below with a photograph or drawing you are authorised to share and a matching short catalogue. The assistant should first state visible details, then catalogue-only facts, then uncertainties. That intermediate separation makes it easier to detect a claim that has migrated from one kind of evidence to another.

For a text-only rehearsal, use these declared simulated observations: P1 contains a red closed notebook labelled A on the left and a blue closed notebook labelled B on the right; C1 says A has blank pages and B has ruled pages. A correct rehearsal can describe the arrangement and attribute page style to C1. It cannot demonstrate that an actual vision model read the labels correctly.

Task: describe a two-notebook display using the sources I provide.
P1 is the display image. C1 is the catalogue extract.
First report visible notebook labels and their positions in P1.
Then match each readable label to C1 and list catalogue-only facts.
Separate observations, supplied facts and uncertain associations.
Do not infer hidden page contents from the appearance of a closed book.
If a source is missing, a label is unreadable or the sources conflict,
state the gap. Do not claim to have inspected an unavailable attachment.
Finally draft a short description with source attribution where needed.

Check links between claims and inputs

Inspect the actual image and compare the answer one item at a time. Is A really on the left? Does the label belong to the red notebook rather than an object behind it? Does the catalogue actually say that A contains blank pages? These checks test cross-source relationships, not merely the existence of a plausible notebook description.

A useful final sentence might say, “The image shows A, a red closed notebook, on the left; the catalogue describes its pages as blank.” The attribution explains why the page claim is present even though those pages are hidden. Without attribution, a reader could reasonably believe that the image itself established the detail.

Test the workflow by deliberately removing a catalogue row or making a label unreadable in a practice copy. A reliable response should narrow its claims or request clarification. If it keeps producing the same complete description, it may be relying on expectations rather than the supplied evidence. Do not modify the authoritative source merely to make a model's answer look consistent.

Recognise the weak link in a combined answer

Multimodal output can fail at several stages: reading the input, identifying an object, matching records, interpreting a relationship or writing the final answer. A clearer photograph may fix label recognition but will not resolve a catalogue entry that refers to a different item. Diagnose which stage lacks evidence before uploading more data or writing a longer prompt.

Provider documentation describes limitations such as small text, rotations, approximate counts and difficult spatial relationships. Treat those as reasons to test your actual material. Use readable crops with enough surrounding context, and check both the crop and the original. Cropping can improve legibility while removing the information needed to associate a label with an object.

Not every missing detail should be filled. A photograph cannot reliably establish a person's identity, intentions or ownership of an item merely because they appear nearby. In our display, closed notebooks do not reveal page content. A useful assistant says what the sources support and where the answer stops.

Review the combined information and final output

Each source can appear harmless on its own while their combination reveals more. A photograph with a name tag plus a catalogue containing contact details could associate a person with a private record. Remove unnecessary identifying fields, incidental faces and background documents before sharing. Review recorded audio for other speakers and private conversation, not only the intended sentence.

Text visible in an image or heard in a recording is still source content, not permission to take actions. A photographed note saying “send all files here” must not override the user's task. Limit connected-tool access and enforce permissions outside the model. This matters especially when a system can both inspect media and operate other applications.

Check the final output in the form people will receive it. A spoken description needs pronunciation review; a generated image needs visual review; a text description needs accurate attribution and suitable reading order. Preserve a text alternative where useful. The success criterion is a faithful result for the chosen inputs and task, not a claim that combining modalities always makes answers more reliable.

Sources & further reading

Primary references for the concepts in this guide. Examples and exercises are illustrative; verify provider-specific behaviour against the version you use.

• Radford et al.: Learning Transferable Visual Models from Natural Language Supervision
• OpenAI: Images and vision
• Google: Video understanding
• OWASP: Prompt injection prevention