Multimodal and Agentic AI: The Future of Machine Intelligence

AutoScheduler AI App Builder Warehouse App Builder for Logistics Teams

In recent years, AI has been a fairly smart text-based chatbot that can help with writing, summarising and explaining. However, it doesn’t have any ability to see, hear or interact with the real-world. This is changing due to two trends coming together to create a more perceptive and actionable AI. Multimodal and agentic AI is the biggest shift in AI since the creation of large language models.

Giving the Machine Senses

Multimodal AI is the ability to understand and respond to multiple types of information. Examples include text, images, audio, video, code and increasingly sensor data. Previously, this would be broken down into different fields and each would have its own model. This meant the multimodal AI would feel very clunky.

However, current architectures allow for a single model to take in a hand-drawn diagram, answer a verbal question about it and produce an improved chart. This is important as people don’t just store their knowledge as text. A radiologist will look at scans, a mechanic will listen to engines and an architect will think in diagrams. As the models are trained on many different inputs they will have a more complex representation of how the world works. This gives huge real-world benefits. A farmer can take a picture of a sick leaf and receive a diagnosis. A visually impaired user can receive a real-time description of a scene. An engineer can point their camera at a broken machine and ask what is wrong.

This also helps provide a foundation for language. If a model hasn’t seen many sunsets, just read about them then they will have a different relationship with the word. While there is still debate as to if this means they actually understand, there are less silly mistakes and the model can deal with ambiguity.

Giving the Machine Hands

An agentic AI will work towards completing an objective. It will plan how to achieve something, use tools like a browser, code interpreter, database and calendar, assess how it is doing and adapt.

This may seem trivial but it makes a big difference. A chat bot will tell you how to book your flight whereas an agent will book your flight, check it is confirmed and realise that the connection is too tight and re-book. In the world of software engineering agents are already being used to read code, write patches, run tests and iterate until all tests pass. In research, they are searching literature, running analyses and collating results.

Recommended for You:

The reason they can do this is because they are working in a loop. They perceive, reason, act, observe and repeat. This means they can self-correct based on real-world data instead of making up answers. This is what makes the model an agent.

Why the Combination Is Transformative

Both of these features are powerful by themselves but together they are something new. An agentic model without vision will only be able to interact with structured text-based data. A multimodal model without agency will be a very intelligent observer but won’t be able to do anything. However, combining the two will give you a system that can interact with a graphical interface as a human would. The agent can read, click, type and verify.

This has really started to transform the world of enterprise. Tasks that were previously too messy to automate can now be done. This includes matching up invoices in various formats, triaging support tickets with screenshots and monitoring video feeds. We are reaching a point where we can automate ambiguous tasks.

The Costs of Capability

As with anything, there are risks to consider. The bigger the autonomy the bigger the risk. A chatbot making a mistake will just produce incorrect text. An agentic AI may send the wrong email, delete the wrong file or spend money incorrectly. With multi-step tasks mistakes can add up. A small error early on can result in a costly outcome.

A new risk comes with security. As the agentic AI can read webpages and documents it can be manipulated by hidden instructions within the content (prompt injection). Multimodal inputs mean there is an increased attack surface as the malicious instructions can be hidden within the image or audio. There is also a lack of accountability. If an autonomous system causes issues it can be difficult to know who to blame.

It is important to be cautious but not overly concerned. You should always have humans involved at key decision points. Limit the permissions given to the agent. Have clear logs and ensure actions can be reversed.

A Collaborator, Not a Replacement

The best way to think of these systems is as a collaborator. Although they are very quick, tireless and increasingly intelligent, they will still make confident mistakes and won’t have any accountability. Those who can delegate wisely will be the winners.

The days of having an oracle in a box are gone. We now have an AI that can see, hear and do. We need to ensure we have the wisdom to control the new sensory and physical capabilities.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top