Apple just unveiled new Ferret-UI LLM — this AI can read your iPhone screen

Adobe Firefly generated image of a ferret
(Image credit: Adobe Firefly/Future AI image)

Apple researchers have created an AI model that can understand what’s happening on your phone screen. It is the latest in a growing line-up of models.

Called Ferret-UI, this multimodal large language model (MLLM) can perform a wide variety of tasks based on what it can see on your phone’s screen. Apple's new model can for instance identify icon types, find specific pieces of text, and give you precise instructions for what you should do to accomplish a specific task.

These capabilities were documented in a recently published paper that detailed how this specialized MLLM was designed to understand and interact with mobile user interface (UI) screens. 

What we don't know yet is whether this will form part of the rumored Siri 2.0, or is just another Apple AI research project that goes no further than a published paper.

How Ferret-UI works

Apple Ferret

(Image credit: Apple)

We currently use our phones to accomplish a variety of tasks — we might want to look up information or make a reservation. To do this we look at our phones and tap on any buttons that lead us to our goals. 

Apple believes that if this process can be automated, our interactions with our phones will become even easier. It also anticipates that models such as Ferret-UI can help with things like accessibility, testing apps, and testing usability.

For such a model to be useful, Apple had to ensure that it could understand everything happening on a phone screen while also being able to focus on specific UI elements. Overall, it also needed to be able to match instructions given to it in normal language with what it’s seeing on the screen.

For example, Ferret-UI was shown a picture of AirPods in the Apple store and was asked how one would purchase them. Ferret-UI replied correctly that one should tap on the ‘Buy’ button.

Why Ferret-UI is important?

Apple Ferret

(Image credit: Apple)

With most of us having a smartphone in our pocket, it makes sense that companies are looking into how they can add AI capabilities tailored to these smaller devices. 

Research scientists at Meta Reality Labs already anticipate that we’ll be spending more than an hour every day either in direct conversations with chatbots or having LLM processes run in the background powering features such as recommendations.

Meta's chief AI scientist Yann Le Cun goes as far as to say AI assistants will mediate our entire digital diet in the future. 

So while Apple didn’t spell out what exactly its plans for Ferret-UI are, it’s not too hard to imagine how such a model can be used to supercharge Siri to make the iPhone experience a breeze, possibly even before the year is over.

More from Tom's Guide

Category
Arrow
Arrow
Back to Mobile Cell Phones
Storage Size
Arrow
Colour
Arrow
Condition
Arrow
Price
Arrow
Any Price
Showing 10 of 262 deals
Filters
Arrow
(256GB Blue)
Our Review
1
Apple iPhone 15 Pro Max Blue...
Mint Mobile
Our Review
2
Apple iPhone 15 128BG
Visible
(256GB Black)
Our Review
3
Apple iPhone 15 Pro Max /...
Amazon
Our Review
4
Apple iPhone 15 128GB
AT&T Mobility
(128GB)
Our Review
5
Apple iPhone 15 128GB Blue
Apple
Our Review
6
Apple iPhone 15 128GB
Verizon Wireless
(1TB Blue)
Our Review
7
Apple iPhone 15 Pro 1TB...
Straight Talk
(White)
Our Review
8
Total by Verizon - Apple...
Total by Verizon
(128GB Black)
Our Review
9
Apple iPhone 15 Plus Black...
Mint Mobile
Our Review
10
Apple iPhone 15 Pro 128GB
Visible
Load more deals
Christoph Schwaiger

Christoph Schwaiger is a journalist who mainly covers technology, science, and current affairs. His stories have appeared in Tom's Guide, New Scientist, Live Science, and other established publications. Always up for joining a good discussion, Christoph enjoys speaking at events or to other journalists and has appeared on LBC and Times Radio among other outlets. He believes in giving back to the community and has served on different consultative councils. He was also a National President for Junior Chamber International (JCI), a global organization founded in the USA. You can follow him on Twitter @cschwaigermt.