Gemini’s Focus on Desktop Web

The current focus of Gemini’s “Computer Use” initiatives is on desktop web functionality, as showcased by the Gemini Agent available for AI Ultra subscribers. With the impending release of Android 16 QPR3 Beta 2, it appears that mobile integration is on the horizon, particularly through its new feature: “Screen Automation.”

Understanding ‘Screen Automation’

In the latest Android 16 QPR3 Beta 2, a new permission known as “Screen Automation” can be found under Settings > Apps > Special app access. This feature allows apps to assist users by interacting with the screen content of other applications. Presently, it is available exclusively on Pixel 10 devices, leaving its expansion to other devices uncertain.

Supported Applications and Permissions

The Google app, which serves as the backbone for Gemini, appears to be the only application supporting this feature at this stage. Users can choose from three distinct permissions: Always allow, Ask every time (default), and Don’t allow.

This app will be able to see and interact with other apps’ screen content to help you complete tasks, even when the apps are in the background.

Future Capabilities of Gemini

Strings and descriptions within the app hint at a promising future for this feature, referred to as “computer_control.” Google has ambitious plans for Gemini, with the goal of mimicking human-like interactions through clicking, typing, and scrolling across various applications and websites.

Previous Demonstrations of Project Astra

During past showcases, particularly in May, Project Astra demonstrated its capacity to scroll through Chrome for Android and interact with the YouTube app seamlessly. This points toward a tangible direction for Gemini, setting the stage for more sophisticated AI-driven capabilities in mobile environments.

The Outlook for Android 16 and Gemini

As Google lays the groundwork for these capabilities in Android 16 QPR3, anticipation grows around when these Gemini features will officially launch. Users may soon experience a new level of integration and automation across their mobile devices, altering how they interact with apps on their phones.