Computer use (an AI feature that operates the screen)
Japanese コンピューター・ユース
Computer use is a feature where an AI looks at screenshots and operates apps or websites by clicking the mouse and typing on the keyboard. It saves work, but it can click the wrong thing, so a person should watch.
Computer use is a way for an AI to look at a picture of the screen (a screenshot), decide on an action such as "click here and type this", and move the mouse and keyboard for you. After each action it takes a new screenshot and decides the next step. Anthropic released it as a public beta for developers in October 2024, and Google's Gemini API and the ChatGPT app have the same kind of feature. It is like a coworker sitting next to you, watching your screen and clicking for you.
Watch out Both Anthropic and Google recommend running this feature on a separate virtual computer, and having a person confirm actions that are hard to undo, such as payments, agreeing to terms, or sending messages. One reason is that the AI may follow instructions hidden inside a web page. It is safer not to let it work on a screen where login details or customer data are open.
Source: Anthropic, “Computer use, Claude 3.5 Sonnet, and Claude 3.5 Haiku” · Anthropic, “Computer use tool” · Google, “Computer use” (Gemini API)