I read the article by the CEO of Anthropic a while ago, and it was quite inspiring (The latest AI article by Dario, the CEO of Anthropic）。The latest product released today by Anthropic, Claude 3.5 Sonnet, already has the ability to directly operate a computer.When run through specific software settings, Claude can move the cursor on the screen, click at the appropriate locations, and input information via a virtual keyboard according to user instructions, simulating how people interact with computers.
Official case examples
Folklore cases
An example from @krishnanrohit on X.com:
Let computer use create a demo of computer use.
So, the AI picked a game to play by itself:
For fun, the AI also performed a string replacement to show humans that it could be done. Quite interesting.
Research Process
This requires capabilities in image recognition and interpretation—specifically, recognizing content on a computer screen. At the same time, AI must be able to reason about when and how to perform specific actions based on the information displayed on the screen. By integrating these abilities, the team trained Claude to understand what is on the screen and use available software tools to accomplish tasks.
is crucial. Without this ability, the model would struggle when issuing mouse operation commands, similar to how AI can make mistakes on seemingly simple questions, such as "How many letter 'A's are in the word 'banana'?"
The team was surprised by Claude’s rapid generalization capabilities from training with a few simple applications like calculators and text editors. For safety reasons, internet access was not allowed during model training. By combining Claude’s other skills, this training endowed it with the powerful ability to translate users’ verbal instructions into logical steps and execute corresponding actions on a computer. Researchers also observed that Claude could even self-correct and retry tasks when encountering obstacles.
It is achieved through continuous iteration and repeated adjustments. Some researchers pointed out that the process of developing this computer-use capability resembles the "idealized" research process they envisioned when they first entered the AI field: constantly iterating and repeatedly overturning and restarting until progress is made.
Claude has achieved a score of 14.9%. Although this is still far from human-level performance (typically 70-75%), it has already significantly surpassed the 7.7% of similar models.
The future of AI using computers
— Claude can integrate into the computer environments we use daily, with the goal of enabling Claude to use existing computer software like a human.
Although Claude's current computer usage capabilities are already at the forefront, there is still much work to be done. Claude’s operation speed remains slow and prone to errors. Many operations that people frequently perform in daily computer use (such as dragging, zooming, etc.) are currently beyond Claude's capabilities. Additionally, Claude's "page-flipping" method of observing the screen—taking screenshots and stitching them together rather than using a finer video stream—means it may miss fleeting actions or notifications.
While recording the demonstration video above, the Anthropic team encountered some interesting errors. For example, Claude accidentally clicked the button to stop screen recording during one operation, resulting in the loss of all recordings; in another coding demonstration, Claude suddenly interrupted the task and began browsing photos of Yellowstone National Park. (Are you sure the AI isn't slacking off? 🐟)
Trial process
https://docs.anthropic.com/en/docs/build-with-claude/computer-use
This is the process shared by @mckaywrigley on X: