Like Anthropic’s Pc Use and Google DeepMind’s Mariner, Operator takes screenshots of a pc display screen and scans the pixels to determine what actions it might probably take. CUA, the mannequin behind it, is educated to work together with the identical graphical consumer interfaces—buttons, textual content bins, menus—that individuals use once they do issues on-line. It scans the display screen, takes an motion, scans the display screen once more, takes one other motion, and so forth. That lets the mannequin perform duties on most web sites that an individual can use.
“Historically the best way fashions have used software program is thru specialised APIs,” says Reiichiro Nakano, a scientist at OpenAI. (An API, or utility programming interface, is a chunk of code that acts as a form of connector, permitting completely different bits of software program to be hooked as much as each other.) That places loads of apps and most web sites off limits, he says: “However in case you create a mannequin that may use the identical interface that people use each day, it opens up a complete new vary of software program that was beforehand inaccessible.”
CUA additionally breaks duties down into smaller steps and tries to work by means of them one after the other, backtracking when it will get caught. OpenAI says CUA was educated with methods just like these used for its so-called reasoning fashions, o1 and o3.
OPENAI
OpenAI has examined CUA in opposition to quite a lot of trade benchmarks designed to evaluate the flexibility of an agent to hold out duties on a pc. The corporate claims that its mannequin beats Pc Use and Mariner in all of them.
For instance, on OSWorld, which checks how properly an agent performs duties similar to merging PDF information or manipulating a picture, CUA scores 38.1% to Pc Use’s 22.0% Compared, people rating 72.4%. On a benchmark known as WebVoyager, which checks how properly an agent performs duties in a browser, CUA scores 87%, Mariner 83.5%, and Pc Use 56%. (Mariner can solely perform duties in a browser and due to this fact doesn’t rating on OSWorld.)
For now, Operator can even solely perform duties in a browser. OpenAI plans to make CUA’s wider talents accessible sooner or later through an API that different builders can use to construct their very own apps. That is how Anthropic launched Pc Use in December.
OpenAI says it has examined CUA’s security, utilizing pink groups to discover what occurs when customers ask it to do unacceptable duties (similar to analysis how one can make a bioweapon), when web sites comprise hidden directions designed to derail it, and when the mannequin itself breaks down. “We’ve educated the mannequin to cease and ask the consumer for data earlier than doing something with exterior unintended effects,” says Casey Chu, one other researcher on the crew.
Look! No fingers
To make use of Operator, you merely kind directions right into a textual content field. However as an alternative of calling up the browser in your laptop, Operator sends your directions to a distant browser operating on an OpenAI server. OpenAI claims that this makes the system extra environment friendly. It’s one other key distinction between Operator, Pc Use and Mariner (which runs inside Google’s Chrome browser by yourself laptop).
Like Anthropic’s Pc Use and Google DeepMind’s Mariner, Operator takes screenshots of a pc display screen and scans the pixels to determine what actions it might probably take. CUA, the mannequin behind it, is educated to work together with the identical graphical consumer interfaces—buttons, textual content bins, menus—that individuals use once they do issues on-line. It scans the display screen, takes an motion, scans the display screen once more, takes one other motion, and so forth. That lets the mannequin perform duties on most web sites that an individual can use.
“Historically the best way fashions have used software program is thru specialised APIs,” says Reiichiro Nakano, a scientist at OpenAI. (An API, or utility programming interface, is a chunk of code that acts as a form of connector, permitting completely different bits of software program to be hooked as much as each other.) That places loads of apps and most web sites off limits, he says: “However in case you create a mannequin that may use the identical interface that people use each day, it opens up a complete new vary of software program that was beforehand inaccessible.”
CUA additionally breaks duties down into smaller steps and tries to work by means of them one after the other, backtracking when it will get caught. OpenAI says CUA was educated with methods just like these used for its so-called reasoning fashions, o1 and o3.
OPENAI
OpenAI has examined CUA in opposition to quite a lot of trade benchmarks designed to evaluate the flexibility of an agent to hold out duties on a pc. The corporate claims that its mannequin beats Pc Use and Mariner in all of them.
For instance, on OSWorld, which checks how properly an agent performs duties similar to merging PDF information or manipulating a picture, CUA scores 38.1% to Pc Use’s 22.0% Compared, people rating 72.4%. On a benchmark known as WebVoyager, which checks how properly an agent performs duties in a browser, CUA scores 87%, Mariner 83.5%, and Pc Use 56%. (Mariner can solely perform duties in a browser and due to this fact doesn’t rating on OSWorld.)
For now, Operator can even solely perform duties in a browser. OpenAI plans to make CUA’s wider talents accessible sooner or later through an API that different builders can use to construct their very own apps. That is how Anthropic launched Pc Use in December.
OpenAI says it has examined CUA’s security, utilizing pink groups to discover what occurs when customers ask it to do unacceptable duties (similar to analysis how one can make a bioweapon), when web sites comprise hidden directions designed to derail it, and when the mannequin itself breaks down. “We’ve educated the mannequin to cease and ask the consumer for data earlier than doing something with exterior unintended effects,” says Casey Chu, one other researcher on the crew.
Look! No fingers
To make use of Operator, you merely kind directions right into a textual content field. However as an alternative of calling up the browser in your laptop, Operator sends your directions to a distant browser operating on an OpenAI server. OpenAI claims that this makes the system extra environment friendly. It’s one other key distinction between Operator, Pc Use and Mariner (which runs inside Google’s Chrome browser by yourself laptop).
