Perception & Operation Bridge

Pob

Gives AI a pair of eyes and a pair of hands on your desktop.

Pob is an overlay. It sits on top of whatever application you point it at, perceives that application through the window that contains it, and operates it — pointer, keys, and everything in between. Applications with no API get one.

Download PobSource on GitHub
Windows·macOS·Linux
Pob running as an overlay above a desktop application

What it can do

Sees
Screen capture of the window underneath, on demand or as the macro runs.
Points
Absolute mouse movement, clicks, drags and scrolls.
Types
Text entry and named key presses, including modifiers and function keys.
Remembers
Every session is written to its own folder, so you can read back what happened.
Repeats
Record a run once and replay it as a macro, as often as you like.
Travels
Windows, macOS and Linux — including a Windows VM driven from a Mac.

Macro PSL — step through one

A macro is read line by line. Where a line needs a judgement rather than a number, it holds an instruction in :: delimiters, and that instruction is answered from what is on screen before the line runs. So a step can say the message box instead of guessing at a coordinate that moves the next time the window resizes.

macro.psl
▸// Reply to every unread message, then sign out.
 move(398, 915)
 click()
 if (:: a chat window is open ::) {
   loop (:: another unread message in the list ::, 10) {
     move(:: the x offset to the message box ::, 738)
     click()
     typeText(:: a short reply to the message on screen ::)
     keyPress("return")
   }
 }
 call("../sign-out.psl")
pobReady.

The instruction language is PSL, written by the same hand and usable on its own.

Four ways to drive it

From an AI client

Pob speaks MCP, so Claude Code, Claude Desktop and Gemini CLI can see the screen and work the machine themselves. Ask for the thing you want done and let the client do it.

From a macro

Record what you do once, then replay it. A recorded macro is an editable text file, so you can go back in and replace a brittle coordinate with an instruction.

From your phone

Every running instance serves a remote control page. Open it on a phone on the same network to watch what Pob sees and drive it from the couch.

From the keyboard client

A desktop keyboard and trackpad that types into the machine Pob is running on — useful when the machine you are driving is a VM or across the room.

Getting started

  1. Download the release for your platform and unpack it. On macOS, drag Pob to Applications; on Windows and Linux, run the installer that comes with it. Either way the pob command lands on your PATH.
  2. Open it over the app you want automated. Pob floats above it and reads that window.
  3. Give it an instruction, record a macro, or connect an AI client to its MCP server and let the client take it from there.