Perception & Operation Bridge

Pob

Gives AI a pair of eyes and a pair of hands on your desktop.

Pob is an overlay. It sits on top of whatever application you point it at, perceives that application through the window that contains it, and operates it — pointer, keys, and everything in between. Applications with no API get one.

Download Pob Source on GitHub

Windows·macOS·Linux

What it can do

Sees: Screen capture of the window underneath, on demand or as the macro runs.
Points: Absolute mouse movement, clicks, drags and scrolls.
Types: Text entry and named key presses, including modifiers and function keys.
Remembers: Every session is written to its own folder, so you can read back what happened.
Repeats: Record a run once and replay it as a macro, as often as you like.
Watches: A once block sits on the screen and plays again every time what it sees changes into something its condition holds of.
Reaches out: run("…") hands a line to the shell, and launch("Firefox") opens an application and fits its window into the frame — so a macro recorded on your desktop lands correctly on a machine nobody has arranged first.
Travels: Windows, macOS and Linux — including a Windows VM driven from a Mac, and Linux microVMs it launches itself.

Macro PSL — step through one

A macro is read line by line. Where a line needs a judgement rather than a number, it holds an instruction in :: delimiters, and that instruction is answered from what is on screen before the line runs. So a step can say the message box instead of guessing at a coordinate that moves the next time the window resizes.

src/main.macro.psl

▸// Reply to every unread message, then keep replying as they arrive.
 move(398, 915)
 click()
 if (:: a chat window is open ::) {
   loop (:: another unread message in the list ::, 10) {
     move(:: the x offset to the message box ::, 738)
     click()
     typeText(:: a short reply to the message on screen ::)
     keyPress("return")
   }
 }
 once (:: a new message has arrived ::) {
   move(:: the x offset to the message box ::, 738)
   click()
   typeText(:: a short reply to it ::)
   keyPress("return")
 }

pobReady.

The instruction language is PSL, written by the same hand and usable on its own.

Five ways to drive it

From an AI client

Pob speaks MCP, so Claude Code, Claude Desktop and Gemini CLI can see the screen and work the machine themselves. Ask for the thing you want done and let the client do it.

From a macro

Record what you do once, then replay it. A recorded macro is an editable text file, so you can go back in and replace a brittle coordinate with an instruction.

From your phone

Every running instance serves a remote control page. Open it on a phone on the same network to watch what Pob sees and drive it from the couch.

From the keyboard client

A desktop keyboard and trackpad that types into the machine Pob is running on — useful when the machine you are driving is a VM or across the room.

In a machine of its own

pob launch --msb starts Pob on a Linux microVM — its own kernel, its own screen, Firefox in it, and a copy of your ~/.pob — so a macro can run while you keep using this one. Every launch is a fresh machine, and --count 10 is ten of them side by side.

The pob command

Every install puts a pob on your PATH — the same instance the overlay is working in, driven from a terminal.

launch

pob launch "Work laptop"

Start the app on that instance. Named none and it asks which; pob on its own prints the machine — the instances and their state, and any microVMs with their addresses. --fullscreen puts the overlay over the whole display, with nothing of Pob's on it.

start

pob start

Execute src/main.macro.psl on the running instance — the toolbar's Execute button, from a terminal. pob stop stops it.

check

pob check && pob launch --start

Reads the macro and this machine and prints what is wrong with either — psl, the key behind it, the app, the settings file. It asks nothing of a running Pob, so it answers before there is one.

mcp

pob mcp start

Every command and every flag: the CLI reference.

Getting started

One command on macOS and Linux. It works out which release fits this machine, downloads it, and puts it where a hand would have — Pob.app in Applications on a Mac, the app tree and a link on Linux. Either way the pob command lands on your PATH, and there is no toolchain to install first.

curl -fsSL https://raw.githubusercontent.com/lhypds/pob/master/get.sh | sh

On Linux it installs for you alone, under ~/.local; pipe it into sudo sh instead and it installs for everyone. On macOS an admin account needs no sudo to write to /Applications. Anything after sh -s -- reaches the install: --prefix DIR, --bin DIR, --version VER, or --uninstall to take it back off again.

macOS then needs two permissions. Screen Recording asks for itself the first time Pob captures anything; Accessibility never asks — open the pane and add Pob by hand, or every click is dropped in silence. Pob checks both at launch and says which is missing, with a button that opens the right pane — and after an update, when the switches read as on while nothing works, its Reset Permissions and Quit clears them to be granted afresh.

pob update

Keeps itself current from there: this installs the latest release over the copy that ran it and leaves ~/.pob — settings, instances, macros and logs — untouched. Quit Pob first: the app being replaced is the app running. pob update --check asks whether there is a newer one and changes nothing.

Install it — the command above, or the release zip for your platform, unpacked and installed by hand.
Open it over the app you want automated. Pob floats above it and reads that window.
Give it an instruction, record a macro, or connect an AI client to its MCP server and let the client take it from there.