MCP server for virtual desktop control

Give your AI agent eyes, hands, and a full desktop.
An MCP server that lets LLM agents see the screen, move the mouse, type on the keyboard, launch apps, and run shell commands — in a sandboxed virtual desktop, or on the one in front of you.

If a human can do it on a desktop, your agent can too.

Your browser does not support the video tag.

GhostDesk demo — from a single prompt ("open the browser, go to Google News, and tell me the latest headlines in the Technology section"), the agent launches Firefox, navigates to Google News, switches to the Technology section, and reports the latest stories back.

---

One binary, one MCP endpoint, three desktops. The tool surface is the same everywhere — an agent calls mouse_click and app_launch without knowing which desktop it is on.

	Linux	macOS	Windows
How it runs	the Docker container, in one command	a native binary	a native binary
The desktop it drives	a virtual one, shipped inside the image	the Mac in front of you	the PC in front of you
Sandboxed	yes — disposable, one per agent	no	no
Driven through	zwlr_virtual_pointer_v1, Sway IPC, grim	Quartz Event Services, Accessibility, screencapture	SendInput, GDI, EnumWindows

Linux is the server. The container is the deployment this README shows unless it says otherwise, and the only one of the three with a sandbox around it. macOS and Windows run the very same server against a real desk — no container, and therefore no isolation. Jump to macOS or Windows, or read what differs between the three.

Table of contents

• Quick start
• Tools
• Model requirements
• Why GhostDesk?
• How it works
• Secure local run (TLS + auth)
• Running many agents
• Custom image
• Run on macOS, without the container
• Run on Windows, without the container
• What differs from the container
• Configuration
• Security
• Troubleshooting
• Build from source
• License

---

Quick start

1. Run the container

One command, plain HTTP, no password. Fine for kicking the tires on a laptop you trust — not fit for anything beyond that. Ready to harden it? Jump to Secure local run.

docker run -d --name ghostdesk-demo \
  --shm-size 2g \
  -p 3000:3000 \
  -p 6080:6080 \
  ghcr.io/yv17labs/ghostdesk:latest

The latest image ships with Firefox, the foot terminal, mousepad (text editor), galculator, and passwordless sudo for the agent user — enough to demo a browsing + note-taking workflow out of the box. Need a different app set? Build your own on top of base — see Custom image.

The container boots in the dev posture: plain HTTP on both ports, every auth gate disarmed on purpose. You'll see warnings in the logs reminding you of that — they go away once you follow the secured path below.

2. Connect your AI

GhostDesk speaks MCP over the Streamable HTTP transport — any MCP-compatible client can drive it. Point your client at http://localhost:3000/mcp:

Claude Desktop / Claude Code

{
  "mcpServers": {
    "ghostdesk": {
      "type": "http",
      "url": "http://localhost:3000/mcp"
    }
  }
}

SpecterChat — the chat client we build for this, open source: YV17labs/SpecterChat. Most chat UIs drop the image an MCP tool returns — they render it or they forward it to the model, rarely both — and a screen_shot() the model never sees is the whole product missing. SpecterChat displays it inline and sends it back as base64. It talks to any OpenAI-compatible endpoint (llama.cpp, vLLM, Ollama, LM Studio), so it pairs with the local stacks below; macOS, Linux and Windows builds are on its releases page.

Any other MCP-compatible client — same URL, no headers, no auth. That's the whole demo posture.

3. Watch your agent work

Open http://localhost:6080/ in your browser to see the virtual desktop in real time. No password prompt — the dev posture skips it.

Service	URL
MCP server	http://localhost:3000/mcp
noVNC (browser)	http://localhost:6080/
Health probes	http://localhost:3000/health/{live,ready,startup}

The probes are what the container's HEALTHCHECK reads, and they answer about the desktop rather than about the processes: ready goes down when the compositor stops answering the window seam, live when the connection behind the virtual pointer and keyboard is gone — a state in which every tool still replies and none of them does anything.

Give your agent a first prompt to confirm the wiring is right:

"Take a screenshot of the desktop, list the installed applications, then open Firefox and go to wikipedia.org."

You should see Firefox launch in the noVNC tab, the URL bar fill in, and the page load — all under your agent's control.

4. When you're done

docker stop ghostdesk-demo && docker rm ghostdesk-demo

The demo run creates no named volume, so this leaves nothing behind.

---

Tools

Fourteen tools, named verb_noun, and this is the whole surface — no hidden endpoint, no second protocol. Defaults are in parentheses, ? marks an optional parameter, and every coordinate is a pixel offset in the last screen_shot(): the moment the screen changes, coordinates computed from the previous capture are stale.

Screen

Tool	Parameters	Returns
screen_shot	region? — {x, y, width, height}, cropped at native resolution · format "webp" | "png" (webp) · stabilize bool (true) — wait up to 5 s for the screen to settle · quality 1–100 (50, WebP only; raise it for fine fonts or design surfaces)	one image block — {"type": "image", "data": "<base64>", "mimeType": "image/webp"}

Mouse and keyboard

The seven input tools answer with the same verdict, and screen_changed is the field worth branching on: false means the act landed on nothing. It is a signal, not an error — the answer is a fresh capture, never a retry at the same coordinates.

{"action": "Clicked left at (612, 335)", "screen_changed": true, "reaction_time_ms": 180}

Tool	Parameters
mouse_move	x int · y int
mouse_click	x · y · button "left" | "middle" | "right" (left)
mouse_double_click	x · y · button (left)
mouse_drag	from_x · from_y · to_x · to_y · button (left)
mouse_scroll	x · y · direction "up" | "down" | "left" | "right" (down) · amount 1–5 wheel notches (3)
key_type	text string — Unicode, newlines and tabs, layout-independent
key_press	keys string — one chord, + between tokens. Modifiers: ctrl/control, alt/option, shift, super/meta/win/cmd/command. Named keys: return/enter, escape/esc, backspace, delete, tab, space, home/end, pageup/pagedown, left/right/up/down, f1–f12

Past a sentence or two, clipboard_set(text) plus the paste chord beats key_type: it is instant, and immune to autocomplete and to the app's own key handlers.

Clipboard

Tool	Parameters	Returns
clipboard_get	—	the clipboard as text — an empty string when it is empty or holds something that is not text
clipboard_set	text string	Clipboard set (N characters)

Apps

Tool	Parameters	Returns
app_list	—	result[], one entry per installed app: name, exec. This is the launch whitelist, and exec is the exact string app_launch takes
app_running	—	result[], one entry per real client window: app, title, pid, focused
app_launch	command string — an exec from app_list, arguments not accepted · wait_for_window bool (true)	pid, log_file, action, plus window and window_wait_ms once a window appeared — and the settled screen as an image block, so no follow-up screen_shot() is needed
app_status	pid int — one returned by app_launch · lines int (50)	pid, running, log_file, tail — the tail of the captured stdout/stderr

app_list is a whitelist rather than a hint: app_launch refuses anything outside it, arguments included, whatever name the model sends.

On the wire

A call is ordinary MCP over Streamable HTTP — POST /mcp, whose Accept header has to name both application/json and text/event-stream or the endpoint answers 406.

// the params of a tools/call request
{"name": "mouse_click", "arguments": {"x": 612, "y": 335}}

The result carries its payload twice — once in structuredContent, once as a text block holding the same JSON, which is what a client older than structured output reads. screen_shot is the exception and answers with an image block.

Two headers are GhostDesk's own: Authorization: Bearer …, required once a cert is mounted (Secure local run), and GhostDesk-Model-Space, for models that emit normalised coordinates (Model requirements).

---

Model requirements

Your inference stack must cover four capabilities — all four are mandatory:

1. Text + vision — the agent perceives the desktop through screenshots and needs a model that can interpret them.
2. Tool use — GhostDesk exposes its tools as function calls; the model must be able to invoke them.
3. MCP client — the host needs to speak Streamable HTTP MCP to reach the GhostDesk server.
4. WebP image support — GhostDesk returns screenshots as WebP by default to keep payloads small and inference fast. A stack that can only decode PNG or JPEG will not work out of the box.

Points 3 and 4 are where most stacks fall short, and both halves have an answer here: SpecterChat on the client side, and the llama.cpp forks below on the inference side.

Coordinate space — GhostDesk-Model-Space header

By default no header is needed: Claude and the other major frontier LLMs work out of the box. Qwen3.x need the client to send GhostDesk-Model-Space: 1000 on every MCP request.

Example MCP client config:

{
  "mcpServers": {
    "ghostdesk": {
      "url": "http://localhost:3000/mcp",
      "headers": {
        "GhostDesk-Model-Space": "1000"
      }
    }
  }
}

Running locally

Three backends are tested here. Two are llama.cpp forks we maintain, both kept current with upstream, both adding the WebP decoding upstream still lacks — the day it lands there, they are archived and this points at upstream directly. The third is upstream mlx-vlm, on Apple Silicon.

• YV17labs/llama-cpp-webp — branch feature/webp. Start here. WebP decoding and nothing else on top of upstream, so it stays close to master and inherits its backend work. It is the faster of the two on Metal and on CUDA — on an Apple Silicon Mac or an NVIDIA card, this is the one to run.
• YV17labs/llama-cpp-turboquant-webp — branch feature/turboquant-webp. The same WebP support plus the turbo-quant KV cache (--cache-type-v turbo3). Still maintained and still tracking upstream, but turbo quant is no longer where the interest is, and this is no longer the first recommendation.
• Blaizzy/mlx-vlm — upstream as it ships, since it decodes WebP already and needs nothing from us. Apple Silicon only, and the MLX path rather than a llama.cpp one.

Run whatever local model you like — nothing in GhostDesk is pinned to one. The one behind my own runs is Qwen3.6-35B-A3B: 35B parameters with only 3B active per token, and on desktop control that ratio is the whole point — the agent decides where to click on every step, so tokens per second is what you feel.

The commands

One tested invocation per backend. They do not take the same flags, so each gets its own rather than one command with a switch — and the model in them is an example, not a requirement: swap in whatever you run.

llama-cpp-webp — --image-min-tokens 1024 is the one that matters for desktop control: it floors how much of the token budget a screenshot gets, and a screenshot the model reads at too coarse a scale is where off-target clicks come from.

build/bin/llama-server \
  --model ~/Models/Qwen3.6-35B-A3B-Q4_K_M.gguf \
  --mmproj ~/Models/Qwen3.6-35B-A3B-mmproj-F16.gguf \
  --alias 'Qwen3.6-35B-A3B-Q4_K_M' \
  --host 127.0.0.1 --port 8080 \
  --ctx-size 131072 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --flash-attn on \
  --image-min-tokens 1024 \
  --reasoning on --reasoning-format deepseek --reasoning-preserve \
  --jinja

llama-cpp-turboquant-webp — the KV cache goes to --cache-type-v turbo3, and --cache-reuse 256 keeps the prefix across turns, which a desktop session hits constantly: the conversation grows by one screenshot and one tool result at a time. --spec-type draft-mtp turns on the model's own multi-token-prediction draft head, so speculative decoding needs no second model loaded beside it.

build/bin/llama-server \
  --model ~/Models/Qwen3.6-35B-A3B-Q4_K_M.gguf \
  --mmproj ~/Models/Qwen3.6-35B-A3B-mmproj-F16.gguf \
  --alias 'Qwen3.6-35B-A3B-Q4_K_M' \
  --host 127.0.0.1 --port 8080 \
  --ctx-size 131072 \
  --cache-type-k q8_0 --cache-type-v turbo3 \
  --flash-attn on \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  --reasoning on --reasoning-format deepseek \
  --jinja --cache-reuse 256

mlx-vlm — the flags are the same three ideas under other names: --kv-bits 8 --kv-quant-scheme uniform quantises the KV cache, --enable-thinking turns reasoning on, and --draft-kind mtp is the multi-token-prediction draft head. That last one is the difference that costs something: here the head is a second set of weights on disk (--draft-model), not a switch on the model already loaded.

.venv/bin/mlx_vlm.server \
  --model ~/Models/Qwen3.6-35B-A3B-MLX-4bit \
  --host 127.0.0.1 --port 8080 \
  --kv-bits 8 --kv-quant-scheme uniform \
  --enable-thinking \
  --draft-model ~/Models/Qwen3.6-35B-A3B-MLX-mtp --draft-kind mtp

Each of the three exposes an OpenAI-compatible endpoint on http://127.0.0.1:8080; point your MCP host's inference backend at it — SpecterChat's endpoint field takes that URL as is — and remember the GhostDesk-Model-Space: 1000 header for the Qwen family.

---

Why GhostDesk?

Browser automation tools (Playwright, Puppeteer, Selenium…) were built for human test engineers driving a browser with selectors. They do one thing, and they do it well — inside the browser.

GhostDesk is built from the other end: for AI agents, driving everything a desktop runs. Browsers, native apps, IDEs, terminals, office suites, legacy software, internal tools. If it renders pixels on screen, your agent can see it and use it — in one conversation, across many applications, without a line of glue code.

You don't write selectors. You write a prompt:

"Open the CRM, export last month's leads as CSV, open LibreOffice Calc, build a pivot table, screenshot the chart, and email it to the team."

The agent opens the browser, logs in, downloads the file, switches to LibreOffice, processes the data, captures the result, composes the email, sends it. One prompt, multiple apps, fully autonomous — no glue code, no per-site scraper, no brittle selector chain.

That is what agents using a desktop looks like.

Runs on models you can actually host

Desktop control needs to be fast — an agent that takes twelve seconds to decide where to click is unusable. The local path is first-class here, and it is three concrete things rather than a promise: screenshots ship as WebP so a capture costs a small payload, the coordinate space a model emits is one header away, and the two llama.cpp forks below carry the WebP decoding upstream still lacks. No API bill, and no screenshot of your desktop leaving your network.

Frontier models (Claude, GPT-4o, Gemini) work too and remain the smoothest path — but they are not the bar. See Model requirements for the supported stacks and the one coordinate-space setting that matters.

---

How it works

GhostDesk drives a desktop and exposes it as an MCP server. The three ways to run it are the three columns at the top of this page — the container, a macOS binary, a Windows binary — and the tool surface is identical in all of them. The container is what the rest of this README shows unless it says otherwise.

The agent perceives the screen by calling screen_shot(), which captures the full desktop at native resolution and returns it as WebP (or PNG). An optional region= argument can crop to a sub-rectangle when the agent explicitly wants to narrow its focus.

This works with any application — web apps, native apps, legacy software, Canvas, WebGL.

Built in Rust

GhostDesk is a single compiled binary. It links libc and nothing else — no interpreter, no virtual environment, no package tree to harden at build time.

The tool host mounts itself on the HTTP transport, each domain is a service resolved by type, and the whole dependency graph is verified at boot — a missing binding is a startup error, never a runtime surprise. The endpoint is closed by default: a guard has to bind before /mcp answers anything at all.

The operating system sits behind five traits — input, screen, windows, clipboard, application catalogue — and each OS is one directory implementing those five, under crates/platform/src. Nothing above that boundary names a desktop, which is what made the second one possible at all, and the third one routine.

On Linux the compositor is driven from pure Rust: GhostDesk speaks zwlr_virtual_pointer_v1 and zwp_virtual_keyboard_v1 directly over the Wayland socket, with an XKB keymap it generates on the fly — which is why text entry produces identical output on a French AZERTY host and a US QWERTY one. On macOS the same five contracts are answered by Quartz Event Services, the Accessibility API, screencapture and the pasteboard. On Windows they are answered by SendInput, EnumWindows, GDI, the Win32 clipboard and the Start Menu — all in process, because Windows is the one of the three that ships no capture or clipboard tool to shell out to.

---

Secure local run (TLS + auth)

The Quick start above drops every gate so you can kick the tires in thirty seconds. The moment you want to expose this to anything beyond your own laptop — another machine on your LAN, a devcontainer port-forward on an untrusted network, a teammate's browser — flip to the secured posture: real TLS + bearer-token auth on MCP + password prompt on noVNC.

GhostDesk couples TLS and auth: mount a cert and you get wss:// + bearer-token on MCP + a single-password prompt on noVNC (see Security → Auth ≡ TLS). mkcert issues a browser-trusted cert for localhost in two commands:

# Issue a locally-trusted cert (first time only — installs a local CA in your trust store)
mkcert -install
mkdir -p tls
mkcert -cert-file tls/server.crt -key-file tls/server.key localhost 127.0.0.1 ::1

# Generate the MCP and VNC secrets
export GHOSTDESK_AUTH__TOKEN=$(openssl rand -hex 32)
export GHOSTDESK_VNC_PASSWORD=$(openssl rand -hex 16)

Pick a container name that matches the agent's role — sales-agent, research-agent, accounting-agent… Below we use my-agent as a placeholder; replace it everywhere in the command.

# Run the container — cert mounted, TLS + auth enabled everywhere
docker run -d --name ghostdesk-my-agent \
  --restart unless-stopped \
  --cap-add SYS_ADMIN \
  --shm-size 2g \
  -p 3000:3000 \
  -p 6080:6080 \
  -v ghostdesk-my-agent-home:/home/agent \
  -v "$PWD/tls/server.crt:/etc/ghostdesk/tls/server.crt:ro" \
  -v "$PWD/tls/server.key:/etc/ghostdesk/tls/server.key:ro" \
  -e GHOSTDESK_AUTH__TOKEN \
  -e GHOSTDESK_VNC_PASSWORD \
  -e TZ=America/New_York \
  -e LANG=en_US.UTF-8 \
  ghcr.io/yv17labs/ghostdesk:latest

echo "MCP token:    $GHOSTDESK_AUTH__TOKEN"
echo "VNC password: $GHOSTDESK_VNC_PASSWORD"

Once the container is up, update your MCP client config — same shape as the demo, now over https:// with a bearer token:

Claude Desktop / Claude Code

{
  "mcpServers": {
    "ghostdesk": {
      "type": "http",
      "url": "https://localhost:3000/mcp",
      "headers": {
        "Authorization": "Bearer <paste $GHOSTDESK_AUTH__TOKEN here>"
      }
    }
  }
}

Any other MCP-compatible client — same URL, plus an Authorization: Bearer <token> header in whatever form your client accepts.

Then open https://localhost:6080/ in your browser — the mkcert CA installed by mkcert -install is already in your trust store, so the browser accepts the cert with no warning. noVNC will prompt for $GHOSTDESK_VNC_PASSWORD.

Going to production? Swap the mkcert leaf for a real cert, source both secrets from your secret manager, and front port 6080 with an identity-aware proxy — SECURITY.md has the full contract.

--cap-add SYS_ADMIN — Required by Electron apps (VS Code, Slack, etc.) and other applications that need Linux user namespaces to run their sandbox. Safe to remove if you don't need them.

The named volume persists the agent's home directory across restarts — browser passwords, bookmarks, cookies, downloads, and desktop preferences are all preserved. On the first run, Docker automatically seeds the volume with the default configuration from the image.

---

Running many agents

One agent is one container. Two of them share nothing — not the filesystem, not the desktop, not the clipboard — so a second agent is a second port pair, a second volume and a second name. What differs between two of them is the system prompt you give the model, the applications in the image (Custom image), and the networks you attach the container to.

Three agents, one compose file

# docker-compose.y