# Hand Control 1.2.5: instructions for AI agents

Hand Control is a free Windows app that turns the user's webcam into a mouse. This file is the
plain-text copy of https://seidrlabs.online/hand-control/developers/#for-the-agent

## How to connect

- App version: 1.2.5.
- Connector: `HandControlMCP.exe`, installed next to the app, usually at
  `C:\Users\<you>\AppData\Local\Programs\Hand Control\HandControlMCP.exe`
  (the app installs to `%LOCALAPPDATA%\Programs\Hand Control`, per user; an install for all users
  puts it in `C:\Program Files\Hand Control`). Run it as a **stdio** MCP server with no arguments.
  It finds the access key by itself, and reads `api_port` from settings.json when it starts (restart
  the agent after the user changes the port). `--port <n>` (1024-65535) forces a port; any other
  value is ignored.
- The agent API is **off** by default. The user turns it on in Hand Control's **Agents** tab by
  ticking **Allow local programs to connect (agent API)**. That also creates the key. The two extra
  permissions (turn control on/off and change the program; take a camera snapshot) are off by
  default too.
- Address: `http://127.0.0.1:47294` by default, this PC only. The port is `api_port` in
  `%APPDATA%\Hand Control\settings.json`.
- Key: the file `%APPDATA%\Hand Control\api-key.txt`. **Copy key** in the Agents tab copies the same
  key. The connector reads it for you on every call; scripts calling the HTTP API directly need it.
  Any program running as the user can read this file. **New key** stops only programs that kept a
  copy of the old key; unticking the box shuts every program out. How scripts sign with the key:
  [the local API reference](https://seidrlabs.online/hand-control/developers/#api)

## For the agent

You are connected to **Hand Control**, a Windows app that turns the user's webcam into a mouse. The
user points to move the pointer, pinches thumb and index to click, pinches thumb and middle to
right-click, and scrolls with a V-sign and move. In the "Finger tap" click style they tap the index to
click, make a V-sign and tap the middle finger to right-click, and pinch and move to scroll. Two hands
rotate, pan and zoom 3D views. Your tools **observe** this, and with permission **adjust**
it. You cannot start any mouse or keyboard input. Do not offer to, and do not try to simulate input through
these tools.

### Tools

| Tool | Use it to | Notes |
|---|---|---|
| `get_status` | Check running, control on/off, current gesture, hands, profile, and your permissions (`allowed`) | Call this first. |
| `get_hands` | Read hand geometry: 21 landmarks (0 = wrist, 4/8/12/16/20 = fingertips), `fingers_up`, `age_s` | x and y run across the camera image, normally 0 to 1 (slightly outside at the frame edge), mirrored if Mirror is on. Ignore readings with a large `age_s`. |
| `get_events` | Read what happened since `after`: gestures, clicks, scrolls, modifier key presses (Ctrl/Shift), control changes | Pass the returned `next` back as `after`; `more: true` means another page is waiting. `wait_seconds` (up to 25) waits for the next event. |
| `wait_for_gesture` | Block until the user makes a gesture (for example `rotate`, `pan`, `point`, `rock`) | With the default settings two pinching hands report `rotate`, and pulling them apart or together zooms within it; `zoom` is reported only when **Two-hand pinch = rotate** is off in the Gestures tab. While control is off only `none`, `rock`, `off` and `waiting for the previous release` are reported. Default timeout 30 s, maximum 120 s. |
| `list_profiles` | See which 3D programs are supported and the buttons and keys each one sends | |
| `set_control` | Turn hand control on or off | Needs permission. Turning it ON makes the user's hands move their mouse, so only do it when the user asked. |
| `set_profile` | Choose `solidworks`, `fusion360`, `blender`, `onshape` or `browser3d` | Needs permission. Use it when the user says which program they are in. |
| `camera_snapshot` | See the camera image, with the tracked hands drawn on it while the preview is on | Needs permission. It shows the user; take one only when it helps with what they asked. |

### Rules

1. **Respect `allowed`.** If an action tool returns an error saying it is off, tell the user which box
   to tick (the error names it). Do not retry in a loop.
2. **Ask before turning control on** unless the user just told you to.
3. **Tool errors are instructions for the user**, such as "start Hand Control" or "tick the box".
   Pass them on plainly.
4. **Gestures while control is off:** only `none`, `rock`, `off` and `waiting for the previous release` are
   reported. To react to hands while control is off, read `get_hands` and use `fingers_up`.
5. **The rock sign** (index and pinky up for one second) is the user's own on/off switch. Do not
   treat it as a command to you.
6. **Privacy:** hands of other people are filtered out when the user has **Only me** on. Do not ask
   for snapshots you do not need.

### Useful patterns

- *"Tell me when I rotate the model"*: `wait_for_gesture {"gesture": "rotate"}`.
- *Count clicks for a minute*: `get_status` gives `events.latest` = `after`. Then loop
  `get_events {"after": after, "wait_seconds": 25}` (then `after = next`), counting `type == "gesture"`
  events whose `gesture` is `left down` or `right down`, until 60 s have passed. Each click gives one, so a
  double-click counts as two. Do not count `button` events: two-hand rotate and pan hold a mouse button
  too (the right button in Onshape), and every zoom step lets it go and presses it again.
- *"Set it up for Onshape"*: `set_profile {"profile": "onshape"}`, then tell the user that two-hand
  rotate now sends right-drag.
- *Is a hand held open?* `get_hands`, then check `fingers_up` has all four fingers (`index`, `middle`,
  `ring`, `pinky`; the thumb is never listed).

The full HTTP reference behind these tools is
[the local API reference](https://seidrlabs.online/hand-control/developers/#api).
The 3D program bindings and their sources are in
[the 3D programs section](https://seidrlabs.online/hand-control/developers/#programs).