Your Mac stays at home, powered on, logged in. You are outside with a phone and a pair of AR glasses. You tap the temple, say a sentence, and a coding agent starts working on that Mac — reading files, running commands, opening pull requests — while a single line of green text tells you what it is doing. No laptop open, no phone unlocked, no keyboard anywhere in the loop.
That is the whole pitch. This post is the build log: what we used, what broke, and the two real vulnerabilities we ended up patching along the way.
We Adopted, We Didn't Rewrite
An open-source project called Rokid_Claude, by GitHub user williamlzz, already implements almost exactly this loop: an Android client for Rokid's smart glasses that captures voice, renders a HUD, and talks WebSocket to a small Node/TypeScript relay; the relay transcribes locally with whisper.cpp and spawns claude -p --output-format stream-json, streaming every event back to the glasses. We forked it rather than building our own, because the gap between what it already did and what we needed was small, and the cost of a rewrite was not.
What we brought to it: a design spec, an implementation plan run task-by-task with verification commands instead of guesses, eight patches against the upstream code, and a recompiled APK. Two of those patches close real security holes. The rest are the kind of fix that only exists because someone actually ran the thing on real hardware instead of reading the README.
The Architecture
The loop has four components, and only one of them lives outside your own network.
The flow: tap the temple to capture PCM audio over WebSocket, the relay transcribes it locally with whisper.cpp and builds a prompt, the relay spawns claude -p in streaming JSON mode and forwards every event to the HUD, and a PreToolUse hook intercepts risky tool calls — Bash, Write, Edit — and pushes a confirmation request that a gesture on the glasses answers. When the task finishes, the HUD shows the result, the token count, and the cost.
Because the relay speaks a small JSON protocol over a plain WebSocket, the glasses client is not actually load-bearing. A browser or a phone can hit the same relay for debugging — the glasses are just the input device we chose.
The Glasses Fought Back Before the Software Did
The design spec assumed a fairly open Android device. The actual unit is a locked production build — ro.build.type=user, ro.debuggable=0 — and Rokid disables most of the adb surface a normal Android developer relies on.
Three consequences fall out of that table, and none of them are documented anywhere in the upstream project:
None of this shows up as an error message. It shows up as a plausible-looking failure: the QR scanner silently closes, the app falls back to a Chinese interface with an empty token, Wi-Fi refuses to reconnect after a reboot. Every one of those has a real cause, and none of them announce it.
Two Real Vulnerabilities, Found by Reading the Source
The relay executes Claude Code on your Mac. Its token is functionally remote code execution. Reading the source before trusting it turned up two bugs that were not cosmetic.
The token leaked into a page served publicly. The relay serves a local session mirror and injects the auth token into it automatically, but only for requests it judges to be local — checked by looking at the socket's source address. Tailscale Funnel, the tunnel that exposes the relay to the public internet, connects to that same relay from 127.0.0.1 while relaying traffic from anywhere. The old check saw a loopback address and injected the token into HTML served to the entire internet. The fix checks for proxy headers — x-forwarded-for, x-forwarded-proto, x-forwarded-host, forwarded — and refuses to treat a request as local if any of them are present, even from 127.0.0.1.
Confirmation requests expired into silent approval — or silent denial, depending on which side you were on. A permission prompt was broadcast to connected clients exactly once. If the glasses dropped and reconnected mid-request — which happens constantly on a mobile connection — the new connection never saw the pending request, and it expired into a default choice after sixty seconds with nobody able to answer it. The fix retains the payload of every open request and replays it to any client that connects while it is still pending.
Both patches shipped with regression tests: a test asserting a genuinely local loopback request still gets the token, a proxied one from the same address does not, and a client connecting after a permission request was broadcast still receives it and can answer it.
Silent Failure Modes That Each Cost Real Time
Every failure on this project looked like something else. None of them raised an exception. This is the list we wish had existed before we started.
Tailscale, Then a Compromise on Tailscale
The original design called for the glasses to join a private Tailscale network directly, so there would be no public entry point at all — the same class of exposure the project's own ngrok option creates, avoided entirely. That did not survive contact with the hardware. Installing the Tailscale Android client still requires the companion-app install path, which works, but connecting that client to an account requires an OAuth flow in a browser — and the HUD has no browser, and scrcpy cannot stand in for one because it needs the same adb push the device already refuses.
The fallback is Tailscale Funnel: the Mac exposes the relay on a stable HTTPS URL, reached over the phone's mobile data. It is the same category of exposure as the ngrok path the design set out to avoid, with three differences — a stable domain, a managed certificate, and no separate third-party account. The honest accounting: the URL is publicly discoverable, and only the token and the permission-confirmation gesture stand between that URL and code execution on the Mac. The spec wanted better; the hardware did not allow it.
What the CLI Looks Like Day to Day
From the glasses, a fixed vocabulary of exact phrases drives everything — matched as a full phrase rather than a substring, so a task that happens to contain the word "sessions" does not accidentally trigger the sessions list. Saying help reads the current list back from the code that defines it, so it can't drift out of date.
From the Mac, a small CLI wraps the same relay: sending a one-off instruction, sending one and following the result inline, watching the session live, replaying the transcript across every turn (glasses included, since a phrase spoken to the glasses never appears in any already-open Claude Code window even when every process reads the same transcript file), listing resumable sessions, resuming one by number, and granting or denying a pending permission request from the keyboard instead of a gesture.
Every risky tool call still waits for a human gesture before it runs, whether that gesture comes from the glasses or from the CLI's own allow/deny commands — the control point is the sentence you approved, not each individual tool the agent then calls to execute it.
The Proof
The design's success criterion was specific: from a network outside the home, speak a task, watch it progress on the HUD, approve one risky action with a gesture, and have it complete on the Mac without touching the Mac or the phone.
We ran that test with the USB cable still physically connected but its adb reverse port forward deliberately removed, so a false pass over the cable was not possible, and the glasses on the phone's own mobile hotspot rather than home Wi-Fi. The round trip completed with the connection arriving through the tailscale process rather than adb, confirming the network path was the one the test was supposed to exercise. The full exchange, glasses to Mac and back: a reply came back reading exactly what was asked for, the correct working directory was already loaded as the active session, and the run finished successfully at a cost of a few cents.
What's Still Broken
The honest limitations list, because a build log that only lists wins isn't one.
The Takeaways
If you are wiring voice into any agent that can execute code, three things from this build generalize past AR glasses specifically. Confirmation has to be about the sentence, not the tool — the actual point of failure here was never a misbehaving tool call, it was a mistranscribed instruction, so that's the layer that needs the human gate. A public tunnel is not automatically the honest fallback it looks like — Tailscale Funnel relays every request through 127.0.0.1, which silently defeats any "is this local?" check built on socket address alone, and that exact gap is what leaked our token in the first place. And undocumented hardware constraints cost more time than the actual application code — almost every hour on this project went into a locked adb surface, a config file with the wrong file owner, or a TMPDIR that quietly disabled a safety hook, not into the relay, the HUD, or Claude Code itself.
Frequently Asked Questions
- Is this built on an open-source project, or from scratch?
- It forks an existing open-source project, Rokid_Claude by GitHub user williamlzz, which already implemented the glasses-to-relay-to-Claude-Code loop. We added a design spec, a task-by-task implementation plan, eight patches to the upstream code — two of which close real security vulnerabilities — and a recompiled APK with the permissions the published release was missing.
- How is the connection secured, given the relay can execute code on a Mac?
- Four layers: the relay listens only on 127.0.0.1 unless explicitly told otherwise, every connection needs a 48-character token or it's refused with a 1008 close code, every risky tool call (Bash, Write, Edit) requires a human gesture on the glasses before it runs, and the tunnel to the public internet is TLS end-to-end. The honest caveat is that the public tunnel is discoverable by URL, and only the token and the gesture confirmation protect it from there.
- Why Tailscale Funnel instead of keeping the glasses on a fully private network?
- That was the original plan, but the glasses' HUD has no browser, and joining a private Tailscale network requires an OAuth login in one. Screen mirroring tools that could work around that need adb push, which this hardware refuses outright. Tailscale Funnel was the fallback that kept a stable domain and a managed certificate without adding a third-party tunneling account.
- Does dictation work in languages other than English?
- Not reliably yet. The relay currently locks to English because the project defaults to Chinese for its whisper.cpp transcription. French was tested with vocabulary priming and improved short jargon like 'commit' and 'push', but longer phrases can still transcribe into a completely different, grammatically valid word — a failure mode nothing downstream can automatically detect.
Tagged with
