Your Mac stays at home, powered on, logged in. You are outside with a phone and a pair of AR glasses. You tap the temple, say a sentence, and a coding agent starts working on that Mac — reading files, running commands, opening pull requests — while a single line of green text tells you what it is doing. No laptop open, no phone unlocked, no keyboard anywhere in the loop.
That is the whole pitch, and it has not changed. What has changed is everything underneath it: this started as eight patches on top of someone else's project, and it is now fifty — session management, project switching, camera recording with its own demo-video pipeline, usage statistics exported straight to the photo gallery, and a French interface. This post is the full build log, rewritten from scratch to cover all of it.
We Adopted, We Didn't Rewrite
An open-source project called Rokid_Claude, by GitHub user williamlzz, already implements almost exactly this loop: an Android client for Rokid's smart glasses that captures voice, renders a HUD, and talks WebSocket to a small Node/TypeScript relay; the relay transcribes locally with whisper.cpp and spawns claude -p --output-format stream-json, streaming every event back to the glasses. We forked it rather than building our own, because the gap between what it already did and what we needed was small, and the cost of a rewrite was not.
The first wave of work was reliability and security: a design spec, an implementation plan run task-by-task with verification commands instead of guesses, eight patches against the upstream code, and a recompiled APK — two of those patches closing real security holes. That shipped, we used it daily, and the daily use is what generated everything since. Every one of the forty-two patches that followed came from something that broke, felt clumsy, or was missing while actually using the thing on the street: choosing which project to talk to, resuming a session after walking back inside, seeing what the last hour cost, filming a demo for a client. The patch count is now fifty, spanning roughly 3,300 lines of Kotlin and 3,400 lines of TypeScript, backed by 196 relay tests and 181 Android tests.
The Architecture
The loop has four components, and only one of them lives outside your own network.
The flow: tap the temple to capture PCM audio over WebSocket, the relay transcribes it locally with whisper.cpp and applies a small dictionary of spoken shortcuts (saying "how much" becomes /cost, for instance), the relay spawns claude -p in streaming JSON mode and forwards every event to the HUD, and a PreToolUse hook can intercept risky tool calls — Bash, Write, Edit, MultiEdit, NotebookEdit — and push a confirmation request that a gesture on the glasses answers. When the task finishes, the HUD shows the result, the token count, and the cost.
Because the relay speaks a small JSON protocol over a plain WebSocket, the glasses client is not actually load-bearing. A browser or a phone can hit the same relay for debugging — the glasses are just the input device we chose. The protocol itself grew with the feature set: it started with a handful of message types and now carries close to thirty, covering sessions, projects, statistics, and recording, on top of the original prompt/event loop.
The Glasses Fought Back Before the Software Did
The design spec assumed a fairly open Android device. The actual unit is a locked production build — ro.build.type=user, ro.debuggable=0 — and Rokid disables most of the adb surface a normal Android developer relies on.
Consequences fall out of that table, and none of them are documented anywhere in the upstream project:
None of this shows up as an error message. It shows up as a plausible-looking failure: the QR scanner silently closes, the app falls back to a Chinese interface with an empty token, Wi-Fi refuses to reconnect after a reboot. Every one of those has a real cause, and none of them announce it.
Two constraints specific to the Android API level bit later features directly. WifiManager.setWifiEnabled() is a no-op since API 29 — measured after a reboot left Wi-Fi off, the app running, and no code path able to turn it back on; the only remaining move is opening the system panel and asking the human. And the clipboard is readable only by the app that currently has focus, with nothing shared between the glasses and a phone — they are, after all, two separate devices, a fact easy to forget when they are six inches apart.
Real Vulnerabilities, Found by Reading the Source
The relay executes Claude Code on your Mac. Its token is functionally remote code execution. Reading the source before trusting it turned up two bugs that were not cosmetic, and a third defensive rule that later features had to inherit.
The token leaked into a page served publicly. The relay serves a local session mirror and injects the auth token into it automatically, but only for requests it judges to be local — checked by looking at the socket's source address. Tailscale Funnel, the tunnel that exposes the relay to the public internet, connects to that same relay from 127.0.0.1 while relaying traffic from anywhere. The old check saw a loopback address and injected the token into HTML served to the entire internet. The fix checks for proxy headers — x-forwarded-for, x-forwarded-proto, x-forwarded-host, forwarded — and refuses to treat a request as local if any of them are present, even from 127.0.0.1.
Confirmation requests expired into silent approval — or silent denial, depending on which side you were on. A permission prompt was broadcast to connected clients exactly once. If the glasses dropped and reconnected mid-request — which happens constantly on a mobile connection — the new connection never saw the pending request, and it expired into a default choice after sixty seconds with nobody able to answer it. The fix retains the payload of every open request and replays it to any client that connects while it is still pending.
Voice-driven project switching only accepts a project the relay already knows about. Once "switch project" became a real command, the obvious naive version reads whatever path a transcription hands it and opens a Claude Code session there. A mistranscribed sentence would then be free to point the relay — and therefore Claude Code — at an arbitrary directory on the Mac. The rule that shipped instead: setProject validates against a fixed list of known projects, and the same rule applies to a working directory restored from disk after a relay restart, so a hand-edited state file can't reopen it either.
Both original patches shipped with regression tests: a test asserting a genuinely local loopback request still gets the token, a proxied one from the same address does not, and a client connecting after a permission request was broadcast still receives it and can answer it.
Fifty Patches Later: What Got Built On Top
Living with the first version daily is what produced the second. Four capabilities came out of that use, each with its own sharp edge.
Sessions and projects, driven entirely by voice. Saying sessions lists resumable Claude Code sessions as a numbered, swipeable menu; saying projects does the same across every working directory the relay knows about, most-recent first. Resuming replays a session's last messages before voice continues it — deliberately not the word resume itself, which was tried and then removed, because that word means something to Claude ("pick up where we left off") and claiming it for navigation took away a sentence people actually wanted to say. A voice command started from the Mac terminal and one started from the glasses now stay in sync: switching sessions or projects broadcasts to every connected client, because a switch that only updated the initiator once left the glasses silently showing a stale thread.
A menu instead of an interrupt. The very first version treated any tap during a running turn as a kill switch — the only way to check on a session was to end it. The fix opens a menu instead: interrupt, queue the new instruction for after this one finishes, switch session, or switch project, all without losing the turn in flight. Getting the bookkeeping right took several follow-up patches: which session is "active" is now decided by the switch itself, not by whichever session happened to be running when a background turn's first event arrived, and each turn is pinned to its own session so two concurrent claude processes can no longer write into the same transcript.
Camera recording, and a whole pipeline to turn it into a demo. The glasses can film silently while a task runs, driven by voice (record / stop recording), with a state machine — idle, starting, recording, stopping, failed — that is fully testable without the hardware attached. Producing an actual demo video from that footage turned into its own small production process: since the real screen can't be captured, a presentation.html page served by the relay stands in for the HUD, captured full-screen with ffmpeg on the Mac while the glasses record the scene, and a compositing script layers the two. The two clips never start at the same instant — the relay announces "recording started" the moment it receives the command, not when the camera has actually spun up, and that lag was measured at anywhere from 0.6 seconds (camera already warm) to 4.7 seconds (the first shot of a session). Getting the offset right meant reading the moment the on-screen "● REC" dot actually lights up and adding the measured camera lag to it, not using either number alone.
Usage statistics, drawn on the glasses and dropped straight into the photo gallery. Saying stats shows requests, sessions, tokens, and dollars spent over three rolling windows — seven days, thirty days, and all time — deliberately rolling rather than calendar-based, since a Monday-morning calendar week would report "almost nothing" every single week. The dollar figures come from Claude Code's own done event, because the relay is the only part of the system that keeps them; Claude Code's own transcripts record tokens but not cost. Saying export writes that same card into the camera roll through Android's MediaStore API, marking the file pending until the write completes so the gallery never shows a half-drawn image — the only path a sandboxed app has into shared storage at all.
A French interface, with real measurements instead of guesses. The HUD, its menus, and its voice commands now exist in Chinese, English and French, and the French vocabulary was not written from a phrasebook — it comes from actually recording a French voice into the same whisper.cpp model (ggml-small, -l fr) and fixing what broke. Command matching runs every phrase through a canonical form that lowercases it, strips trailing punctuation, folds accents, and singularizes plurals, because whisper's output on this hardware is not stable enough for exact-string matching: "réglages" comes back as "Réglage," "enregistrer" comes back as "enregistré," and a bare "aide" sometimes comes back as the English word "head" because whisper switches languages on very short utterances. All three get caught — the first two by the canonical form, the third by adding "head" itself to the set of phrases that open help, since the word opens no other door in this app. The honest caveat still holds where it always did: this fixes short, well-defined command words, not open-ended dictation. A full sentence spoken in French to describe an actual coding task remains far more likely to land on a wrong-but-valid word that nothing downstream can flag as a transcription error rather than a real instruction.
Notifications that survive a dark screen. A session started elsewhere finishing, or a permission request waiting, used to render into a screen that had already gone to sleep — the message existed, but under the same black overlay that hides everything else, and a permission request would quietly time out unread. The fix wakes the screen for a floating notification, positioned to survive the optical combiner's one real quirk: true black is transparent on this display, so a full-screen dark backdrop makes everything else in the room — a menu bar, an open tab strip, the Dock — float on top of the HUD as if it belonged there. The notification box now floats directly over the live session instead of inside its own opaque frame.
One new, quieter finding along the way: the relay's own permission posture has a trap in it. ROKID_PERMISSION_MODE can be default (the original behavior — every Bash, Write, Edit call blocks on a glasses gesture), auto (a classifier decides, no per-tool confirmation — what the always-on service now runs), or bypassPermissions (no tool-side guardrail at all). The PreToolUse hook and this mode are independent settings, and passing auto without also removing the hook left every tool call waiting on a confirmation nobody was ever going to send, expiring in silent denial after sixty seconds. The relay no longer installs the hook outside default mode, and it now announces its permission posture on every single startup line, so it is never something you have to infer.
That last change is also a real shift in the security model, not just a bug fix: the daily-use configuration now runs in auto, meaning individual tool calls no longer wait on a glasses gesture at all. The actual gate moved earlier — to confirmPrompts, the step that shows the transcribed sentence and waits for Send or Discard before anything is submitted to Claude at all. That is not a workaround; it is the same conclusion the original two vulnerabilities already pointed to, now built into the default configuration instead of just argued for at the end of a blog post.
Tailscale, Then a Compromise on Tailscale
The original design called for the glasses to join a private Tailscale network directly, so there would be no public entry point at all — the same class of exposure the project's own ngrok option creates, avoided entirely. That did not survive contact with the hardware. Installing the Tailscale Android client still requires the companion-app install path, which works, but connecting that client to an account requires an OAuth flow in a browser — and the HUD has no browser, and scrcpy cannot stand in for one because it needs the same adb push the device already refuses.
The fallback is Tailscale Funnel: the Mac exposes the relay on a stable HTTPS URL, reached over the phone's mobile data. It is the same category of exposure as the ngrok path the design set out to avoid, with three differences — a stable domain, a managed certificate, and no separate third-party account. The honest accounting: the URL is publicly discoverable, and only the token and the permission-confirmation gesture stand between that URL and code execution on the Mac. That whole chain now also restarts itself: a background launch service brings the relay and its tunnel back up at every reconnect and every Mac reboot, rather than requiring a manual restart before the first "away from home" attempt of the day. The spec wanted better than Funnel; the hardware did not allow it — but at least what exists now stays up without being asked to.
What the CLI Looks Like Day to Day
From the glasses, a fixed vocabulary of exact phrases drives everything — matched as a full phrase rather than a substring, so a task that happens to contain the word "sessions" does not accidentally trigger the sessions list. Saying help reads the current list back from the code that defines it, so it can't drift out of date; each of these phrases also exists in Chinese and French.
From the Mac, a small CLI wraps the same relay: sending a one-off instruction, sending one and following the result inline, watching the session live, listing and resuming sessions by number, replaying the full transcript across every turn — glasses included, since a phrase spoken to the glasses never appears in any already-open Claude Code window even when every process reads the same transcript file — and granting or denying a pending permission request from the keyboard instead of a gesture. That transcript replay counts in exchanges, not raw messages: on this project the assistant writes roughly seven tool-call messages for every one the user writes, so a plain message count buries the actual instructions under the agent's own tool chatter.
Every risky tool call in default mode still waits for a human gesture before it runs, whether that gesture comes from the glasses or from the CLI's own allow/deny commands — but as the permission-mode discussion above lays out, the control point that actually matters day to day is the sentence you approved before it was ever sent, not each individual tool the agent then calls to execute it.
The Proof
The design's original success criterion was specific: from a network outside the home, speak a task, watch it progress on the HUD, approve one risky action with a gesture, and have it complete on the Mac without touching the Mac or the phone.
We ran that test with the USB cable still physically connected but its adb reverse port forward deliberately removed, so a false pass over the cable was not possible, and the glasses on the phone's own mobile hotspot rather than home Wi-Fi. The round trip completed with the connection arriving through the tailscale process rather than adb, confirming the network path was the one the test was supposed to exercise. The full exchange, glasses to Mac and back: a reply came back reading exactly what was asked for, the correct working directory was already loaded as the active session, and the run finished successfully at a cost of a few cents. Everything built since — sessions, projects, recording, stats, French — has run under that same standing setup, not a separate demo rig.
What's Still Broken
The honest limitations list, because a build log that only lists wins isn't one.
The Takeaways
If you are wiring voice into any agent that can execute code, four things from this build generalize past AR glasses specifically.
Confirmation has to be about the sentence, not the tool. That was true when this project had two patches and it is still true at fifty: the actual point of failure here was never a misbehaving tool call, it was a mistranscribed instruction, and the deployed configuration now reflects that directly — per-tool confirmation is off, and the sentence-level check is the one thing that stayed non-negotiable.
A public tunnel is not automatically the honest fallback it looks like. Tailscale Funnel relays every request through 127.0.0.1, which silently defeats any "is this local?" check built on socket address alone, and that exact gap is what leaked our token in the first place.
Exact-string matching does not survive contact with real speech recognition. Every new voice command added after the first version needed the same accent-folding, pluralization, and cross-language fallback the original commands didn't — measured against a real voice, not assumed from a spec — because whisper's output on this hardware simply isn't stable enough to compare literally.
And untestable platforms don't excuse untested code. The Android client's entry point can't run under a JVM test harness at all, so the rule that grew out of necessity — every decision that can be expressed as a pure function moves out of that untestable file and gets its own test, verified by mutation so a test that would still pass with the logic broken doesn't count — is what let fifty patches land on a device that, by its own admission, cannot be screenshotted, mirrored, or debugged in the conventional sense.
Frequently Asked Questions
- Is this built on an open-source project, or from scratch?
- It forks an existing open-source project, Rokid_Claude by GitHub user williamlzz, which already implemented the glasses-to-relay-to-Claude-Code loop. The first wave added a design spec, a task-by-task implementation plan, and eight patches to the upstream code — two of which closed real security vulnerabilities — plus a recompiled APK. Fifty patches in, it now also has voice-driven session and project switching, camera recording with its own demo-video pipeline, usage statistics exported to the photo gallery, and a French interface.
- How is the connection secured, given the relay can execute code on a Mac?
- The relay listens only on 127.0.0.1 unless explicitly told otherwise, every connection needs a 48-character token or it's refused with a 1008 close code, the tunnel to the public internet is TLS end-to-end, and voice-driven project switching only ever resolves to a project the relay already knows about. Per-tool confirmation on the glasses is now off by default in favor of a sentence-level check — you approve the transcribed instruction before it's ever sent to Claude, rather than approving each tool call it makes to carry it out. The honest caveat is that the public tunnel is discoverable by URL, and only the token and that sentence-level check protect it from there.
- Why Tailscale Funnel instead of keeping the glasses on a fully private network?
- That was the original plan, but the glasses' HUD has no browser, and joining a private Tailscale network requires an OAuth login in one. Screen mirroring tools that could work around that need adb push, which this hardware refuses outright. Tailscale Funnel was the fallback that kept a stable domain and a managed certificate without adding a third-party tunneling account, and it now restarts itself automatically after a reconnect or a Mac reboot.
- Does dictation work in languages other than English?
- The interface and its fixed voice commands now work in Chinese, English and French, with the French vocabulary tuned against a real recorded voice rather than guessed — accent-folding and singularization catch most of whisper's quirks on short command words. Free-form dictation of an actual task is a different problem: a full French sentence can still transcribe into a completely different, grammatically valid word — 'commits' becomes 'commis,' for instance — a failure mode nothing downstream can automatically detect.
- What can the glasses actually do beyond sending a single instruction?
- List and resume Claude Code sessions by voice, switch between known projects, queue or interrupt a running turn instead of only being able to kill it, film a silent demo of a session, pull up usage statistics over rolling 7-day/30-day/all-time windows and export that card straight to the photo gallery, and get a screen-waking notification when a task started elsewhere finishes.
Tagged with
