Sign of Love
A Vision Pro app, built in 19 weeks, that watches your hands and corrects them as you learn American Sign Language. The entire product rests on one question: can the device tell a mistake from a moment it simply could not see?
Teach American Sign Language with the feedback a human tutor gives, without the tutor.
Video lessons show you a sign. They cannot tell you your thumb is two centimetres low, or that the sign you produced means something close to, but not, what you meant.
Tutors can, and they are scarce and rarely free at eleven at night. Apple Vision Pro is the first consumer device that can see a gesture in three dimensions well enough to comment on it. That made it worth attempting, and hard in exactly one place.
| Inside a frame | Corrections within a frame. Feedback later than about ninety milliseconds feels like judgement, so each correction had to land within a single frame. |
|---|---|
| Edge of view | Trust the edge of view. The headset sees hands worst at the edges of its view, exactly where signing happens. We mapped where, and designed the lessons around it. |
| Wrong, gently | Silence beats a false correction. When confidence drops the app says less and never guesses. |
The headset loses track of hands at the edges of its view, and a hand it cannot see clearly looks exactly like a badly formed sign.
Signing is not performed politely in front of your face. ASL uses the space beside the body, above the head and close to the chest, exactly where the headset's cameras see worst.
A naïve build punishes the learner for the hardware. Tell someone their signing is wrong when it was perfect and merely unobserved, and you have not built a tutor. You have built something that erodes confidence.
A single threshold puts the cutoff in the middle of where legitimate signing happens. The fix was never a better threshold. It was refusing to judge from a single frame.
Separate "wrong" from "unseen". Never guess.
We stopped judging single frames and judged the whole sign instead. The app gathers what it sees across the gesture, trusting each moment as much as the headset is sure of it, and decides only when it has seen enough. Beside correct and incorrect there is a third answer: I could not see that.
That third answer changed the product. When the app cannot see well enough it says so, and moves the lesson, floating in space, to where the hands can be seen. Teaching and camera angles became one problem.
Prototype the risk
Before any lessons, a bare app that recorded how sure the headset was of each hand position while a fluent signer signed normally. That data shaped everything after.
Running buildJudge the whole sign
A recogniser that weighs the whole gesture and can answer in three ways, tuned on recordings of signers of different heights, hand sizes and speeds.
Core engineLessons that move
Lessons placed where the cameras see best, moving quietly when tracking slips. How we teach and what the headset can see, designed together rather than one after the other.
ProductHarden and ship
Heat, session length, accessibility, App Store review for a brand-new category, and the problems that only appear after forty minutes in a headset.
App StoreWhat we would do differently.
Record the reference data sooner. The recording tool came in week two and should have been week zero. Every decision downstream depended on it.
Bring a Deaf consultant in at week one, not six. We got the engineering right and some of the pedagogy wrong. Domain expertise is not a review step.
Budget thermals as a feature. Session length on a headset is governed by heat. It should have shaped the lesson structure from the start.
Knowing when the machine cannot know is the whole company now.
The third state is the rule behind everything we have built since: say what the system cannot know, on which device, and stop there. Reach is built on it. What it cannot guarantee, on which OS, is written down rather than hidden.
The hand-tracking research started as Kuji-Vision, one of our experiments in Labs. The recogniser shipped here. The habit of saying "not enough signal" shipped everywhere.
See ReachThe third state, "I could not see that", is the whole product. Everything else is a recogniser.