Mochi: a tiny you that takes your notes

Context

Mochi is a macOS app that sits at the edge of your screen and writes down what you say. Press Option and Space, talk, and a note lands in your Obsidian vault before you have finished the thought.

It looks like you. You upload one photo during setup and Mochi draws a small caricature of you: big head, big eyes, short limbs. That character is the interface. It sleeps while you work and wakes when you speak.

Recognition and speaker separation run on the Mac. No audio is uploaded, and no minute of transcription costs anything.

The problem

Voice notes are easy to record and almost never get read. The recording is the end of the process rather than the start of it, so it sits in a folder with a timestamp for a name and nobody opens it again.

Dictation tools fix half of that by producing text, then hand the text to you rather than to the place your notes actually live. Meeting transcription tools fix a different half by uploading the whole call to somebody’s server, which is a fair trade for a sales team and a poor one for a private conversation.

Both leave the same gap. What I wanted was not a transcript. It was for the sentence I said out loud to already be in the vault, in the folder I would look in, without my having decided to go and put it there.

My role

Everything. Nuits is one person.

The Swift app, the recognition pipeline, the vault writer, the Worker backend, the marketing site, the brand, the mascot, the pricing, the legal pages and the privacy position. There is nobody to hand the uncomfortable trade-off to, which mostly shows up in the places where the honest answer costs a feature.

Approach

I decided the order of writes before anything else, and it settled every argument that came later. The verbatim transcript goes into the vault first, always, before a language model is called at all. The tidied version is appended to the same note afterwards as a second block.

That ordering is the product. If the network is down, if the key is wrong, if the provider is having a bad afternoon, you still have your words. A model failure degrades the note; it can never lose it. Everything else falls out of putting that write first. Recognition has to be local, because a cloud recogniser makes the first write depend on the network. The model has to be yours, because the second write is optional and should be billed to whoever chose it.

The mascot is the same decision at the level of feeling. A menu bar icon is a utility, and you would not notice its absence. A small character that looks like you and reacts within a tenth of a second of the hotkey is something you would miss. That was worth building an animation system for.

Building it

Native Swift. SwiftUI for the panels, AppKit for the overlay, because a window that floats above everything, ignores clicks on request and behaves correctly across Spaces and multiple displays is not something SwiftUI wants to give you. The pet is vector sprite compositing with keyframe and spring animation, not a game engine.

Recognition is Parakeet through FluidAudio, exported to CoreML in three tiers: 227.5 MB for English, 335.7 MB across 25 languages, and 483.1 MB for accents and noisy rooms. Those numbers are measured rather than remembered. A script reads the model repository tree, counts only the exact files the audio SDK fetches, and generates a Swift file that the onboarding cards read from. Nobody can type a size into the interface by hand. An earlier draft of the spec carried estimates that were roughly double the truth in the product’s favour in every single row, which is precisely the direction estimates drift, and the script exists so that cannot happen twice.

The most important thing I found was a silence bug, and it changed the architecture. A decode window that ends inside a run of silence returns an empty string, with no error and a high reported confidence. Twelve seconds of speech transcribes correctly. The same twelve seconds with three seconds of silence appended transcribes to nothing at all. For a product whose whole promise is the verbatim record, a failure that looks like success is the worst one available. So silence never reaches the recogniser. Silero VAD splits the recording into speech regions first, each region is transcribed on its own, and short ones are widened using the surrounding audio rather than padded with zeros, because appended silence is the exact trigger. The voice detection model is always installed and never optional, and that is a correctness decision rather than a performance one.

Meeting mode records system audio and the microphone as separate tracks, so your own voice is never something the software has to guess at. Streaming diarisation separates up to four speakers live. Recognising the same person across two different meetings needs a second engine entirely, which took me too long to accept: a streaming diariser keeps no embeddings between sessions, so speaker one today and speaker one next week are unrelated voices, and no amount of tuning changes that. A second pass now runs after the call. It takes eight to fifteen seconds from each voice’s longest turns, extracts a 256-dimension embedding, and matches it against the profiles stored on the Mac by cosine similarity. The threshold sits deliberately on the cautious side, because a missed match costs one rename while a false match puts a colleague’s name on a stranger’s words. The assignment is decided across all voices at once, so in a two-person call the first speaker cannot claim a profile the second matched better.

The vault writer is the one part with a real test suite: 29 cases over the markdown building and the on-disk behaviour. It writes atomically, appends rather than replaces, and never rewrites a note you have edited, because a vault synced by iCloud punishes all three mistakes with conflict files.

Language models are yours. One registry, one OpenAI-compatible request shape, per-provider overrides where the shape differs, across OpenRouter, Anthropic, Gemini, DeepSeek and others. You paste a key, it goes into the Keychain, and it never touches my backend. Inference is therefore never a cost to the business, which is what lets one $12 plan include unlimited transcription with no asterisk on it.

Pet generation is the only thing that costs me money per user, and it became the most interesting engineering in the project. It runs in two stages against a single vendor through OpenRouter. Stage one turns your photo into four candidate characters at roughly thirteen cents each, and you pick one. Stage two draws that character’s nine artwork states as true SVG at roughly eight cents each. The four-up picker is a deliberate expense: three of those drawings are thrown away, and the alternative is making somebody accept the first thing a model drew of their own face.

The vector model exposes no transparency parameter, which should have meant paying a background-removal vendor as well. Instead every call forces a flat known chroma fill, and the single full-canvas rectangle is stripped from the returned SVG afterwards. The strip only considers direct children of the root, so a rectangle inside a clip path is never mistaken for the backdrop. One vendor, no cutout service, real transparency.

That endpoint taught me most of what I know about verifying against output rather than status codes. Image models are absent from the main model listing and appear only on a separate one. A format parameter documented in the request schema selects nothing, because vector output is chosen by picking a vector model and never by setting the field. And unrecognised provider keys are silently dropped rather than rejected, so a misspelled field name returns a perfectly successful response with none of the controls applied. The provider is pinned explicitly on the photo call, which is compliance rather than routing hygiene: the image vendor is the one named in the subprocessor list, and a silent fallback would send a photograph of somebody’s face to a company that is not on it.

Clerk handles sign-in through its macOS hosted flow. Polar is merchant of record, so global tax is somebody else’s job, and it issues the licence key with an activation limit that lets you move Macs without a support ticket. Sparkle ships updates, and never restarts during a recording. The site is Astro 7 on Cloudflare Workers, with the API and D1 next to it.

There is no crash reporting, no analytics and no telemetry in the app. That is a decision rather than an omission, and it is the one I expect to regret operationally and in no other way. Mochi spends its day holding a live microphone, an open meeting recording and the text of whatever was just said. A crash reporter inside that process uploads stack traces, thread names and, on a bad day, whatever happened to be in a buffer when it died. You cannot promise that recordings never leave the Mac while shipping an agent whose job is to send me data from a process that had a recording open. What that buys is a sentence that needs no asterisk anywhere on the site, in the app, or in the privacy policy. What it costs is that nothing arrives on its own, so a bug is only ever as visible as the report someone chooses to send. The mitigation is a button that copies the last fifteen minutes of the app’s own log, its version and the macOS build onto the clipboard, and touches nothing else. You read it, then you decide.

Outcome

Not shipped. The site is live at mochiformac.com and collecting a waitlist, and the download page says plainly that there is no build to hand you today and that I am not going to guess at a date.

The capture-to-vault path works end to end, which was the thing I insisted on finishing before touching auth, billing or pet generation.

What I’d do differently

I wrote model sizes into the specification from memory and every one of them was wrong by roughly a factor of two, always in my favour. Now a script measures them and the interface can only display what the script wrote. The general version of that lesson was expensive: any number a user will read should be generated, never typed.

I also built the streaming diariser first and assumed that recognising a returning voice was a tuning problem on top of it. It is a different problem needing a different model, and I lost real weeks to trying to get embeddings out of something that has none to give.