Building https://nitter.gq/t.co/JAvQO9DfoC (AI Note taker for healthcare) Quit 9-5 in 2023. Ex @amazon.
New Jersey, United States
Joined January 2009
- Tweets2.5K
- Following611
- Followers873
- Likes2.8K
I lost all Claude artifacts after I cancelled claude plan on one of my accounts.
did anyone else have same experience?
I love WisprFlow. Amazing Product, UX and a great team.
But here's the thing with not just WisprFlow but most voice AI app. The privacy settings are easy to miss.
Somewhere, there's a checkbox that decides what happens to your words. Whether they stay yours. Whether they become training data.
And most don't care till you find out that you consented. By accident. One toggle you never noticed, and your confidential, sensitive words are inside someone's server.
It feels like being robbed. Except worse, because there's nothing you can do about it. You can't unsend your data. There is no undo button.
Here's what I keep coming back to.
Privacy shouldn't be a setting.
A setting can be missed. If your privacy depends on finding the right checkbox on the right screen, it is one misclick from gone.
The only privacy setting you can't get wrong is the one that doesn't exist.
I care about this problem. So I built NoteGenerator.com
A private, local AI note taker for Mac.
You dictate or record meeting notes, everything stays on the Mac. The audio never leaves your Mac. The transcript stays on your Mac. The note stays on your Mac.
Turn off WiFi, it still works.
There is no checkbox deciding where your words go, because your words don't go anywhere.
But more so, whatever tool you use, open its settings tonight and find out where your words end up.
Privacy is more than a checkbox.
We'd like to clear this up, as privacy is central to everything we do at Wispr:
1) This data is from a small subset of users that have opted into this to improve models.
2) Our onboarding surfaces privacy mode as an option for all users: nothing you say is used for model training.
3) The astroturfing on this platform is getting out of hand
I told my customers the AI would get cheaper. I was wrong for three years straight.
When I started building AI note-taking for healthcare providers, I promised that the day the models got cheaper, I'd pass the savings straight through.
Then every new state-of-the-art model shipped at a higher price than the one before it. Not lower. Higher.
And the price you pay today isn't even the real price. It's subsidized. Somebody is eating the difference right now, and nobody eats it forever.
A model in the cloud comes a price tag. A model on your machine comes with none.
Nobody needs a frontier reasoning model to turn a conversation into a clean set of notes. You need something competent, fast, and there. That's it. That work doesn't require the most expensive intelligence on earth.
Local isn't free. But that memory cost is a one-time tax on a machine your user already bought.
So when the model runs on your Mac, there's no meter running. No per-token cost.
Which raises an honest question. How to charge for such a product?
Pay once. Own the thing.
I've been selling privacy as the reason to go local. Privacy is the smaller half.
Your data never leaves your machine. Your bill doesn't either.
Follow along as I build this.
The box is coming back.
If you are old enough, you remember the box.
A CD-ROM with the latest IDE on it. A stack of DVDs so you could install the operating system. Software arrived at your door, in a sleeve, with a serial number printed on the back.
Then it stopped. The pipe got fat enough that you could just pull it down, and the box quietly disappeared. Nobody mourned it. Downloading beat driving to a store, and for ordinary software that argument is over. It is not coming back.
So why would anyone put software in a box again?
Because open weight AI showed up, and the download stopped being the hard part.
Getting the file is trivial. Anyone can pull one down this afternoon.
You can download the model. You can't download the machine.
The model is the easy half. The hard half is the combination: this model, on this silicon, with this much memory, tuned so it actually finishes the job instead of running out of room halfway through.
That tuning is not a file. It is an assembly. Somebody has to sit down and make the model and the hardware agree with each other.
And the business that wants private AI does not want to be that somebody. They do not want to sit there picking quantization settings. They want it to work.
So I think we go back to shipping.
Not CD-ROMs. Not DVDs. A Mac Mini. A DGX box. Something that arrives on a Tuesday, plugs into a wall, and puts private AI on the desk behind an interface a normal person can use. Their models, their workflows, their data staying in their building.
That is the trade people are actually reaching for. Not trust our policy. Just: the thing is here, in the room, and it already works.
You could always download the model. Now you buy the machine it already runs on.
The box is coming back. This time it has a power cable.
For my first post, I’m sharing a letter @NVIDIA signed on why open models matter.
AI will transform every industry, power every company, and be built by every country.
Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty.
The world needs both frontier closed models and frontier open models.
images.nvidia.com/pdf/Open-W…
"It Is a Great Time to Build"
I see it on my feed every single day. AI has made building so easy, it's a great time to build.
And honestly? It's true. Especially for the folks in their late thirties, their forties, their fifties. Everyone who spent years believing building was hard, that shipping an idea was something other people did. They're all building now. And that's a beautiful thing.
Because you always had the idea. What you didn't have was the time. You didn't have the resources. You didn't have the team. And now AI hands you the whole team, right there at your fingertips. In your terminal, in your Claude window, you've got agents: product managers, QA, release engineers, whatever role you need. You spin them up, and the work gets done.
But here's what nobody wants to say out loud.
For all of this, for all this newfound power, you still don't see the flood of stories you'd expect. Where are the solo builders actually making money? Where's the wave of one person, one laptop, paying off the mortgage from a side project?
They're rare. And that rarity is telling you something.
Building was never the bottleneck.
Think about it. The Googles and the Amazons of the world already have infinite building power. They experiment with hundreds, thousands, of products every single year. And only a handful ever take off. If unlimited ability to build were the answer, those companies would win every time. They don't.
So now the individual has that same power, the same ability to build anything. But having it doesn't mean it converts. It doesn't magically turn into sales. Into revenue. Into money in the bank.
All AI has really done is strip away the excuse. It's exposed the truth that was sitting there the whole time: the bottleneck was never building. It was sales. It was distribution. It was getting the thing in front of people who will actually pay for it.
So before you build one more prototype, before you tell yourself this is the side project that pays off the mortgage, that clears the loans, stop. Set realistic expectations.
Building was never the bottleneck. And now that it's easier than ever, building still isn't the solution.
i'm building a completely private, local AI note taker for Mac.
the hardest part was making the whole thing work as if the internet didn't exist.
that meant
- running speech-to-text on-device, not streaming audio to the cloud
- getting note generation fast enough locally to feel instant
- encrypting everything by default
- keeping the app fully usable with Wi-Fi switched off
today i tested exactly that
i took my mac completely offline, dictated my meeting notes, and the app:
- transcribed my voice
- structured it into notes
- pulled out decisions and action items
- saved it all to an encrypted local vault
no cloud. no API calls.
if the network is off, there's nowhere for your words to go.
that's not a privacy policy. that's physics.
building this changed how I think about AI apps.
i expect tighter hardware + software integration as local AI evolves.
follow along to get updates.
#buildinpublic
In past 10 years, Google has delivered almost 10x returns.
Can it be another 10x from here in the next decade?
Q2 was an amazing quarter, with our AI investments redefining what’s possible across every part of our business.
Alphabet revenue grew 24% YoY and Google Cloud accelerated to 82% growth. We saw exciting momentum across the board from Search to YouTube to the Gemini app (which reached 950M monthly active users). Our model APIs are processing 22B tokens/min (up from 16B+ last quarter) driven by our workhorse Flash models. We’re also seeing great adoption of Gemini Enterprise, used by 90% of the Fortune 100, as well as strong demand for our security solutions.
Outstanding results and momentum, and such an exciting moment—thanks to all of our partners and employees around the world! 🙌 About to hop on the call!
I doubt people doing serous work switch between models frequently, let alone every month.
AI-assisted writing is now the default.
Great tips and skills here on how not to sound ai slop. 👇
Substack just shipped AI slop detection to protect readers and featured my writing as an example of how to use AI to edit without pumping out slop.
Basically, I work with AI during the middle 50% of my writing process:
→ First 25%: I voice dictate or write a rough draft without AI.
→ Middle 50%: I use AI and my /no-ai-slop skill to fix grammar, improve clarity, and test my arguments.
→ Last 25%: I go through the piece line by line and edit it manually.
The difference between pumping out slop and making something you’re proud of is giving a damn and staying in the driver’s seat throughout this 25/50/25 process.
📌 More here: creatoreconomy.so/p/use-my-n…
It's amazing how many people were surprised by this insight from @karpathy.
Wispr Flow highlighted this long back.
Claude Code added /voice earlier this year.
If you are not interacting with AI using voice, you are not using AI efficiently.
One pattern I find useful for working with LLMs is a nice long ramble session. Sometimes the LLM needs more bits to understand what you're trying to achieve, but you're too lazy to type them. In these cases I like to lean back, switch to /voice and just ramble for like 10 minutes, total mess, anything goes, full stream of consciousness. Sometimes I declare it up top, something like "switching to speech recognition sorry for any typos...". Sometimes I turn it into a small interview of a few turns. But I find that the LLMs are somehow very good at reconstructing long incoherent rambles and often their echo of your own tangle of thoughts comes out quite a bit cleaner than what you started with. The result is that you improve the mind meld and have to correct things less from that point on.
> It's a magical time to be building.
> You don't need to overthink it.
> Build.
my hot take is
don't build if you don't have anyone to sell to.
So let me get this straight....
1. Agents are about to outnumber humans on the internet, so most of the traffic, transactions, and conversations online will soon be machines talking to other machines while we sleep.
2. Superintelligence exists now, and for $20/mo you pretty much get it all.
3. Cloud agents allow you to run a business 24/7 and from literally a phone while you're waiting to order a latte.
4. Voice AI is wide open. This industry has barely changed since the 90s. Infinite opportunities. Voice AI is finally getting good enough.
5. Mobile apps are interesting again for the first time in 10 years, because an AI-first app that thinks and acts on its own is a different species than the passive ones in the store today. There are kids doing $100k/MRR.
6. It's the golden age of open source. The models you download off Hugging Face for free and own forever are landing within months of the ones behind paywalls, so the smartest thing on earth can't be throttled, priced up, or shut off by a company having a bad quarter.
7. The keyboard is on its way out. We spent 40 years learning to type fast, and it's about to feel like handwriting, because soon you just talk and the computer goes and does it.
8. Every company is about to hire agents with their own logins, their own inboxes, their own track records, and a shadow economy is forming where agents pay, hire, and vouch for other agents.
9. Software stopped being something you buy and became something you rent by the hour. The whole industry is repricing from $50 a seat to thousands per outcome. Tons of opportunity.
10. Robots are about to have the moment software agents just had. The intelligence got solved. Now it's dropping into machines with arms and legs, and the people who can wire AI into hardware are about to be the most fought-over hires on earth.
11. The 10-person startup can now out-ship the 500-person company, and everyone can feel it happening.
12. Every white-collar task is getting a "do it for me" button, and most people haven't pressed it yet.
13. Data you've been sitting on for years is suddenly worth something, because now an agent can actually use it.
14. Language stopped being a barrier. Real-time translation that actually works means the next huge consumer app might get built for a market you can't even read, by someone you'll never meet.
15. Every SOP is turning into a product. The way a business does one thing, written down as a markdown file an agent can run, is now something you can sell. Knowledge that used to live in someone's head became downloadable.
16. The moat moved from what you know to how well your business is written down. The company an agent can actually run wins, so being legible beats being big.
17. Since agents will get scammed by other agents, a whole trust layer has to get built. The Yelp for agents. The escrow for machines. Wide open.
18. You keep the whole thing now. A business that needed 50 people to hit $10M needs 3 and some agents, so instead of splitting it 200 ways with investors, it's just yours.
19. Search engines minted a generation of millionaires and billionaires. LLM search is about to do it all over again. Billions of eyeballs are moving off Google and onto ChatGPT, so being the answer inside the models is the new front page.
20. The margins are stupid. You charge someone what they'd pay a human, and it costs you a few bucks in tokens. That gap used to go to payroll. Now it goes to you.
Any one of these would DEFINE a decade on its own.
We got ALL of them at once, stacked on top of each other in the same 18 months.
It's a magical time to be building.
You don't need to overthink it.
Build.
3+ years of building AI wrappers, and I'd never once met the KV cache.
I only ever called an LLM. The inference ran in someone else's datacenter, behind an API.
Now I'm building an AI note taker that runs the model on your own machine. That dropped me into a layer I didn't know how it worked.
The model holds two kinds of memory, not one.
Memory 1: the context window. It isn't RAM. It's a hard limit on how many tokens the model can look at once. Training sets that number. You can't raise it at runtime.
And attention costs O(n²). Every new token looks back at every token before it. Double the tokens, and the work roughly quadruples. So you keep the window small.
Memory 2: the KV cache. This is the part the API hides. The model writes one token at a time. Each step needs every earlier token.
So the model stores them once and reuses them. That cache is live state. It grows with every token. It sits in RAM the whole time. It saves the recompute not the quadratic cost.
So a token stopped being a billing unit for me. It became a memory unit.
Calling an API is like querying a database. Running the model yourself is like writing the storage engine. Same engineer, one layer down.
Here's the bug that taught me all of it.
To turn a meeting into notes, I split the transcript into chunks. I turn each chunk into a short fact sheet. Then I feed every sheet to the model at once to write the note.
That last step can't stream. The whole thing has to fit the window in one pass.
Then the twist. I made the chunks smaller and the note started failing.
Smaller chunks compress less, so they keep more facts. Overlapping chunks repeat a few more. More facts make a bigger sheet. The sheet grows too big for the window.
Higher quality broke it. The good outcome caused the failure.
The fix: a fold. It's a middle step. It dedups the sheet and pulls it back under the ceiling before the writer ever sees it.
Here's the real shift. Run the model yourself, and it stops being a dependency you call. It becomes a resource you schedule. Context length, KV-cache precision, attention strategy, you own all of it now.
And it made me sharper at the API too. Now I know why long context costs more. Why latency scales with tokens. Why prompt caching exists at all so that it caches the KV state.
I'm building a completely private, local AI note taker for Mac.
Follow along if you want to see what it actually takes to move AI out of the datacenter and onto your laptop.
“Download an open-weight model and run a 12B LLM on your laptop.”
It sounds almost too easy.
And technically, it is until you give the model a real job.
I’m building a private AI note-taking app for Mac. The entire model runs locally. Your meetings, transcripts, and notes never leave your computer.
On paper, the math looked simple.
Gemma 4 12B supports a 256K-token context window.
An hour-long meeting is roughly 8,400 words. Add the system prompt, instructions, and enough room for the model to generate the notes, and the entire job comes to around 15,000 tokens.
That’s only about 6% of the advertised context window.
Feed in the transcript. Get back the notes.
Done.
Except…it didn’t work.
The problem wasn’t the context-window limit.
The problem was RAM.
For the past three years, almost all my LLM inference has happened inside someone else’s data center. I sent an API request, got an answer back, and never had to think much about what was happening underneath.
Running a model locally changes that.
Two things now compete for your laptop’s memory.
The first is the model itself.
A 12B model quantized to roughly 4 bits is still about 6.7GB. Those weights have to sit in memory before the model reads even one token.
That’s the fixed cost.
Then there’s the KV cache.
That was the part I had never really needed to think about.
As the model processes your transcript, it stores information for every token across every layer so it can continue generating without recomputing everything from scratch.
That memory usage grows with the context.
In my testing, an 8,192-token context used approximately 2,688 MiB of KV cache around 0.3 MB per token.
So a 15,000-token job needs roughly another 5GB of active memory.
Now add it up:
6.7GB for the model weights.
About 5GB for the context.
That’s already close to 12GB and macOS, your app, and everything else running on the computer still need memory too.
On a 16GB Mac, you’re already hitting the ceiling.
In the actual build, the largest context I could allocate was 12,288 tokens.
The job needed 15,451.
The model didn’t slowly struggle through it.
It generated nothing.
That was the moment the whole “256K context window” claim started to feel very different.
A context window is not free.
Every token you keep available occupies memory for the duration of the run. The model may technically support 256K tokens, but a 16GB laptop cannot necessarily afford to use them.
The full context window in this configuration could require more than 80GB just for the cache.
That’s why cloud inference had always felt effortless.
A data-center GPU may have tens or even hundreds of gigabytes of memory. A 5GB cache disappears inside that infrastructure.
The cost was always there.
The API provider was simply paying it, so I never saw the bill.
And memory isn’t the only issue.
Even if the laptop could allocate the full transcript, the model still has to process every token before writing the first word. That initial prefill becomes increasingly slow and compute-heavy as the context grows.
So this is the wall I hit:
You can download a 12B model.
You can run it on a 16GB Mac.
You can even give it a very large advertised context window.
But that does not mean you can push an hour-long meeting transcript through it in one shot.
So how do you generate useful notes without sending the transcript to the cloud?
That’s the problem I’m working through now and the subject of the next post.
I’m building a completely private, local AI note taker for Mac.
Follow along if you’re interested in what it actually takes to move AI from the data center onto your laptop.
the world doesn't need another AI note taker. i'm building one anyway.
not because it needs one, because nearly every other one ships your conversations to the cloud, and i wanted one that just doesn't.
no server, no account, no bot joining your call. runs on your mac.
follow along, building it in public 👇
Kunal Modi retweeted
Beginning July 20, Claude Fable 5 will be included in all Max and Team Premium plans, at 50% of limits.
Pro and Team Standard users will continue to have access to Fable via usage credits, and will receive a one-time $100 credit.
Demand for Fable has been challenging to predict, which is why we rolled it out to subscription plans in stages, extending access several times as we secured additional capacity.
my best performing tweet in more than a month.
If you are seeing this, what else should I post about?
