Jake Jennings (@jakej) header
Jake Jennings (@jakej) profile picture

iOS, embedded, developer tools, hardware, gaming, ML, CAD/CAM, CNC • By day: Apple Frameworks team @google • Opinions are my own • Like ≠ endorsement

St. Louis, MO
Joined April 2009
Photo by Jake Jennings (@jakej)Photo by Jake Jennings (@jakej)Photo by Jake Jennings (@jakej)Photo by Jake Jennings (@jakej)Photo by Jake Jennings (@jakej)Photo by Jake Jennings (@jakej)Photo by Jake Jennings (@jakej)Photo by Jake Jennings (@jakej)Photo by Jake Jennings (@jakej)
I gave my RTX 5090 a smaller job: reading what my coding agents skip. Nine local models, a fixed 1,024-token answer allowance, and a practical search for fast summaries that keep their evidence. jacob-jennings.com/posts/rtx…
67
Jake Jennings retweeted
Gemini 4 Neon: frontier-level intelligence with fast end-to-end task completion across complex software engineering, knowledge work, and science workflows. Begins rolling out today in the Gemini API in @GoogleAIStudio, Gemini for business, for paid Google AI Pro and Ultra subscribers in Google @Antigravity, and for Google AI Pro and Ultra subscribers in the @GeminiApp.
368
527
305
6,748
391,540
Jake Jennings retweeted
Gemini 4 Carbon is rolling out today. It’s built for sustained, long-horizon tasks, reasoning deeply through complex software engineering, cybersecurity, and scientific discovery. Coming globally via the Gemini API in @GoogleAIStudio and Gemini for business, and for Ultra subscribers in @antigravity, the @GeminiApp, and AI Mode in Search.
570
605
266
8,568
798,009
Beyond the slop: how good have image and video models actually become? Join us for our first image and video event in San Francisco on Tuesday, November 17! AI-generated images and video have gone from spaghetti-eating slop to commercially valuable outputs, sometimes indistinguishable from reality. Every leap has made us rethink how we measure the best models, from the prompts we test to the way people judge the results. On Tuesday, November 17, we will share what we've learned about the challenging craft of measuring the subjective, and hear from a panel of leaders building the media generation frontier - across image, video, and beyond. Submit your questions when you register and we'll answer a selection live. Register 👇 luma.com/artifi-7w32?utm_sou…
2
4
37
12,015
Jake Jennings retweeted
RIP @SirPatStew, thanks for all the Star Trek TNG growing up. Once upon a time, when I worked at @twitter, I was asked if I wanted to be in a Vine the company was making. Turns out it was none other than Captain Jean-Luc Picard...
21
212
3
1,902
58,835
What if we could break the enshittification cycle by forming a bank backed by VC money that gives high interest loans to enable bootstrapping of nonprofit startups that use AI to replace software that serves investors instead of employees and users
1
55
Cmonnn Qwen 4 by the weekend pls
87
Introducing the Artificial Analysis Decision Index, benchmarking decision models across classification, tool calling, knowledge and reasoning. Jev leads among decision models; language models such as Claude Haiku 5.5 still deliver higher accuracy, but at much higher cost and latency Decision models deliver structured answers or classifications with associated probabilities, offering faster responses and lower costs than using language models. The Artificial Analysis Decision Index v1.0 compares the capability, cost, and speed of decision models and also compares to language models. Benchmarks in the Artificial Analysis Decision Index v1.0: ➤ AA-Omniscience-Decision: A private dataset of 500 questions adapted from the proprietary AA-Omniscience benchmark, measuring decision models’ knowledge and tendency to hallucinate ➤ HLE-MultiChoice: 513 text-only multiple-choice questions from Humanity’s Last Exam, testing decision models’ reasoning capabilities ➤ When2Call: A tool-calling benchmark from NVIDIA, testing whether models can correctly decide to call a tool, request clarification, decline to answer, or respond directly, evaluating capabilities relevant to agentic workflows and tool routing ➤ WildGuardTest: A safety benchmark evaluating models’ classification capabilities across harmful user requests, assistant refusals, and harmful responses. We use a subset of 835 of the original 1,725 test rows where all three human annotators agreed on all three labels, after excluding questions that consistently trigger model refusals Key results: ➤ Jev 1.13 leads the Decision Index, and sits on the Capability vs. Cost Pareto frontier: Jev 1.13 from @typesafeai scores 58 on the Artificial Analysis Decision Index, ahead of GLiDE (54) from @fastinoAI and Microsoft-Decision-1 (54) from @Microsoft. Decider v1.1 27B from @perplexity_ai has the lowest Cost per 1M Tasks at $17, with Microsoft-Decision-1 ($31) and Jev 1.13 ($42) also on the Capability vs. Cost Pareto frontier. Celeris-1 Decision has the fastest Time per Task at 0.06 seconds, more than 2x faster than Jev 1.13 (0.13 seconds per task) ➤ Decision models are poorly calibrated on knowledge and reasoning tasks: Across almost all probability levels for chosen answers, accuracy falls below that probability for all decision models. This gap is largest on AA-Omniscience-Decision and HLE-MultiChoice, while most models are relatively well-calibrated on WildGuardTest (classification) and When2Call (tool calling) ➤ Decision models offer significant speed and cost advantages over language models, but trail on knowledge and reasoning: On classification and tool-calling tasks, leading decision models perform comparably to language models while being significantly faster and cheaper. However, language models substantially outperform on knowledge and reasoning. Claude Haiku 5.5 (Low) scores 61 on the Decision Index, 3 points ahead of Jev 1.13, but takes 1.4 seconds per task vs. 0.13 seconds (11x slower) and costs $184 per 1M Tasks vs. $42 for Jev 1.13 (4.4x more expensive) We will continue to develop the Artificial Analysis Decision Index to help developers and companies better understand when and how to use decision models.
60
49
21
807
57,112
Next experiment on 5090: Qwen3-4B-Instruct-2507 with maxxed total throughput worker count. Simple skill frontend to chunk large inputs. 1000tok/s log and transcript summarizer as a tool for haiku workers. (In theory)
68
I got a DGX spark to run local models and it was fun. But the reality is most people are reporting tok/s numbers using unrealistically small contexts. Qwen 3.8 Flash Next with 8 workers and 96k compaction limit is around 9 tok/s/worker, and smaller worker counts don't scale well.
2
1
287
I plan to use my 5090 and Spark for experiments with fine tuning and other ML projects. And the spark went up $2k+ since I bought it so it's fine. But if you think you're going to save money on local AI I would think again. There are plenty of other reasons to go local,
1
1
28
But cost+quality is not one of them.
17
Legacy companies that embrace API access and scraping, while their competitors block and hide data access, will have a serious advantage in the AI era. I don't want to use your website. I want my agent to.
92
Astra seems over-aligned.
1
1
89
Astra is pretty good, but a default limit of 4 subagents was not a welcome surprise.
94
Going from Fable 5 to Fable 5.1, 5.1 is autocompacting at 150k tokens, just constantly compacting. Back to 5 for now.
1
80
Jake Jennings retweeted
hey all, I’m getting back into the job market.. I’m talking with a couple of co’s but interested in talking with more. I’m also open to medium/long-term freelance if that’s what you need 17 yrs of iOS/Mac dev (5 years at Apple), shipped dozens of apps, love hard problems
8
20
1
93
7,768