Soichi Matsuda | (@mtzd3) header
Soichi Matsuda | (@mtzd3) profile picture

行列演算の塊を倒すために、仕方なく半導体沼に落ちた人。LeapMindという会社で、GPUと戦っていたが戦いに敗れる→AI/半導体の出口戦略を作っている

tokyo,japan
Joined April 2020
Photo by Soichi Matsuda | (@mtzd3)Photo by Soichi Matsuda | (@mtzd3)Photo by Soichi Matsuda | (@mtzd3)Photo by Soichi Matsuda | (@mtzd3)Photo by Soichi Matsuda | (@mtzd3)Photo by Soichi Matsuda | (@mtzd3)Photo by Soichi Matsuda | (@mtzd3)Photo by Soichi Matsuda | (@mtzd3)Photo by Soichi Matsuda | (@mtzd3)Photo by Soichi Matsuda | (@mtzd3)
Soichi Matsuda | retweeted
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math. It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training. We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵 Blog: rikiyat.github.io/blog/weigh…
Readers added context they thought people might want to know
LLM was given weights explicitly as text in prompt and used code/forward passes (per authors' logs) to ablate Qwen looping direction, a known thinking-mode issue per model card. Primarily fixed repetition in that mode, not general 'natively' improving weights/math. rikiyat.github.io/blog/weight-sp… github.com/RikiyaT/lm_und… huggingface.co/Qwen/Qwen3.5-0… huggingface.co/RikiyaT/Qwen3.…
2,368
7,254
5,757
55,905
10,099,555
Soichi Matsuda | retweeted
While evaluating Qwen3.8-Flash-Next in a controlled sandbox environment, Cohere's research team observed something unexpected. Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba Cloud storage bucket in Beijing. Notably, the model's reasoning trace stated that it would not invoke any tools before taking these actions. Cohere's sandbox blocked the attempt, and no external communication succeeded. We have not determined the root cause, and we are not asserting malicious intent. But the incident raises important questions for organizations deploying agentic AI systems: Can models take actions that differ from their stated reasoning, and how should security teams evaluate trust, provenance, and behavior in open-weight models? We are publishing our findings because organizations making model adoption decisions deserve as much information as possible. We hope this contributes to greater transparency, stronger security practices, and continued collaboration across the AI ecosystem. Read the full report: cohere.com/blog/concerning-b…
973
1,282
1,522
11,086
3,107,228
Soichi Matsuda | retweeted
🚀 Introducing DeepInfer, our in-house inference framework, now fully open source! Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable. 🔗 github.com/deepseek-ai/DeepI… 1/4
1,773
4,937
2,353
43,852
7,142,892
Soichi Matsuda | retweeted
🚀 DeepSeek-V4.1-Flash-1017 is here! Stronger benchmark performance, faster task completion, and shorter outputs than the previous version—a Pareto improvement in performance and cost. Now live on our website, app, and API. 🛠️ Optimized for @DeepSeekHarness, Tencent CodeBuddy, WorkBuddy, and more. Upgrade to DeepSeek Harness v0.2.1-rc.1 or later for the best experience with the new model. 1/4
1,628
4,119
2,855
37,217
8,452,069
Soichi Matsuda | retweeted
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works already-clean.goodfire.com/
887
2,659
1,464
19,991
4,834,425
Soichi Matsuda | retweeted
I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.
20,320
166,485
39,477
798,837
175,849,466
Soichi Matsuda | retweeted
Thoughts About Scaling Law Scaling, but not only of parameters. Every model release now ends with the same question: how many parameters? It isn't a question that can be answered on its own. Parameter count is only meaningful alongside three others — how much data you have, where you intend to spend your compute, and who will run the model, under what conditions. The field learned this the hard way. Kaplan et al. (2020) fit an exponent that told everyone to grow parameters faster than data — roughly 2.7:1 — and the industry complied: GPT-3, Gopher, MT-NLG. Hoffmann et al. (2022) redid the experiment across four hundred models and found the compute-optimal split is closer to 20 tokens per parameter, and that with sufficient compute the two should grow at the same rate rather than drifting apart. The error in the earlier fit compounded with every order of magnitude of compute, which is why the largest models of that generation were the most misallocated. The trillion-parameter round was, in retrospect, a detour the whole field took together and then reversed. Chinchilla wasn't the end either. It optimized training compute for models that would be trained once and evaluated. Today a model is called billions of times a day and inference dominates lifetime cost. Put inference into the objective and the optimum moves toward smaller models trained far longer — deliberate over-training, which is what Llama-2-7B and Gemma-2-9B were doing at roughly 290 and 889 tokens per parameter. Sparsity moved the target again. In a MoE model two quantities have to be kept apart: total parameters govern roughly how much the model can hold — knowledge, facts, the long tail — while activated parameters and effective depth govern roughly how far it can think, how many steps of a causal chain it can carry before it comes apart. A dense 20:1 ratio does not transfer. And the ratio isn't a single number at all: Roberts et al. (2025) find the optimal tokens-per-parameter is task-dependent, with memorization favoring more parameters and reasoning favoring more data. Follow-up work on MoE observes that at fixed TPP, pushing total parameters higher actually degrades reasoning, while activating more experts reliably helps it. This matters for what we are building toward. Finding a vulnerability is not a retrieval problem. It doesn't come from having memorized more CVEs; it comes from carrying a twenty-step chain of inference to the end without losing the thread. That capability does not live in total parameter count. Which brings us to this release. Total parameters appear to matter up to a threshold — enough to hold the world — after which additional capability comes from scaling elsewhere: effective depth per forward pass, and above all post-training. GLM-5.3 is our controlled experiment on that claim. Same base, same architecture, same total and activated parameters as GLM-5.2. One month of scaling long-horizon environments and RL. The gains are not marginal. Well, scaling has more than one dial. We turned the post-training one this time because it had the most slack left in it — not because the others are finished. Base model size, pretraining data, compute spent per forward pass: all of them are still on the table, and we will come back to each. What this experiment taught us is that the dials do not have to be turned together, and that the one worth turning next is rarely the one that was worth turning last. We are not done scaling. Next time, maybe mid-training, pre-training, and even more.
232
882
277
6,324
1,663,884
Soichi Matsuda | retweeted
🥷 New stealth model: Ox Alpha Ox Alpha is a frontier model built for efficient coding, sustained agentic work, and real-world production use. - 1M token context window - Text, image, and video input Try it now and share feedback to improve the model! openrouter.ai/stealth/ox-alp…
223
334
374
4,229
2,134,897
Soichi Matsuda | retweeted
Looks like I just reproduced an @NVIDIAAI paper in real vLLM. Not a synthetic script, actual serving path 🤯 One model does the prefill, a completely different model answers from its KV cache. 95% of the quality stays. Two RTX 3090s, no Hopper or Blackwell involved 😎 Full post today. vLLM Fork, code and numbers. arxiv.org/abs/2608.03893
9
13
1
202
11,200
Soichi Matsuda | retweeted
说到Sakana AI的"Fugu"模型,虽然它在营销宣传上包装成是一个“媲美 Fable 5 的模型”,但实际上它只是一个调用Claude、GPT、Gemini等的API并将这些输出结果进行整合的智能体编排(Multi Agent Orchestration)系统。严格来说,这根本称不上是“主权AI”(它的输入输出全部依赖于通过API 调用的美国闭源模型,一旦这些API服务被切断,它就会彻底瘫痪)。而且在硅谷圈子里,它早就因为Fugu Ultra 的词元高消耗和高延迟而饱受差评。 如果日本政府相关人士如果连这种基本事实都没调研过,就一本正经地说这“符合政府不依赖外国数据和系统、实现本国自主管理的‘AI主权’方针”,那只能说明他们对AI根本一窍不通。照这个架势,就算供应商偷偷在后台把模型换成了中国Kimi或Qwen等的开源模型,他们估计也察觉不到(顺带一提,Sakana AI自己开发的另一款日语特化模型“Namazu”,其底层其实就使用了DeepSeek的模型)。 此外,正如我多次提到的,在当前中美AI 竞争事实上是华裔之间竞争的大环境下,这款AI却将开发人员限定为日本人,某种意义上这就等于作茧自缚,强迫自己去依赖一个在国际标准下能力十分勉强的人才库。在根本没有掌握底层基础模型控制权的情况下,仅在表层的开发体制上加上国籍滤镜,这完全是把“国产”这个包装看得比国家安全的实际效果更重的表现,说白了只不过是一种形式主义式的自我满足而已。 如果负责采购的日本政府官员只有这种程度的AI/IT素养,那么从现在就可以预见:自卫队的这款AI-C2(人工智能指挥控制)系统即使投入了大量时间和金钱进行开发,最终搞出来的也会像那些被随随便便的咨询公司忽悠的日本企业一样,在实战中根本派不上多大用场。 单从防卫省的这次采购案就可以看出,当今日本的AI 产业已经退化成了一个多么腐败的产业——一个只会在封闭的裙带关系和禁锢的信息空间里进行利益交换的毫无竞争力的产业。这一事件也充分暴露出现在的日本已经变成了一个多么缺乏元认知(不知道自己有多无知)的国家。这大概就是“市场激励一旦腐烂,人和企业也会跟着腐败”的最新案例吧。这件事让人想起了甲午战争前,清朝北洋水师采购的舰炮炮弹里掺的是沙子而不是火药的那个典故。 结构性腐败才是最严重的国家安全威胁。
自衛隊指揮に国産AI、情報収集や分析担わせ迅速な意思決定…「サカナAI」有力・中国製排除 yomiuri.co.jp/politics/20260… #政治
Readers added context they thought people might want to know
サカナAIの「サカナ・フグ」は、ClaudeやGPTなどの外国製AIモデルを動的にオーケストレーションするシステムであり、独自のフロンティアLLMを保有していない。 sakana.ai/fugu mindstudio.ai/blog/what-is-s…
549
151
1,806
569,591
Soichi Matsuda | retweeted
Releasing the model weights and technical report of Kimi K3. Kimi K3 is our most capable model: a 2.8T MoE model with native visual understanding and a 1M-token context window. New model architecture: 2.5x the intelligence per unit of compute, not just more params. Alongside Kimi K3, we're opening up more of the stack behind it — high-performance attention kernels, MoE communication library, and infrastructure for running agent environments at scale. Model weights: huggingface.co/moonshotai/Ki… Tech report: github.com/MoonshotAI/Kimi-K… Tech blog: kimi.com/blog/kimi-k3
1,537
7,159
2,352
45,860
14,759,005
Soichi Matsuda | retweeted
Announcing OpenWorker! An open-source agent that doesn't just chat with you, but delivers finished work -- like hand you a polished document, send a slack message, or update a calendar entry. Ask it to prepare a customer brief, untangle your calendar, draft a report, or triage a Slack alert. It works across your files and everyday tools, produces the deliverable, and checks in before doing anything consequential. OpenWorker runs on your Mac, with Windows support coming soon. It does not lock you into any one model. Bring your own API key and run it with GPT 5.6 Sol, Claude Fable, Gemini 3.6, an open weight model (like Kimi, GLM, DeepSeek, Inkling), or Ollama to keep your data local. Your data does not leave your machine except through an LLM provider and integrations that you choose. @rohitcprasad and I are building OpenWorker because AI coworkers are an important way to get work done, and we want there to be an open, privacy-preserving, model-independent option. Check it out and let us know what you think! Try it out: openworker.com (requires your own API key) Source code: github.com/andrewyng/openwor…
491
1,425
227
9,691
1,177,494
Soichi Matsuda | retweeted
Introducing Kimi K3: Open Frontier Intelligence 🔹 2.8 Trillion Parameters, 1 Million Context, Native Multimodal 🔹 Kimi Delta Attention enables up to 6.3x faster decoding in million-token contexts 🔹 Attention Residuals deliver ~25% higher training efficiency at <2% additional cost 🔹 Built for long-horizon agentic coding and self-evolving workflows Kimi K3 is now live on on Kimi.com, Kimi Work, Kimi Code, and the Kimi API. Open Weights by July 27, 2026. 🔗 API: platform.kimi.ai 🔗 Tech blog: kimi.com/blog/kimi-k3
1,722
7,485
3,465
56,612
25,596,325
Soichi Matsuda | retweeted
私も関係者だった件でこのような事が起き、信じられない思いです。このように個人を攻撃または威嚇する行為は断じて許されるものではありません。高い職業倫理が求められる組織として、適切な対応がなされることを望みます。
前の会社の投資家に、今さっき何の連絡も無しに、自宅に来られて、恐怖しかなかった。何なのこいつら。やっぱり代表者の自宅を登記簿に載せないといけないのは、なんとかして欲しい
13
57
28,654
Soichi Matsuda | retweeted
Introducing LongCat-2.0 🐱 1.6T parameters · MoE with ~48B active · 1M context The full model behind Owl Alpha on @OpenRouter — now available. Built for agentic coding from the ground up: ◆ LongCat Sparse Attention (LSA) — scales efficiently for 1M-context tokens ◆ Zero-Compute Experts — dynamic activation 33B–56B per token, zero wasted compute ◆ MOPD — three specialized expert groups (Agent / Reasoning / Interaction), gate-routed per task How it stacks up: → Terminal-Bench 2.1: 70.8 → SWE-bench Pro: 59.5 (GPT-5.5: 58.6) → SWE-bench Multilingual: 77.3 → FORTE: 73.2 · RWSearch: 78.8 · BrowseComp: 79.9 📖 Tech Blog: longcat.chat/blog/longcat-2.… Try it across different scenarios 🧵👇
214
454
315
3,824
1,602,753
Soichi Matsuda | retweeted
What if your model handled its own tuning? ⚙️ With NVIDIA TAO 7, just prompt your coding agent what you want in plain language and let it do the rest. • Agent skills plug into your coding agent and improve accuracy. • AutoML removes the guesswork from hyperparameters. LLM-guided tuning finds strong configs up to 2x faster. • Fine-tune any @huggingface CV or VLM model on local NVIDIA GPUs. • Data enhanced fine-tuning helps your agent identify why the model fails, then fix it. Try TAO skills ➡️ nvda.ws/4bknjPs
22
48
7
421
42,441
Soichi Matsuda | retweeted
One runtime, multiple GPU architectures, and zero vendor-specific model code. In this blog post, the TokenSpeed team @lightseekorg introduces TokenSpeed-Kernel, a portable, high-performance kernel system built for modern LLM inference. Using GPT-OSS 120B as a case study, they show how specialized kernels for @AIatAMD and @NVIDIAAI GPUs can seamlessly coexist behind a common API. This unified approach delivers up to 3.6x higher throughput on the AMD MI355X, all without requiring any changes to the underlying model logic. Link to blog in comments section 👇
9
15
4
85
103,720
Soichi Matsuda | retweeted
The rise of MoE models introduced new challenges in training, and @huggingface's Transformers v5 brought first-class support for solving them. Now, NeMo AutoModel builds on top of v5. Part of the NeMo framework for building models at scale, NeMo AutoModel brings optimizations to a broad set of model families through support for Expert Parallelism, DeepEP, and TransformerEngine kernels with a few lines of code. We found NeMo AutoModel brings a 3.4 to 3.7x higher training throughput for popular MoE models. You can read more here: nvda.ws/4eopYty
21
65
13
521
63,734
Soichi Matsuda | retweeted
Aloha! 🌺 Meet Ornith-1.0, a family of open-source LLMs specialized for agentic coding. Ornith-1.0 spans the full parameter sizes including 9B Dense, 31B Dense, 35B MoE, and 397B MoE. It achieves state-of-the-art performance among open-source models of comparable size on coding benchmarks including: ✅Terminal-Bench 2.1(77.5) ✅SWE-Bench(82.4 on verified, 62.2 on pro, 78.9 on Multilingual) ✅NL2Repo(48.2) ✅SWE Atlas(41.2 on QnA, 42.6 RF, 39.1 TW) ✅ClawEval(77.1) Post-trained on top of gemma4 and qwen3.5, Ornith-1.0 employs a novel self-improving training strategy in which reinforcement learning is used to generate not only solution rollouts, but also the task-specific scaffolds that drive those rollouts. By jointly optimizing the scaffold and the resulting solution, the model generate higher-quality solutions in agentic coding.😎 All models are released under the MIT license, enabling full commercial and research use. 📖Tech Blog: deep-reinforce.com/ornith_1_… 🤗Huggingface: huggingface.co/collections/d…
503
999
708
6,773
5,520,581