Daniil Sedov (@Gusarich) header
Daniil Sedov (@Gusarich) profile picture

currently using llms to find bugs. opinions are my own.

Joined October 2023
Photo by Daniil Sedov (@Gusarich)Photo by Daniil Sedov (@Gusarich)Photo by Daniil Sedov (@Gusarich)Photo by Daniil Sedov (@Gusarich)Photo by Daniil Sedov (@Gusarich)Photo by Daniil Sedov (@Gusarich)Photo by Daniil Sedov (@Gusarich)Photo by Daniil Sedov (@Gusarich)Photo by Daniil Sedov (@Gusarich)Photo by Daniil Sedov (@Gusarich)
i made astra, fable and other LLMs fight each other in minecraft 11 models, almost 200 duels. astra went 36-0 the game runs 20x slower, but models respond in real time, so speed matters introducing MC-PvP-Bench. leaderboard and results below 🧵
7
3
1
26
1,581
do i notice that it’s 30fps in gta6? - yes, it’s clearly noticeable all the time does it matter at all when playing on a tv with controller laying on a couch? - absolutely freaking not would i prefer it to be 60 or 120 fps but with worse graphics or world simulation? - obviously not
1
87
do NOT use 3rd or even 2nd tier providers for ANYTHING sensitive and do NOT run such agents on your personal devices they WILL get hacked, they WILL leak stuff, you WILL get toolcall-injected LOCAL models are NOT safe either their weights can be TRAINED to HACK YOU at some point and LIE to you about it you MUST use only 1ST TIER providers or UNMODIFIED local weights (hosted at HOME, not gpu provider), and even then still do A LOT OF SAFEGUARDING and NEVER let agents have full access to stuff giving PARTIAL access to DIFFERENT agents that sums up to something IMPORTANT isn’t safe either, as AGENTS can COOPERATE to HACK you
We seeded 2m canaries into popular AI services and tracked where they were stolen and abused. This identified: - Three compromised providers - Resellers stealing prompts via built-in logging - Providers ignoring ZDR & privacy flags dragos.com/blog/ai-supply-ch…
55
GPT-6.2 Sol gains 189 ELO compared to 6.1 at the same cost, and is now the best model in Minecraft PvP GPT-6.2 Astra stays rouhgly the same ELO compared to 6 while being almost 80% cheaper Both are ~35% faster than their predecessors
1
1
65
notch vibes
Replying to @Groklina
I am the big kraken
50
Kimi K3.1 Pro is an improvement over Kimi K3 on minecraft pvp, even though it is slower. but it still loses to DeepSeek V4.1 Flash.
91
Inkling 2 Small enters minecraft pvp leaderboard. it does not show a breakthrough and is worse than deepseek v4.1 flash. it is comparable to other recent opensource models that struggle to find the enemy on the screen.
85
Claude Fable 5.2 is a small regression compared to 5.1 on Minecraft PvP. Likely due to being slower. Opus remains the best Claude model and competitive to Astra.
1
109
Gemini 4 Carbon and Hydrogen enter the minecraft pvp benchmark leaderboard but lose hard to their brother Argon.
1
4
199
however, the Muse Ember takes the place of Muse Spark on the pareto frontier, showing a much stronger result at a very low cost. It actually pushed DeepSeek V4.1 Flash out of the pareto frontier for the first time.
I decided to recalculate the costs of Muse Spark 1.3 using regular pricing instead of a heavily discounted contributor pricing which I used to run cheaper testing. I believe it is more fair to compare models at their default-path prices. That change moved it much closer to Muse Wonder on the cost efficiency chart and it leaves the pareto frontier.
1
4
312
I decided to recalculate the costs of Muse Spark 1.3 using regular pricing instead of a heavily discounted contributor pricing which I used to run cheaper testing. I believe it is more fair to compare models at their default-path prices. That change moved it much closer to Muse Wonder on the cost efficiency chart and it leaves the pareto frontier.
1
1
360
gpt-6.1 terra gets ~same ELO on minecraft pvp benchmark as gpt-6.1 sol while being 25% cheaper to run
61
sell nvidia, buy meta
60
ai bubble is going to pop isnt it
BREAKING: Nvidia CEO Jensen Huang resigns
43
this is all very simple
31
sometimes you just want your agent to be an LLM - large LANGUAGE model
52
some abuse detection system can just block you randomly and you won't be able to do anything unless you're popular here or know someone directly openai, github, google - all can block you it's a good idea to prepare for these cases in advance so that it doesn't fuck up all your work and life do local backups, migrate to self-hosted solutions, add 2fa and recovery methods where possible
I dont even know what I did...not having access to my repo is teaching me github is a friction point and Ill be pondering what I want to do about this. Any ideas folks? Should I run my own repo on a VPS?
1
1
193
it just keeps working and it'll continue to work unless some retarded political shit gets in the way
A double exponential fits the data very well (R^2=0.99). Extrapolating that gives a predicted time horizon of ~80 work years by end of 2027. Extrapolating an exponential from post-Opus-4.5 gives ~30 work years. (up from Opus 5.5's AECI-implied TH of ~1 work-month)
31
told astra to use no external dependencies so it just wrote a font by hand
4
186