Josh Herritz (@jherritz) header
Josh Herritz (@jherritz) profile picture

Obsolete Data Wrangler | AI, data, startups, code, economic development, YIMBY, urbanism, current events and running

Madison, WI
Joined June 2008
Josh Herritz retweeted
Classic JVL "How anyone could live through the last decade and not change his analysis of American political life is beyond me. If you always knew that the Constitution was run on the honor system, and that a third of the country wants fascism, then good for you. Speaking only for myself: I did not." thebulwark.com/p/the-axis-of…
23
103
3
589
30,539
Josh Herritz retweeted
This is the first 100% we've seen on v3 The way we've measured models GPT-1 through GPT 6.2 won't hold up for GPT-7+. We need a categorically new benchmark Three ways to create benchmarks: 1. Out-of-distribution tasks A models knowledge coverage looks like swiss cheese (not exactly, but you get the idea). You can always find gaps in knowledge as you niche your topic. Ex: Animals identification > Mammals > Dogs > Herding dogs > Border collies This is the *easiest* way to create a new benchmark, it's also the *least* robust. Depending how much the labs care, you're only 2-3 months of model releases away from your swiss cheese gap filling in. 2. Increase task difficulty Take an existing domain of tasks which models saturate and make the problems harder. Hire PhDs and experts to come up with problems. This is operationally intensive and harder to execute. You want to avoid cheap difficulty (a larger maze) and lean into rich difficulty (more reasoning required). 3. Measure the frontier This is the highest signal (imo) way to test models, it'll be the defining benchmark for post-AGI systems. Instead of measuring performance on a static set of tasks, measure the rate at which models push the frontier of knowledge forward. The ultimate environment is reality/physics. Net new knowledge here is proof of innovation. However, physics is hard to benchmark with, sythenetic environments will do ok here in the short term. To be clear, this is an incredibly high bar to hold models to. Maybe the highest? All RSI, prosperity, advanced capability questions are downstream of AI's ability to create net new knowledge. Once you have a system that can autonomously create new knowledge in any domain (like human as a collective can) then every other domain is a harness/training session away. TLDR: Static task completion is no longer an adequate way to measure frontier models. Rate of learning is the last hurdle. Net new knowledge (what humans don't know) is the most robust setting to measure this.
GPT-6.2 Astra from @OpenAI on ARC-AGI (Verified): - ARC-AGI-3: 100.0% (SOTA), $4.1K (provider adapter), 70.3%, $24.2K (standard) - ARC-AGI-2: 95.8% (SOTA), $0.50/task - ARC-AGI-1: 98.5%, $0.17/task GPT-6.2 Astra is the first model to achieve a 100% score on ARC-AGI-3.
21
8
5
133
6,923
Josh Herritz retweeted
Sources: the US SEC is investigating Nvidia, including whether the company's disclosures misled investors (Wall Street Journal) (Visit Techmeme dot com for the link and full context!)
1
2
4
2,301
Josh Herritz retweeted
Russia no longer can enter or leave the Azov Sea with ships The Kerch Bridge with its arch collapsed into the water, blocking the 200 meter passage way for ships The Azov Sea is important for Russia’s economy, ~25% of Russia’s total grain exports move through the Azov Sea
334
3,358
235
13,989
363,179
Josh Herritz retweeted
NEW!! Senate Armed Services Chair Roger Wicker just called Putin “the most inept military leader of our century,” saying Russia’s mounting casualties and shrinking battlefield gains show Ukraine is turning the tide of the war. 1/2
5
135
3
484
9,353
Josh Herritz retweeted
A HISTORIC WEEK FOR THE AI INDUSTRY Between the Anthropic IPO and $NVDA (polar opposite companies in many ways), this week is huge. And as noted by @qcapital2020, $NVDA now trades at a lower PE than Campbell's soup. The last time this was so? The month before ChatGPT's release.
11
8
91
15,285
Josh Herritz retweeted
💥 Visualization of FP-5 "Flamingo" cruise missile strikes during the combined attack on the Kerch Bridge on the night of November 17. Two missiles hit the bridge. One struck the road span arch, collapsing the structure along with the roadway. The second hit the railway section, causing structural damage. To be continued…
49
342
30
1,665
60,818
Josh Herritz retweeted
Ha. When I had Astra 6.2 make a puzzle spy game using any setting it wanted, it set the game in "Bellwether Station." I thought it was an odd choice but didn't realize how deep a pattern this was. What a weird attractor for the model.
I was saving this for my video but I'll spoil it now. Ben and I both made new levels in games using 6.2 Astra during the early testing. He did Majora's Mask, I did Outer Wilds. We both prompted ourselves, but it was along the lines of "make a creative new level for [game]" We both got water levels with music themes, specifically around bells. Both were named "Bellwether". This model regresses to the mean like nothing I've ever seen. Go ask it to make a new level for a game you love, lmk if it chooses "bellwether" as the name.
47
13
5
668
33,085
Josh Herritz retweeted
Mercedes’ lobbying against the connected vehicle security act probably not to make much progress…
Today, Select Committee Chairman @RepMoolenaar and Ranking Member @RepRoKhanna @CmteOnCCPDems launched a bipartisan investigation into Mercedes-Benz’s ownership ties to China and its technology partnerships with China-based companies. The committee is seeking information about China’s influence over Mercedes-Benz’s strategic decisions, technology sharing, Chinese components used in vehicles manufactured or sold in the United States, and the company’s engagement with Chinese government officials. reuters.com/business/autos-t…
1
2
13
2,587
Josh Herritz retweeted
Ukraine's former DM Fedorov: November will be the first month in which we eliminate 40-50 thousand Russians. As defence minister my goal was to eliminate 50,000 Russians a month. People are the one resource Russia cannot replace. 1/
3
36
1
382
13,369
Josh Herritz retweeted
Fedorov: Ukraine has intercepted 95% of Russian drones in our sky, and we have invented new equipment against them. We need only three to five more types of equipment. Then Putin will have no cards left to continue the war. 2/
2
5
69
1,405
They're probably wishing Jensen had self-regulated a little better.
NEW: SEC Is Probing Nvidia Amid Departure of CEO Jensen Huang The civil investigation adds to the crisis facing Nvidia, whose shares have fallen more than 10% since Huang was ousted. w/ @davidamichaels and @RWhelanWSJ for @WSJ More: wsj.com/business/sec-is-prob…
51
Josh Herritz retweeted
QAnon but for tech billionaires
40
134
13
1,089
40,076
Josh Herritz retweeted
10/10, no notes.
33
123
12
647
21,491
Josh Herritz retweeted
New neurons research: introducing Neuron Communities. We discovered communities of neurons that work together inside LLMs. They’re surprisingly small, but powerful – just a few neurons work together to perform common behaviors, like arithmetic, ALL CAPS, or recalling a date. Work led by research fellow @_igorshilov:
16
72
15
581
20,476
Josh Herritz retweeted
Trump has handed Putin at least 10 “gifts” since returning to office — and received zero concessions in return. The latest: Starshield was throttled just as Kushner and Witkoff arrived in Moscow — Marc Champion for Bloomberg. 1/
10
210
6
620
18,797
Josh Herritz retweeted
I'm trying to imagine what the board might say if it reinstates Jensen. "Our CEO had sex with a customer and one of his direct reports. We've decided those offenses are nowhere near enough to fire him. Yes, we know the customer is being investigated as a possible Chinese intelligence asset, suspected of leading a shell company in Singapore to provide Chinese access to our chips that aren't permitted for sale there. Furthermore, I know some of you might have concerns about possible collusion with the Commerce Secretary. Perhaps you're even worried the national security of the United States was compromised. We don't share any of those concerns. Welcome back, Jensen!" wsj.com/tech/jensen-huang-st…
150
68
17
445
22,386
Josh Herritz retweeted
CNN’s live feed was cut in China when my report revealed details of the investigation into Alice Huang’s suspected ties to China’s spy agency, claims Beijing denies. – at National Capital Region, Republic of the Philippines
68
448
38
986
98,085
Josh Herritz retweeted
An illustration of both the power and issues with Astra 6.2 in a way that might be accessible: I asked it to create a game in Blender and Unity where you are the pilot of a Coast Guard cutter who travelled back in time to the War of Independence to stop the British invasion of New York. It made incredible 3D models and a working game with no errors - its even fun to play! But Astra got obsessed with random ideas that didn't make a lot of sense and built them out without asking along with following my instructions: you could put down false buoys to lure the ship in the wrong direction (like the British would understand a modern buoy?) or you could persuade the British ships to go the wrong direction by talking to them, or that you would spend your time rescuing sailors from other parts of the harbor. It didn't even initially have a shooting mechanism until I pointed it out. This is all kind of minor stuff individually, but it was frustrating to have the AI constantly do its own things without telling me, given its instincts were often off, and then have to redirect it constantly. Other models, including GPT-6 Asta, didn't require the same struggle to control and operate. And these same issues show up in academic research and other more serious use cases.
24
9
4
273
22,040
Josh Herritz retweeted
Delusional. Take Jensen's name out of the equation. If a CEO materially misleads the board regarding personal relationships, obstructs succession planning & presides over illegal business practices (wittingly or unwittingly), no director in their right mind would reinstate that CEO. Ever.
Over the past week, I’ve been overwhelmed by the love and support of friends, industry colleagues, and people I've never met. Thank you to everyone who has shown kindness to my family and me. It means the world to me. I came to the United States 53 years ago, and becoming an American citizen is one of the greatest gifts and proudest moments of my life. NVIDIA has been my life’s work. Building it into the company it is today alongside my colleagues has been a privilege and an honor. Nowhere but in America could a company like NVIDIA be possible. I love our country, and I love NVIDIA. I did not live up to the standards expected of everyone at NVIDIA, and I have made serious mistakes in my personal life. I created problems for the Board that I deeply regret. I own my mistakes completely, and I am committed to cooperating with the Board’s independent investigation. Since my departure, I’ve seen an outpouring of calls from NVIDIA’s employees, investors, business partners, and customers for me to return as CEO. Many continue to reach out to me directly as the company navigates complex technology, market, competitive, and geopolitical dynamics. NVIDIA is a once-in-a-generation company. If the Board of Directors decides that my return is in the best interest of our shareholders, our employees, the company, and America, I would be honored to answer these calls and return as CEO. Thank you again to everyone who has shared their good wishes. I am profoundly humbled by your support. To my family: I love you, and I will spend the rest of my life being worthy of yours. Jensen
1,404
166
90
1,379
170,215