Assistant Professor @FollowStevens (2024-) PhD @UofT, @VectorInst Areas: #NLProc #Explainable #AI
Hoboken, New Jersey
Joined January 2014
- Tweets355
- Following614
- Followers852
- Likes1K
Zining Zhu retweeted
I interviewed ~2 dozen researchers, engineers, research / engineer managers etc. asking them how they / their teams use coding agents in their work. Then I sprinkled it with my opinions and hot takes and wrote a substack post (link in the next tweet, I'd be delighted if you subscribe; of course, the post is free and any future post will be free too!) trying to triangulate the current reality of software engineering / industry research as opposed to what gets peddled on social media.
TL;DR (3. and 4. are the most important!):
1. Models, harnesses, workflows: don't bother with fancy workflows, they don't really work. Use stock Claude Code / Codex (or Copilot with Claude / GPT models if you are a Microsoft stan), perhaps with subagents in some very special cases. Write a design doc, then plan a change, then implement it, then merge. Wash, rinse, repeat. Don't get swayed by people online who are trying to sell you snake oil in the form of their new harness or workflow. None of this stuff works (in general).
2. Correctness/tests/reading or not reading code: it's OK not to read code if you do one of the following three things. a) One-off stuff / demos, which you need to work on exactly one input. b) Testable / verifiable stuff, e.g., GPU kernels. c) Stuff, where it's OK to be "directionally correct" (ML / data work / research code). In other cases, especially if you require any degree of maintainability / reliability or your code is customer-facing, reading code is still necessary. Reading tests and steering the model to write better tests is a good idea too. It's often good to use a different model to review the code that the other model wrote. Even if you approve the PRs without changing them, reading still helps for future more productive prompting.
3. Team dynamics: if we are not careful, agents can really undermine team dynamics. Two biggest problems are "slop infestation", where over time all PRs / emails / Slack messages end up being unreadable slop, and "overbuilding", where we are building and shipping useless stuff to get recognition or as a form of procrastination. Other, less severe but still important problems include: reduced communication, being secretive about ideas, decreased diversity of thinking, mode collapse of the ideas, trusting LLMs more than people.
4. Call for action: we really need to be careful with preserving team work and collaboration. We still need to "form packs to hunt mammoths", but with coding agents this becomes increasingly harder. Overall, many of the agents-induced problems stem in myopic short-term thinking of the sort "we have a couple of years left until our jobs are fully automated, so I can as well extract as much value as possible, everyone else be damned". While somewhat understandable, I think it’s important to resist these urges, if nothing else, for your own sanity’s sake.
[Full disclosure: until 2 months ago, I worked at OpenAI and my work directly contributed to Codex, I'm a big fan of coding agents and quite strongly AI-pilled, but there is little I hate more than fearmongering, gaslighting, lies and rage-bait that are ubiquitous in this space, unfortunately, hence spending time on all of this.]
If you are at #EMNLP 2026 please check out our three papers at the Main Conference and Findings!
Our papers tackle problems across LLM evaluation, text provenance, and intermediate reasoning:
Using a medical benchmark, we show how explicit reasoning-path supervision can improve reasoning fidelity.
With Zhishan Yuan, Tiago Mateus Moreira de Almeida, Shudong Hao, Nickolas Elias Dawlabani, Ghazia Pervaiz, Yue Ning aclanthology.org/2026.findin…
Zining Zhu retweeted
Brilliant idea! Next up: Apple randomly reboots your Mac if you're building competing tech, Gmail silently edits your email if you mention rival platforms, and Tesla Autopilot swerves if it detects you're working on self-driving cars.
All in the name of safety, of course. Because malicious actors controlling the world’s operating systems, inboxes and cars would be extremely dangerous!
One single score can almost never tell {estimated AI participation extent, type of AI works, detector confidence, etc.} at the same time. Humans using predictor outputs for decisions should be more careful.
NeurIPS 2026 just desk-rejected hundreds of papers because an AI detector said they were AI-written (blog.neurips.cc/2026/06/02/a…).
I understand why they did it. Reviewer time is scarce, and a flood of slop submissions is a real threat to peer review.
But here's where it gets tricky. The Position Paper Track requires papers be "substantially human-written," so I ran one of my own through the same detector they used. It flagged 73% as AI-written — on pretty weak evidence (see screenshots).
The funny part? It's actually closer to 99%. I only hand-edited a few sentences. But the idea, the literature review, the positioning, the data analysis ... all human. The AI wrote the text, across many rounds of back-and-forth with me.
It's like coding: most code today is generated almost entirely by AI. Do we "reject" those systems too?
To me, whether the words were typed by a human matters less than the substance behind them.
What do you think?
PS. I ran this very post through the detector too. It came back 100% AI-generated — even though it took me almost 30 minutes to write, and I think it carries my thinking and is worth reading.
#AI #PeerReview #NeurIPS #AIDetection #AcademicResearch #MachineLearning #ResearchIntegrity #AIEthics #ScholarlyPublishing #AIWriting
Zining Zhu retweeted
Qwen first release on interpretability (qwen scope) is very interesting
they use SAE features to identify what causes repetition in model outputs, then use steering to manufacture a "bad" rollout where the model repeats a lot. this gives RL a clear negative signal to learn from, since repetition barely shows up in normal rollouts so the model never gets punished for it
they also use SAE features as a fingerprint for benchmarks, you look at which features each benchmark activates and compare overlap. lets you find redundancy inside a benchmark and across benchmarks without running any model. for instance 63% of GSM8K features are in MATH but only 10% the other way
Zining Zhu retweeted
AutoResearch saves 90% time for research solution.
But in research, 90% time is spent for the right question.
Zining Zhu retweeted
Announcing new ARENA material: 8 new exercise sets on alignment science, interpretability & AI safety - each containing 1-2 days of structured, hands-on content replicating key papers in the field.
All open source on a public GitHub, and available for study. Here's what's in it:
Zining Zhu retweeted
This is absolutely shameful. Agents of a federal agency unnecessarily escalating, and then executing a defenseless citizen whose offense appears to be using his cell phone camera. Every person regardless of political affiliation should be denouncing this.
Zining Zhu retweeted
TIL that ACL 2026's theme track is "Explainability of NLP Models"! 😮🤩 @aclmeeting
Zining Zhu retweeted
❌ LMs can’t express all next-token distributions — embedding-space limits constrain what’s possible.
🤔 But have you wondered which ones are hardest to elicit?
Our #EMNLP2025 paper finds medium-entropy distributions (without outliers) are the toughest.
🧩arxiv.org/abs/2505.12244
Zining Zhu retweeted
#COLM2025 Our last invited talk if you are still around: Prof. Zining Zhu is presenting "Improving LLM reasoning with mechanistic insights" @zhuzining
Zining Zhu retweeted
New paper: What happens when an LLM reasons?
We created methods to interpret reasoning steps & their connections: resampling CoT, attention analysis, & suppressing attention
We discover thought anchors: key steps shaping everything else. Check our tool & unpack CoT yourself 🧵
Reviewers should perhaps be prohibited from changing the scores they give on the day of seeing the scores of their own papers submitted to @ReviewAcl.