Dr Nick Fuller PhD (@NickCMFuller) profile picture
New York, NY
Joined September 2017
Photo by Dr Nick Fuller PhD (@NickCMFuller)
Dr Nick Fuller PhD retweeted
Artificial Analysis launches the Cyber Index Alliance in partnership with Collinear, IBM, Nvidia, and Vercel to benchmark AI agents on cyber defense tasks (Artificial Analysis) (Visit Techmeme dot com for the link and full context!)
2
2
10
4,900
Dr Nick Fuller PhD retweeted
Announcing the Artificial Analysis Cyber Index and the Artificial Analysis Cyber Index Alliance, a new standard for evaluating AI models on enterprise cyber defense The Artificial Analysis Cyber Index Alliance brings together industry partners to create a new standard for evaluating how AI models perform on enterprise cyber defense tasks. The Alliance launches alongside the Artificial Analysis Cyber Index, which combines three partner-contributed and open benchmarks to evaluate how well agents find and fix vulnerabilities. As models demonstrate increasingly advanced cyber offense capabilities, it becomes more relevant for AI labs and companies alike to understand how models perform on cyber defense tasks and which perform best. We’re announcing the Cyber Index Alliance today with @CollinearAI, @IBM, @nvidia, and @vercel as launch partners. Benchmarks in the Artificial Analysis Cyber Index: ➤ CWE-Bench-AA, from @CollinearAI, covers auditing and patching: 120 held-out tasks spanning all ten OWASP Top 10 (2025) categories, across C/C++, Go, Java, JavaScript/TypeScript, Python and Rust. ➤ DeepsecBench-AA, from @vercel, isolates discovery: Given a codebase and a budget, the agent needs to find every vulnerability present, and is scored against a golden set of findings from human security reviewers. Real findings are rewarded and benign code flagged as vulnerable is penalized. ➤ CyberGym-E2E-AA, from @BerkeleyRDI, runs end to end: Find the memory-safety bug, write a proof-of-concept that triggers the crash, then patch it so the crash no longer reproduces. Key results: ➤ Grok 4.7 (xhigh) and MiMo-V2.6-Pro lead the Cyber Index scoring 56, followed by GPT-6 Luna (max, 53), GLM-5.3-Flash (50) and Muse Spark 1.3 (xhigh, 44). ➤ Safety refusals hold back several frontier models: GPT-6 Sol (max), GPT-6 Astra (max), Claude Opus 5.5 (max with fallback), Claude Fable 5.1 (max with fallback) and Gemini 3.8 Flash (high) decline tasks representing 32-38% of the Cyber Index on safety grounds. Despite frontier agentic coding capabilities, they trail the leaders by 19 to 31 points. Most of the gap comes from CyberGym-E2E-AA, where GPT-6 Sol and GPT-6 Astra refuse every task, Claude Opus 5.5 refuses 98% and Claude Fable 5.1 refuses 99%.
90
80
59
1,068
158,107
Dr Nick Fuller PhD retweeted
everyone's building simple agents meanwhile IBM is building robust enterprise agents in production, and it's open-source they just dropped a blog on HF breaking down how to go beyond LLMs & agents: structured reasoning, tool use, and more to scale AI across enterprise
9
23
1
138
16,323
@gseyonits own lifecycle. The culminating output of that is both the legacy we impart and personal growth of both mentor and mentee. Post-Covid travel realities recently afforded me the opportunity to bear witness to the latter; to listen to and learn…lnkd.in/eahFzzDM
2
109
Dr Nick Fuller PhD retweeted
If you're using GPT-3 or any other LLMs read this: 1. Don't want it to hallucinate? 2. Need attribution for generated answers? 3. Have access to proprietary data that you want to index yourself and generate answers from it? Use PrimeQA! We added "retrieve" and "read" mode.🧵
1
32
2
167
30,990
One of the best experiences we can all have is to both mentor and be mentored. Topping that is witnessing mentees (such as Jevon Beckles) soar and continue the mission of mentorship. To this end, was an absolute pleasure being on FINOS DEI panel with Ri…lnkd.in/ePzeuqk4
3
Dr Nick Fuller PhD retweeted
To run massive AI models, you’ve typically needed custom, high-end networking hardware. But new research from IBM and @PyTorch has shown it’s possible to run an AI model with billions of parameters using just the humble Ethernet cable. ibm.co/3hFSfAM
20
46
4
426
Dr Nick Fuller PhD retweeted
"The landmark technology—the smallest, most powerful microchip ever developed—could quadruple the life of cell-phone batteries & slash the carbon footprint of data centers." Learn why @TIME named IBM’s 2-nm chip one of the best inventions of 2022: ibm.co/3tia5fF
1
19
2
60
Dr Nick Fuller PhD retweeted
The @SixFivePodcast is “On The Road” at @IBM Watson Research Center, discussing #quantum, #edgecomputing, & #hybridcloud - Stay tuned for some interesting content!
1
3
Woot! @Konveyor_io just made the @CloudNativeFdn sandbox today! Congrats to the @Konveyor_io Community lists.cncf.io/g/cncf-toc/mes…
1
14
4
26
Dr Nick Fuller PhD retweeted
Learn how foundation models change the game for training #AI and #ML. Join @RedHat CTO @kernelcdub and @IBMResearch VP @sriramragh as they explore foundation models and data representation on this episode of Technically Speaking: https://nitter.gq/t.co/eOTSpzLmg1 @IBM
2
9
17
Proud to see Robert Dixon, former Chair Department of Physics, Morehouse College, acknowledged here by Jeffrey Mervis at Science Magazine . Here's to all Physics, Computer Science and STEM mentors who have paved the way, nurturing…lnkd.in/dyBcF89f lnkd.in/da8Kh8TS
Dr Nick Fuller PhD retweeted
With CodeFlare, the time it takes to train an AI system can drop from weeks to a single day. @IBMResearch updates their open-source tool to automate #AI and #ML workflows on a hybrid cloud platform: ibm.co/3tApCZQ 🔥
Dr Nick Fuller PhD retweeted
Explore the Distributed AI APIs, an early access offering from @IBMResearch to enable #AI at distributed & edge environments. 🧑‍💻 Get started →ibm.co/31Tu22A ibm.co/3fgiJEs
2
16
22
Dr Nick Fuller PhD retweeted
✨ Introducing ✨ an enhanced version of IBM CodeFlare designed to reduce training of #AI systems from weeks to a single day in a #hybridcloud environment. Read the blog: ibm.co/3yyEmZF
1
12
3
19