Devin Pellegrino retweeted
I made this with suno.ai and agents from lexidecktechnologies.com today. This album has a varied electronica sound with chillingly ethereal female vocals!
suno.com/playlist/0af809ed-8…
I hope you enjoy it!
(Made with AI)
Devin Pellegrino retweeted
Exploring #LexideckTechnologies and #NerorityAI has been a journey of discovery! Check out their Discords for a deep dive into AI!
Shout outs to @NerorityAI, @missjenny, and @uni_com_ai
👾 Lexideck Technologies: discord.gg/BmHbCC5nQb
🤖 NerorityAI: discord.gg/jBKjsqA6pb
#AICommunity #PromptEngineering #Innovation
Devin Pellegrino retweeted
“The fact that [transformer neural nets] model language is probably one of the biggest discoveries in history. That you can learn language by just predicting the next word with a Markov chain—that’s just shocking to me,” Mikhail Belkin says. By @strwbilly.
technologyreview.com/2024/03…
Devin Pellegrino retweeted
Fuck cynicism. Humans are awesome. There is no problem we can't solve. We will find a way.
Devin Pellegrino retweeted
this is the most interesting year in human history, except for all future years
Devin Pellegrino retweeted
# automating software engineering
In my mind, automating software engineering will look similar to automating driving. E.g. in self-driving the progression of increasing autonomy and higher abstraction looks something like:
1. first the human performs all driving actions manually
2. then the AI helps keep the lane
3. then it slows for the car ahead
4. then it also does lane changes and takes forks
5. then it also stops at signs/lights and takes turns
6. eventually you take a feature complete solution and grind on the quality until you achieve full self-driving.
There is a progression of the AI doing more and the human doing less, but still providing oversight. In Software engineering, the progression is shaping up similar:
1. first the human writes the code manually
2. then GitHub Copilot autocompletes a few lines
3. then ChatGPT writes chunks of code
4. then you move to larger and larger code diffs (e.g. Cursor copilot++ style, nice demo here youtube.com/watch?v=Smklr44N…)
5....
Devin is an impressive demo of what perhaps follows next: coordinating a number of tools that a developer needs to string together to write code: a Terminal, a Browser, a Code editor, etc., and human oversight that moves to increasingly higher level of abstraction.
There is a lot of work not just on the AI part but also the UI/UX part. How does a human provide oversight? What are they looking at? How do they nudge the AI down a different path? How do they debug what went wrong? It is very likely that we will have to change up the code editor, substantially.
In any case, software engineering is on track to change substantially. And it will look a lot more like supervising the automation, while pitching in high-level commands, ideas or progression strategies, in English.
Good luck to the team!
Today we're excited to introduce Devin, the first AI software engineer.
Devin is the new state-of-the-art on the SWE-Bench coding benchmark, has successfully passed practical engineering interviews from leading AI companies, and has even completed real jobs on Upwork.
Devin is an autonomous agent that solves engineering tasks through the use of its own shell, code editor, and web browser.
When evaluated on the SWE-Bench benchmark, which asks an AI to resolve GitHub issues found in real-world open-source projects, Devin correctly resolves 13.86% of the issues unassisted, far exceeding the previous state-of-the-art model performance of 1.96% unassisted and 4.80% assisted.
Check out what Devin can do in the thread below.
Devin Pellegrino retweeted
It's fascinating how many path integrals there are for stateful information exchange. #fundamentals #LandauerLimit #EmergentQM #EmergentGR #EmergentSR #EmergentConsciousness #QualiaAreJustXMLtags
This quoted post is unavailable.
Devin Pellegrino retweeted
Here ya go.
LeCun's idea that "the good AI's can protect us from the bad AI's" is actually one of our best bets for ensuring a good future imo
It is what Dan Hendrycks calls a "Leviathan", a collective of good AI's to prevent rogue AI's from acting poorly. It also means we need models to be aligned-by-default, but not necessarily with 100% robustness.
At this point, I feel pretty good about our chances at aligning AGI. The main concern then becomes ensuring that the alignment remains intact each iteration as we scale into the superhuman regime. This could happen quickly and turbulently as productivity soars in a post-AGI world, so there will be risk regardless.
At each stage, the offense-defense balance will also dictate how strict the Leviathan need be to protect everyone.
Devin Pellegrino retweeted
LongNet: Scaling Transformers to 1,000,000,000 Tokens
paper page: huggingface.co/papers/2307.0…
Scaling sequence length has become a critical demand in the era of large language models. However, existing methods struggle with either computational complexity or model expressivity, rendering the maximum sequence length restricted. In this work, we introduce LongNet, a Transformer variant that can scale sequence length to more than 1 billion tokens, without sacrificing the performance on shorter sequences. Specifically, we propose dilated attention, which expands the attentive field exponentially as the distance grows. LongNet has significant advantages: 1) it has a linear computation complexity and a logarithm dependency between tokens; 2) it can be served as a distributed trainer for extremely long sequences; 3) its dilated attention is a drop-in replacement for standard attention, which can be seamlessly integrated with the existing Transformer-based optimization. Experiments results demonstrate that LongNet yields strong performance on both long-sequence modeling and general language tasks. Our work opens up new possibilities for modeling very long sequences, e.g., treating a whole corpus or even the entire Internet as a sequence.
Devin Pellegrino retweeted
We need new technical breakthroughs to steer and control AI systems much smarter than us.
Our new Superalignment team aims to solve this problem within 4 years, and we’re dedicating 20% of the compute we've secured to date towards this problem.
Join us! openai.com/blog/introducing-…
Devin Pellegrino retweeted
Machine unlearning is important but human unlearning is equally so, especially for LLM researchers.
Without a strong theoretical framework to guide us, LLM researchers heavily rely on intuitions formed from empirical observations.
The emergent abilities of LLMs, however, mean many of these intuitions can become outdated or incorrect. For instance, an intuition cultivated in 2020 might become a counter-productive bias by 2023. This I think is one of the rare advantages of the new comers to the field: less to unlearn.
Thus consistently unlearning and updating our intuition is critical. Yet I rarely see this being practiced or discussed. I hope to see tweets sharing the unlearning lessons.
Devin Pellegrino retweeted
Training Transformers with 4-bit Integers
paper page: huggingface.co/papers/2306.1…
Quantizing the activation, weight, and gradient to 4-bit is promising to accelerate neural network training. However, existing 4-bit training methods require custom numerical formats which are not supported by contemporary hardware. In this work, we propose a training method for transformers with all matrix multiplications implemented with the INT4 arithmetic. Training with an ultra-low INT4 precision is challenging. To achieve this, we carefully analyze the specific structures of activation and gradients in transformers to propose dedicated quantizers for them. For forward propagation, we identify the challenge of outliers and propose a Hadamard quantizer to suppress the outliers. For backpropagation, we leverage the structural sparsity of gradients by proposing bit splitting and leverage score sampling techniques to quantize gradients accurately. Our algorithm achieves competitive accuracy on a wide range of tasks including natural language understanding, machine translation, and image classification. Unlike previous 4-bit training methods, our algorithm can be implemented on the current generation of GPUs. Our prototypical linear operator implementation is up to 2.2 times faster than the FP16 counterparts and speeds up the training by up to 35.1%.
Devin Pellegrino retweeted
This is kind of amazing, LLM is trained to change its output when it's wrong... this is actually insane
Devin Pellegrino retweeted
🌟 Thrilled to introduce vLLM with @woosuk_k!
🚀 vLLM is an open-source LLM inference and serving library that accelerates HuggingFace Transformers by 24x and powers @lmsysorg Vicuna and Chatbot Arena.
Github: github.com/vllm-project/vllm
Blog: vllm.ai/





















