The DAIO project has an official website and a new update is up on the latest. Happy halvening and here's to another interesting four-year cycle! da.io youtube.com/watch?v=lp9OXCMJβ¦
137
Week 3 Update on Project DAIO. It's early days, but I'm enjoying the work. youtu.be/VAKY7PZNP4E?si=YzCmβ¦
2
137
For anyone following along on my attempt to parse the BTC blockchain. Here's an update: we have charts! youtu.be/ZOY3bj79ruA?si=s84cβ¦
1
177
Everybody agrees that X-Men '97 should release new episodes on Saturday mornings right? Can I write my senators about this?
1
191
I started a new project. If anybody's interested in BTC and/or data analysis, then follow along on YouTube! Hit like, subscribe, etc, etc. youtu.be/luxT-WPMo3A?si=xYDtβ¦
1
1
167
Stucki retweeted
I'm proud to announce my candidacy for President of the United States. Let's make an impact in 2024! #votegiantmeteor
Stucki retweeted
Been trying to reconcile a bunch of different aspects of NN behavior and cognition by playing with formulations of latent space at a level (also) suited for reasoning about everyday cognitive phenomena
In part it's led to a compact, diagrammable geometric system
Stucki retweeted
Vision Pro Occlusion Mesh
Aka Night Vision
This is actually a back door into the Matrix
Stucki retweeted
The best cure for loneliness is not more frequent interaction. It's more meaningful interaction.
Many people enjoy solitude. They can spend up to ~75% of their time alone without feeling isolated.
What matters most for well-being is the quality of connections, not the quantity.
Stucki retweeted
The AI models = nuclear weapons analogy is terrible for a lot of reasons, but most importantly it heavily misleads policy-makers. Nuke-inspired regulation won't prevent people from building powerful AIs, but it will protect tech companies from competition π§΅
Stucki retweeted
Introducing OpenWebMath, a massive dataset containing every math document found on the internet - with equations in LaTeX format!
π€Β Download on @HuggingFace: huggingface.co/datasets/openβ¦
π Read the paper: arxiv.org/abs/2310.06786
w/ @dsantosmarco, @zhangir_azerbay, @jimmybajimmyba!
Stucki retweeted
The fact that most individual neurons are uninterpretable presents a serious roadblock to a mechanistic understanding of language models. We demonstrate a method for decomposing groups of neurons into interpretable features with the potential to move past that roadblock.
Stucki retweeted
Created a WebGL neural network visualization for the quadruped robot I've been working on. The more red, the higher the activation. Bluer is less activation, while green is a midpoint.
Stucki retweeted
With many π§© dropping recently, a more complete picture is emerging of LLMs not as a chatbot, but the kernel process of a new Operating System. E.g. today it orchestrates:
- Input & Output across modalities (text, audio, vision)
- Code interpreter, ability to write & run programs
- Browser / internet access
- Embeddings database for files and internal memory storage & retrieval
A lot of computing concepts carry over. Currently we have single-threaded execution running at ~10Hz (tok/s) and enjoy looking at the assembly-level execution traces stream by. Concepts from computer security carry over, with attacks, defenses and emerging vulnerabilities.
I also like the nearest neighbor analogy of "Operating System" because the industry is starting to shape up similar:
Windows, OS X, and Linux <-> GPT, PaLM, Claude, and Llama/Mistral(?:)).
An OS comes with default apps but has an app store.
Most apps can be adapted to multiple platforms.
TLDR looking at LLMs as chatbots is the same as looking at early computers as calculators. We're seeing an emergence of a whole new computing paradigm, and it is very early.
Stucki retweeted
Excited to share #AlphaMissense our new AI system that can classify whether genetic mutations (missense variants) are benign or harmful - a critical step toward uncovering causes of many diseases, from cystic fibrosis to cancer. In @ScienceMagazine today dpmd.ai/AlphaMissenseDH
Stucki retweeted
Language Modeling Is Compression
paper page: huggingface.co/papers/2309.1β¦
It has long been established that predictive models can be transformed into lossless compressors and vice versa. Incidentally, in recent years, the machine learning community has focused on training increasingly large and powerful self-supervised (language) models. Since these large language models exhibit impressive predictive capabilities, they are well-positioned to be strong compressors. In this work, we advocate for viewing the prediction problem through the lens of compression and evaluate the compression capabilities of large (foundation) models. We show that large language models are powerful general-purpose predictors and that the compression viewpoint provides novel insights into scaling laws, tokenization, and in-context learning. For example, Chinchilla 70B, while trained primarily on text, compresses ImageNet patches to 43.4% and LibriSpeech samples to 16.4% of their raw size, beating domain-specific compressors like PNG (58.5%) or FLAC (30.3%), respectively. Finally, we show that the prediction-compression equivalence allows us to use any compressor (like gzip) to build a conditional generative model.
Way too soon .. that scar tissue is still not healed ππ
Sigh .. JavaScript on its best day doesnβt even come close to what we once had
#unity


























