
CEO, Aegean AI Inc. Educator @NJIT and @nyuniversity
Randolph, New Jersey, USA
Joined November 2008
- Tweets223
- Following640
- Followers163
- Likes326
Pantelis Monogioudis retweeted
Replying to @deepseek_ai
@deepseek_ai AI may be a blessing for many. - Dr. Pantelis Monogioudis, Dept. of Data Science
linkedin.com/feed/update/urn…
Pantelis Monogioudis retweeted
If an AI can control 1,000 robots to perform 1 million skills in 1 billion different simulations, then it may "just work" in our real world, which is simply another point in the vast space of possible realities. This is the fundamental principle behind why simulation works so effectively for robotics.
Real-world teleoperation data scales linearly with human time (< 24 hrs/robot/day). Sim data scales exponentially with compute.
There are 3 big trends for simulators in the near future:
1. Massive parallelization on large clusters. Physics equations are "just" matrix math at their core. I hear GPUs are good at matrix math 🔥. One can run 100K copies of simulation on a single GPU. To put this number in perspective: 1 hour of wallclock compute time gives a robot 10 years (!!) of training experience. That's how Neo was able to learn martial arts in a blink of an eye in the Matrix Dojo.
2. Generative graphics pipeline. Traditionally, simulators require a huge amount of manual effort from artists: 3D assets, textures, scene layouts, etc. But every component in the workflow can be automated: text-to-image, text-to-3D mesh, and LLMs that write Universal Scene Description (USD) files as a coding exercise. RoboCasa is one example of a prior work.
3. End2end neural net that acts as simulator itself. This is still bluesky research and quite far from replacing a graphics pipeline, but we are seeing some exciting signs-of-life based on video gen models: Sora, Veo2, CogVideoX, Hunyuan (text-to-video); and action-driven world models: GameNGen, Oasis, Genie-2, etc.
Genesis does great on (1) for certain tasks, shows good promises on (2), and could become a data generation tool for reaching (3). Its sim2real capabilities for locomotion are good, but there's still a long way to go for contact-rich, dexterous manipulation. It shows a bold vision and is on the right path to providing a virtual cradle for embodied AI. It is open-source and puts a streamlined user journey at the front and center.
I had the privilege to know @zhou_xian_ and play a small part in his project since a year ago. Xian has been crunching code non-stop on Genesis with a very small group of core devs. He often replied to my messages at 3 am. @Zhenjia_Xu from our GEAR team helped with sim2real experiments in his spare time. Genesis is truly a grassroot effort with an intense focus on quality engineering.
Nothing gives me more joy than seeing the simulation ecosystem bloom. Robotics should be a moonshot initiative owned by all of humanity.
Congratulations.
Pantelis Monogioudis retweeted
The inventors of flow matching have released a comprehensive guide going over the math & code of flow matching!
Also covers variants like non-Euclidean & discrete flow matching.
A PyTorch library is also released with this guide!
This looks like a very good read! 🔥
Pantelis Monogioudis retweeted
A new tutorial on RL by Kevin Patrick Murphy, a Research Scientist at Google DeepMind who also wrote several comprehensive, well-regarded textbooks on ML/DL.
This ought to be a good read 👀
Pantelis Monogioudis retweeted
Must-read papers for LLM-based agents.
A nice collection in this Github
Pantelis Monogioudis retweeted
This open-source RAG tool for chatting with your documents is Trending at Number-1 in Github from the past few days
🔍 Open-source RAG UI for document QA
🛠️ Supports local LLMs and API providers
📊 Hybrid RAG pipeline with full-text & vector retrieval
🖼️ Multi-modal QA with figures & tables support
📄 Advanced citations with in-browser PDF preview
🧠 Complex reasoning with question decomposition
⚙️ Configurable settings UI
🔧 Extensible Gradio-based architecture
Key features:
🌐 Host your own RAG web UI with multi-user login
🤖 Organize LLM & embedding models (local & API)
🔎 Hybrid retrieval + re-ranking for quality
📚 Multi-modal parsing and QA across documents
💡 Detailed citations with relevance scores
🧩 Question decomposition for complex queries
🎛️ Adjustable retrieval & generation settings
🔌 Customizable UI and indexing strategies
Pantelis Monogioudis retweeted
Microsoft has launched one of the most comprehensive course on Generative AI!
This course is 18 lessons, where each lesson covers its own topic so start wherever you would like!
Lessons are labeled either "Learn" lessons explaining a Generative AI concept or "Build" lessons that explain a concept and code examples in both Python and TypeScript when possible.
Here's the link to the courses: github.com/microsoft/generat…
↓
If you're interested in AI Engineering, I write a weekly Newsletter @ML_Spring , join 11k+ readers now!
Pantelis Monogioudis retweeted
github.com/karpathy/llm.c
@karpathy has now implemented training of GPT-2 (CPU, fp32) in C, ~1,000 lines of clean code in a single file
‘What I cannot create, I do not understand’
Pantelis Monogioudis retweeted
Introducing CodeGemma, a family of open code LLMs based on Gemma.
CodeGemma 7B models excel in mathematical reasoning and match the code capabilities of other open models.
The instruction-tuned CodeGemma 7B model is the more powerful model for Python coding as assessed via the HumanEval benchmark. Results also suggest that the model performs best on GSM8K among 7B models.
The CodeGemma 2B model achieves SoTA code completion and is designed for fast code infilling and deployment in latency-sensitive settings.
Pantelis Monogioudis retweeted
RAG From Scratch: Feedback + self-reflection
Our RAG From Scratch video series walks through impt RAG concepts in short / focused videos w/ code.
This is the final part in our series, focusing on self-reflection + feedback to improve RAG systems.
🔧Problem: RAG systems can suffer from low quality retrieval (e.g., if a user question is out of the domain for the index) and / or hallucinations in generation. A naive retrieve-generate pipeline has no ability to detect or self-correct from these kids of errors.
💡Idea: The concept of "flow engineering" has been recently introduced by @itamar_mar the context of code generation: iteratively build an answer to a code question w/ unit tests to check and self-correct errors.
@HamelHusain has a great blog post that mentions the benefit of unit-testing in inference loop. Several works have applied this RAG, such as Self-RAG (@AkariAsai et al) and Corrective-RAG (@Jiachen_Gu + colleages). In both cases, checks for document relevance, hallucinations, and / or answer quality are performed in the RAG answer flow. We've implemented these ideas using LangGraph to orchestrate the checks and feedback. We've also shown that LangGraph allows both to run reliable w/ smaller OSS models.
📽️ Video:
youtube.com/watch?v=E2shqsYw…
💻 Code:
CRAG:
github.com/langchain-ai/lang…
Self-RAG:
github.com/langchain-ai/lang…
Both with @MistralAI-7b + @ollama:
github.com/langchain-ai/lang…
github.com/langchain-ai/lang…
🧠References:
1/ Self-RAG
arxiv.org/abs/2310.11511.pdf
2/ C-RAG:
arxiv.org/pdf/2401.15884.pdf
3/ Flow-engineering:
nitter.gq/karpathy/status/174804…
4/ Blog on evals, covering unit tests:
hamel.dev/blog/posts/evals/
Pretty interesting summary especially for those looking to position themselves on what they already know and what the can learn next
survey.stackoverflow.co/2023
68
Pantelis Monogioudis retweeted
Anyone who thinks Auto-Regressive LLMs are getting close to human-level AI, or merely need to be scaled up to get there, *must* read this.
AR-LLMs have very limited reasoning and planning abilities.
This will not be fixed by making them bigger and training them on more data.
So my👇 thread about our papers investigating the verification and self-critiquing inabilities of GPT4 has apparently resonated with a lot of folks. Here is a quick response to several issues raised (either in replies or other quote-tweet threads). [Using one of 'em long "vanity tweets" 😛]
1. I am not an LLM luddite--I think they are amazing "idea generators" (in either language or code form). They just can't do their own planning/reasoning with any guarantees. So they are best used in LLM-Modulo settings (with either a sound reasoner or an expert human in the loop). Self-critiquing needs verification, which is a form of reasoning (..and so I was surprised at all the claims about LLM self-critiquing abilities).
2. This should be pretty obvious to anyone with Intro #AI background--unless folks writing papers making these claims have skipped that..? (c.f. nitter.gq/rao2z/status/168146127…). More likely, it happens because the authors anthropomorphize LLMs ("I mean, people do produce wrong solutions, and repair them themselves, right?). (c.f. nitter.gq/rao2z/status/169710160… ).
Finally, the use of haphazardly curated Q&A datasets to make and evaluate self-critiquing claims has also, in my view, contributed to the confusion/conflation in the community (see nitter.gq/rao2z/status/171183590… )
3. While LLMs can't plan, reason or verify, they can however help with those in LLM-Modulo settings
We show a way of doing this for planning in a #NeurIPS2023 spotlight: arxiv.org/abs/2305.15771 (thread: nitter.gq/rao2z/status/166193833…)
4. Some have pointed out that while GPT4 cannot verify colorings, it can *help* come up with python code (especially given how common coloring problem code is in github 🙄), which once "corrected" as needed, can serve as the external verifier. Yes--but once again, it is the human in the loop which guaranteed the correctness of the verifier. Without that, there is no guarantee that verification amounts to much.
5. Continuing on 4, another of our #NeurIPS2023 papers (arxiv.org/abs/2305.14909) shows how a planning domain model can be teased out of an LLM (for any reasonably commonsense domain), refined with the help of humans and specialized external reasoners in the loop, and then thrown over to a plan verifier (such as VAL) or a domain-independent planner. This can then piggyback on the LLM-Modulo approaches mentioned in 3 above. (thread: nitter.gq/rao2z/status/166194489…)
6. [[Yet "band aid" to ensure that LLMs' verification capability is better than it is out of the box would be to *specifically* fine tune it for verification. We tried this in a previous paper nitter.gq/rao2z/status/166352362…. Note that this doesn't make LLM suddenly do better reasoning of verification--but just makes it better at faking verification (c.f. nitter.gq/rao2z/status/169992889…) Enough, at any rate, to make self critiquing work a little.]]
(For further elaboration on all of the above, see the thread about a tutorial on the role of LLMs in planning at nitter.gq/rao2z/status/167942751…, or the CACM blog piece nitter.gq/rao2z/status/170161286…)
Thanks for reading!
Pantelis Monogioudis retweeted
Today may be the ImageNet moment for robotics.
RT-X: the largest open-source robot dataset ever compiled, across 33 institutes, 22 robot hardware, 527 skills, and 1M episodes.
Why is robotics lagging so far behind NLP, vision, and other AI domains? Data scarcity is the main culprit to blame, among other difficulties. Unlike text, images, and videos, you cannot download mass amounts of onboard robot control data from the internet. They simply don't exist in the wild.
11 yrs ago, ImageNet kicked off the deep learning revolution. 3-4 yrs ago, internet-scale data fueled the first GPTs and Diffusions that define this era of foundation models. I think 2023 is finally the year for robotics to scale up.
Robot foundation models like VIMA (vimalabs.github.io/, my team's work at NVIDIA) and RT-1/2 (deepmind.com/blog/rt-2-new-m…, Google DeepMind's effort) are extremely data hungry. While massively parallel simulations like NVIDIA IsaacGym & Omniverse can alleviate the problem to some extent, it's still not quite enough to bridge the gap to the messy, physical world.
This new dataset is not just a technical contribution. I also see it as a commendable effort to overcome institutional bureaucracies and unite researchers from around the world to tackle a grand challenge together.
Robotics will be the final holy grail that we capture in AI. We are not there yet, but ascending in the right gradient direction.
RT-X website: robotics-transformer-x.githu…
Launch blog: deepmind.com/blog/scaling-up…
Pantelis Monogioudis retweeted
Here, in full directly on Twitter, is "A Hackers' Guide to Language Models". This 90 minute tutorial is designed to be the one place I point coders at when they ask "hey, tell me everything I need to know about LLMs!"
It covers both @OpenAI models and open source ones in depth.
Pantelis Monogioudis retweeted
This is the way to unlock the next trillion high-quality tokens, currently frozen in textbook pixels that are not LLM-ready.
Nougat: an open-source OCR model that accurately scans books with heavy math/scientific notations. It's ages ahead of other open OCR options. Meta is doing extraordinary open-source AI, sometimes without as much fanfare as Llama.
My first serious AI research project (back @Columbia, 2012) was to convert chemical engineering PDFs into NLP-ready corpus. I still remember the immense pain of Tesseract, a much older OCR system (github.com/tesseract-ocr/tes…).
Now Nougat runs a powerful Swin Transformer backbone and blows the benchmarks out of the water. We're talking about double-digit improvements across all metrics.
Now, textbooks are all we need for the next GPT!
Website: facebookresearch.github.io/n…
Open-source code: github.com/facebookresearch/…
Paper "Nougat: Neural Optical Understanding for Academic Documents": arxiv.org/abs/2308.13418
Pantelis Monogioudis retweeted
CVPR 2023 just announced the Best Paper Awards! It's the world's most prominent computer vision conference, with citation impact just below Nature & Science. Read with me:
▸ Best Paper 1: VisProg uses GPT to generate executable code that parses an image and does effective visual reasoning, even though GPT itself is blind. Similar principle as Voyager (executable code to play Minecraft).
▸ Best Paper 2: Unified Autonomous Driving (UniAD), a comprehensive framework that incorporates full-stack driving tasks in one network.
▸ Best Paper Honorable Mention: DynIBaR, a new state-of-the-art on synthesizing novel views from a monocular video of a complex dynamic scene.
▸ Best Student Paper: a new 3D point cloud registration technique that finds the optimal pose to align a pair of point clouds.
▸ Best Student Paper Honorable Mention: a diffusion model that can be customized to a particular subject with only 3-5 example images.
Links in 🧵:
Pantelis Monogioudis retweeted
You'll soon see lots of "Llama just dethroned ChatGPT" or "OpenAI is so done" posts on Twitter. Before your timeline gets flooded, I'll share my notes:
▸ Llama-2 likely costs $20M+ to train. Meta has done an incredible service to the community by releasing the model with a commercially-friendly license. AI researchers from big companies were wary of Llama-1 due to licensing issues, but now I think many of them will jump on the ship and contribute their firepower.
▸ Meta's team did a human study on 4K prompts to evaluate Llama-2's helpfulness. They use "win rate" as a metric to compare models, in similar spirit as the Vicuna benchmark. 70B model roughly ties with GPT-3.5-0301, and performs noticeably stronger than Falcon, MPT, and Vicuna.
I trust these real human ratings more than academic benchmarks, because they typically capture the "in-the-wild vibe" better.
▸ Llama-2 is NOT yet at GPT-3.5 level, mainly because of its weak coding abilities. On "HumanEval" (standard coding benchmark), it isn't nearly as good as StarCoder or many other models specifically designed for coding. That being said, I have little doubt that Llama-2 will improve significantly thanks to its open weights.
▸ Meta's team goes above and beyond on AI safety issues. In fact, almost half of the paper is talking about safety guardrails, red-teaming, and evaluations. A round of applause for such responsible efforts!
In prior works, there's a thorny tradeoff between helpfulness and safety. Meta mitigates this by training 2 separate reward models. They aren't open-source yet, but would be extremely valuable to the community.
▸ I think Llama-2 will dramatically boost multimodal AI and robotics research. These fields need more than just blackbox access to an API.
So far, we have to convert the complex sensory signals (video, audio, 3D perception) to text description and then feed to an LLM, which is awkward and leads to huge information loss. It'd be much more effective to graft sensory modules directly on a strong LLM backbone.
▸ The whitepaper itself is a masterpiece. Unlike GPT-4's paper that shared very little info, Llama-2 spelled out the entire recipe, including model details, training stages, hardware, data pipeline, and annotation process. For example, there's a systematic analysis on the effect of RLHF with nice visualizations.
Quote sec 5.1: "We posit that the superior writing abilities of LLMs, as manifested in surpassing human annotators in certain tasks, are fundamentally driven by RLHF."
Congrats to the team again 🥂! Today is another delightful day in OSS AI.
Pantelis Monogioudis retweeted
What if I told you that you can attain human-level sample efficiency without LLMs or a world model, just by scaling up model-free RL? I’m happy to present our new paper, Bigger, Better, Faster: Human-Level Atari with Human-Level Efficiency, at ICML 2023.
arxiv.org/abs//2305.19452






























