- Backend Engineer moving into AI infra - Rust • Go - 23 • koinophobic
Delhi, India
Joined December 2023
- Tweets519
- Following337
- Followers38
- Likes2.4K
Pinned Tweet
Thread of things I've built over the last few years.
Most of them started because I wanted to understand how something actually worked instead of treating it as a black box.
I'll keep adding to this thread whenever I finish something new.
i was reading about qwen2.5 and noticed something odd.
the tokenizer has 151,665 entries, but the embedding table has 151,936 rows.
those 271 extra rows make the table divisible by 128, allowing even splits across different gpu counts.
Abhishek Thakur retweeted
i know these aren't crazy numbers, but damn.
people are actually taking the time to read my stuff
kudos to you :)
i know these aren't crazy numbers, but damn.
people are actually taking the time to read my stuff
kudos to you :)
Abhishek Thakur retweeted
added a request queue to llmgate to handle concurrent requests better
learn learn learn .
Abhishek Thakur retweeted
there is so much to understand when you actually start learning how an llm inference engine works
every time i understand one concept, five more things show up
ahhhhhhhhh
there is so much to understand when you actually start learning how an llm inference engine works
every time i understand one concept, five more things show up
ahhhhhhhhh
Abhishek Thakur retweeted
If Kubernetes feels overwhelming, you’re not alone. I wrote the guide I needed when I started, covering the basics to EKS in nine simple stages :)
Abhishek Thakur retweeted
i keep coming back to go, rust, and python when i think about building systems.
wrote an article on where i'd use each, where this approach falls short, and why ai might make the whole language debate matter less.
therealshek.medium.com/go-ru…
i keep coming back to go, rust, and python when i think about building systems.
wrote an article on where i'd use each, where this approach falls short, and why ai might make the whole language debate matter less.
therealshek.medium.com/go-ru…
slowly adding more to the llmgate arsenal.
currently working on metrics and adding prometheus and grafana support. taking it one piece at a time.
crazy two days working on llm gate.
added token quotas, api key management, daily quota resets, and connected everything to the chat interface.
learned a lot along the way.
phase 1 supports one inference engine. more coming in phase 2. wish me luck.
Abhishek Thakur retweeted
if you want to get good at llm inference engineering, you need to understand more than just running models.
this means learning how inference works, how memory and compute affect performance and how to optimize serving systems.
here's a reading path i'd recommend, in order.
if you want to get good at llm inference engineering, you need to understand more than just running models.
this means learning how inference works, how memory and compute affect performance and how to optimize serving systems.
here's a reading path i'd recommend, in order.
