Philippe Hansen-Estruch (@tokenpilled65B) profile picture

Philippe Hansen-Estruch

@tokenpilled65B

Research Scientist Intern at FAIR. Third-year PhD student at UT Austin. Working on generative modeling, visual understanding, and visual compression.

 Austin, TX
Joined June 2023
  • Tweets2.7K
  • Following889
  • Followers934
96Media
Photo by Philippe Hansen-Estruch (@tokenpilled65B)Photo by Philippe Hansen-Estruch (@tokenpilled65B)Photo by Philippe Hansen-Estruch (@tokenpilled65B)Photo by Philippe Hansen-Estruch (@tokenpilled65B)Photo by Philippe Hansen-Estruch (@tokenpilled65B)Photo by Philippe Hansen-Estruch (@tokenpilled65B)Photo by Philippe Hansen-Estruch (@tokenpilled65B)Photo by Philippe Hansen-Estruch (@tokenpilled65B)Photo by Philippe Hansen-Estruch (@tokenpilled65B)Photo by Philippe Hansen-Estruch (@tokenpilled65B)
Releasing ViTok-v2: open-source ViT auto-encoder codebase + pretrained weights Train your own ViT auto-encoder on any streamed (hf://) or local webdataset. NaFlex pipeline handles any resolution and aspect ratio Includes reproduced 350M and 4.5B models weights competitive at 256p, SOTA at high-res (512p+)
Image from Philippe Hansen-Estruch (@tokenpilled65B)
18
54
4
427
35,919
I think I have thousands of hours on warband napoleonic at this point Vive L'Empereur
Image from Philippe Hansen-Estruch (@tokenpilled65B)
peepeepoopoo (@DeepDishEnjoyer) profile picturepeepeepoopoo@DeepDishEnjoyerSep 26
not even close to my favorite games but if we are talking about formative, childhood experiences:
Image from peepeepoopoo (@DeepDishEnjoyer)
2
10
649
LPIPS has been the go-to perceptual loss for autoencoder training, but it's built on VGG features from 2014. Today we're releasing DINO Perceptual Loss: a modern replacement using DINO that achieves better results without adversarial training: uv pip install dino-perceptual
Pareto frontier results trained on ViTok tokenizers (f16x64) evaluated on ImageNet 256p validation. Compared against Qwen Image VAE and Flux.1 which are leading open source VAEs.

ALT Pareto frontier results trained on ViTok tokenizers (f16x64) evaluated on ImageNet 256p validation. Compared against Qwen Image VAE and Flux.1 which are leading open source VAEs.

6
7
1
55
3,472
Blog link for more information: na-vae.github.io/dino_percep… github: github.com/Na-VAE/dino_perce…
1
10
907
Has anyone worked with the VAE from Qwen Image? I found the huggingface implementation to be super slow to do a forward pass and also get much worse ImageNet recon benchmarks compared to what they claim on their tech report
1
2
477
Heck its even weirded because their claimed Flux result is NOT what the Flux authors claim themselves! (PSNR 32.84 vs 31.1312) (SSIM 0.9155 vs 0.8893) - bfl.ai/research/representati…. Makes me wonder if they ran the benchmark wrong?
Image from Philippe Hansen-Estruch (@tokenpilled65B)
283
Torch.compile takes me from 34 to 49 MFU on A100s damn kind of cracked
Image from Philippe Hansen-Estruch (@tokenpilled65B)
7
1
78
6,498
Anyone ever experience weird behaviors with PyTorch's checkpoint module? When I wrap my transformer blocks with it the loss gets really weird to train for a ViT based VAE. I don't remember this issue from before - maybe something to do with use_reentrant=False?
Image from Philippe Hansen-Estruch (@tokenpilled65B)
1
268
``` def _cast_fwd(_x): with torch.amp.autocast(device_type="cuda", dtype=torch.bfloat16): return block(_x, freqs_cis=freqs_cis, attn_mask=attn_mask) x = _checkpoint(_cast_fwd, x, use_reentrant=False) else: x = block(x, freqs_cis=freqs_cis, attn_mask=attn_mask) ```
1
124
pov what your vit sees post position embedding
Image from Philippe Hansen-Estruch (@tokenpilled65B)
1
4
207
I messed up my data pipeline so the VAE learned the target of patches + 2d posemb. It did a pretty good job!
1
71
what did gemini mean by this
Image from Philippe Hansen-Estruch (@tokenpilled65B)
2
273
Does anyone have good code/resources for converting ViT Jax weights/arch to PyTorch? I've seen some stuff online but huggingface is quite confusing and o3 was totally useless here. I am trying to convert a NaFlex ViT VAE over to PyTorch but its proving very annoying
2
5
794
Why is Jax so annoying to use with dynamic shapes. Trying to use a VGG with variable aspect ratios and resolutions and Jax just does not like it one bit. Padding also is a bad solution because almost >2/3 of the image will be padded. Ugh man... we need a replacement to LPIPS
4
420
thank you claude
Image from Philippe Hansen-Estruch (@tokenpilled65B)
1
6
278
Excited to share my work at Meta! We explore scaling tokenizers w/ ViT (ViTok) & found scaling tokenizers with DiT generation pipeline doesn’t boost performance for the current paradigm of auto-encoders! We develop SOTA tokenizers for images/videos. Thread for findings
Image from Philippe Hansen-Estruch (@tokenpilled65B)
9
59
4
408
57,957
We start our exploration by upgrading the common CNN auto-encoder w/ ViT+Llama architecture (SwiGLU + 3D RoPE) and train on huge datasets (450M images and 30M videos!). We focus our exploration on scaling bottleneck size, encoder size, an decoder size.
2
10
1,899
We present Unified Masked Diffusion (UMD), a self-supervised objective that trains a ViT based auto-encoder for downstream linear probing and class-conditional generation. Philippe Hansen-Estruch, et al. Paper: arxiv.org/abs/2406.17688 Code: github.com/philippe-eecs/sma… A Thread:
Image from Philippe Hansen-Estruch (@tokenpilled65B)
4
49
4
212
41,502
UMD seeks to harness the computational and representation benefits of MAE while also being effective for class-conditional generation. Our objective builds on MaskDiT by combining the fine-grained noise schedule of diffusion with the token-based masking of MAE.
Image from Philippe Hansen-Estruch (@tokenpilled65B)
1
3
723
Good
Dan Hendrycks (@hendrycks) profile pictureDan Hendrycks
@hendrycks22 Jan 2024
- Meta, by open sourcing competitive models (e.g., Llama 3) they reduce AI orgs' revenue/valuations/ability to buy more GPUs and scale AI models
2
231
Mfers will really waste time learning cuda rather than using Jax
Jack Morris (@jxmnop) profile pictureJack Morris
@jxmnop4 Dec 2023
fewer than 100 people deeply understand both (i) transformers and (ii) the GPU programming model want to learn machine learning? gain some esoteric systems knowledge; spend some time really learning CUDA
110
Stuffing myself after neurips rebuttals
Image from Philippe Hansen-Estruch (@tokenpilled65B)
1
712

No more items