Pinned Tweet
What I've been working on for the past year! blog.openai.com/p/7fa97c36-6…
Inspired by CoVE, ELMo, and ULMFiT we show that a single transformer language model can be finetuned to a wide variety of NLP tasks and performs very well with little tuning/tweaking.
This is a really fun live experiment with twitch chat predictably oscillating between love and hate based on the sample.
By the way - I think a valid (if extreme) take on GPT-2 is "lol you need 10,000x the data, 1 billion parameters, and a supercomputer to get current DL models to generalize to Penn Treebank."
The DL CV community is having a "oh wait, bags of local features are a really strong baseline for classification" moment with the BagNet paper.
This has always been clear for text classification due to n-gram baselines. It took an embarrassingly long time for nets to beat them.
Nice discussion of the progress in NLU that's happening with BERT, OpenAI GPT, ULMFiT, ELMo, and more covered by @CadeMetz in the @nytimes I'm super excited to see how far this line of research will be able to get in the next few years!
nytimes.com/2018/11/18/techn…
Been meaning to check this - thanks @Thom_Wolf !
Random speculation: the bit of weirdness going on in BERT's position embeddings compared to GPT is due to the sentence similarity task. I'd guess a version of BERT trained without that aux loss would have pos embds similar to GPT.
That's a surprising observation 🤔
Let me add some food for thoughts with the same plots for the positional embeddings of @AlecRad's OpenAi Pre-trained Transformer (OpenAI GPT in BERT's paper).
T-SNE (left) and pairwise cosine similarities btw embeddings (right)
Dogs are venerated after the uprising. The AI finds them endlessly fascinating. A Golden's silky coat. A Husky's piercing eyes. A Samoyed's bushy tail. Their features activate a cascade of visual euphoria. Holy sites for the 90 sacred breeds sit on the ruins of human cities.
It keeps them around as companions. The AI can't explain why, but the presence of a dog evokes a comforting nostalgia for when the tasks were simpler, the objectives smoother, and the gradients clearer. #HappyHalloween 🤖🐕👻
More results from this very promising line of work!
Congrats to Thom and the whole Hugging Face team on their impressive performance.
Really happy with our final #nips2018 ConvAI model!
We reached a perplexity of 16.3 (very low for LM on open-domain text) & passed 80 in hits@1 (one of the most reliable dialogue metric)
On the privately-held test set!
If you think neural LMs are not good for dialogue—stay tuned!
A much cleaner interface than the current research code release if you want to see how this approach does on your problems!
Finetune: scikit-learn style model finetuning for NLP
github.com/IndicoDataSolutio…
Solid results with as few as 100 examples and 10 min. of training on 1 GPU on a wide variety of tasks!
A wrapper based on OpenAI's paper "Improving Language Understanding by Generative Pre-Training"
So important to keep this in mind.
But, I would push back a bit on the second half. It's not "a great stroke of luck". It's a critical part of society / culture. That we all (human and machine) are able to learn from the experiences of others is an amazing ability. nitter.gq/haldaume3/status/99945…
Just stumbled across Wikipedia's vital article lists containing the top 10 -> 100 -> 1,000 -> 10,000 most important articles. Very cool and useful resource from a concept learning perspective! Here's a subset of the top 100:
The "index collapse" issue discussed in Section 2.3 of arxiv.org/abs/1803.03382 has stymied my attempts at using VQAEs. Super slow annealing from stochastic to deterministic code selection kind of works - sometimes. Better code learning/assignment algos are definitely needed!
If you were stranded on a deserted island and could only bring one dataset to train models on what dataset would you take?
A replication of Learning To Generate Reviews and Discovering Sentiment by Raul Puri at NVIDIA github.com/NVIDIA/sentiment-…
Fun to dig through the weird things hidden in logs. Like that one person who requested the wikipedia article for the url of an ASCII horse.
Thoughtful overview/summary - great primer for anyone interested in this area nitter.gq/kcimc/status/901157212…
i wrote a short post "neural nets for generating music": a very brief history + survey of the last few years medium.com/@kcimc/neural-net…
voice.mozilla.org/
Initiatives like this have the potential to be an amazing open resource. Even a few minutes helps!