A lot of cool results from these older models, but sadly all pretty small
You could try and push the size up a little bit, but they had a tendency to collapse to just repeatedly predict a fixed token at some point
I guess more modern approaches like yarn/rope would help a lot!