Unlock Massive Context Windows: Positional embeddings are just training wheels

2 comments

• sjkoelle 9 hours ago

  Would this generate the same completion for 'the cat sat on the' as 'on the cat sat the'?

• bigwheels 10 hours ago

  ref: https://x.com/hardmaru/status/2010565943269999091