Looped Transformers

Presentation Link#

The seventy-third MeetUp of the Machine Learning Singapore Group, was titled : “MLSG - Looped Transformers, Jev & World Models”, and we covered some of the latest developments in making models better along the road to AGI with new advancements in Transformers and World Models.

My Presentation#

My talk was titled “Looped Transformers” - and was sparked by the recent speculation about OpenAI’s Astra model. However, the actual research dates way back to 2018 - and so my talk explained the thinking, and the curse of being correct-but-too-early.

Overall, I used the following outline :

• Basic Transformers

• The Astra rumours

• Tokenwise Reasoning

• Vertical Expansion

  • Extensions

• Anxiety : Interpretability

• In the news…

• Wrap-up & QR-code

Slides#

The slides for my talk, which contain links to all of the reference materials and sources, are here :

If there are any questions about the presentation please ask below, or contact me using the details given on the slides themselves.

Other Presentations#

The MeetUp also featured:

Sam Witteveen giving a talk about “jev”:

• Idea of System One models
• What jev is
  • and open weights versions
• How to use jev to help build applications

And a lightning talk “Can Small Language Models be World Models?” by Dr Tobe Freeman:

• World Models
• Small, multimodal language models
• Pros / Cons

Acknowledgements#

Many thanks to the Google team, who allowed us to use Google’s Developer Space, and generously provided food for our audience.