Published on

NeurIPS recap and SOTA

Presentation
Authors

The fifty-seventh MeetUp of the Machine Learning Singapore Group, was titled : "2025 Kickoff!! NeurIPS Recap and SOTA".

My Presentation

My talk was titled "The End of Pretraining", and covered the following topics:

  • The NeurIPS conference :
    • Old News (Orals + Posters)
      • with ~6 interesting papers
    • New News (Time-of-time + Workshops)
      • Ilya Sutskever's talk
      • AI-MATHS (Noam Brown telling us to read a blog post)
      • System 2 (lots to enjoy, including Chollet's ARCPrize roundup)
      • and 'extras', eg: @hardmaru talk
  • Newer News (lots of interesting papers since NeurIPS)
    • Newest News (R1, obviously - released < 2 days before) *(Wrap-up & QR-code (the latter to reduce audience distractions)

In particular, the section on R1 discussed the R1-Zero training obvective, and how it was iterated up to give us the full R1 model, and the 'token distilled' smaller versions.

Many thanks to Google for supporting the GCP usage for this project, which was part of their September 2024 #AISprint. My contribution there was titled: "Gemma Fine-tuning with ablations".

The slides for my talk, which contain links to all of the reference materials and sources, are here :

Presentation Screenshot

If there are any questions about the presentation please ask below, or contact me using the details given on the slides themselves.

Presentation Content Example

Other Presentations

In his talk "The State of LLM Patterns & Agents", Sam Witteveen talked about the patterns that are being used for agents, and the various approaches being taken by frameworks. He ended with some recommendations for implementing agentic systems.

Acknowledgements

Many thanks to the Google team, who not only allowed us to use Google's Developer Space, but were also kind enough to provide Pizza for the attendees!