Top AI Research Papers for Enthusiasts

Ilya Sutskever from OpenAI once handed John Carmack a reading list of about 30 research papers, saying, “Master these, and you’ll grasp 90% of what’s crucial in AI today.” To round it out, I’ve added a few more LLM papers to cover the remaining 9%.

Whether you’re passionate about natural language processing (NLP), computer vision, or machine learning, these papers offer invaluable insights into cutting-edge advancements in AI technology.

Must-Read Papers

1. Transformers: Attention is All You Need – Explore the groundbreaking transformer model revolutionising NLP.
2. BERT: Pre-training of Deep Bidirectional Transformers – Learn about BERT’s impact on language understanding tasks.
3. GPT: Language Models are Few-Shot Learners – Understand how GPT models have transformed AI capabilities.

Stay Updated

Stay ahead with the latest AI research and developments. Bookmark this page for ongoing updates and new paper recommendations.

1. The Annotated Transformer (nlp.seas.harvard.edu)
2. The First Law of Complexodynamics (scottaaronson.blog)
3. The Unreasonable Effectiveness of RNNs (karpathy.github.io)
4. Understanding LSTM Networks (colah.github.io)
5. Recurrent Neural Network Regularization (arxiv.org)
6. Keeping Neural Networks Simple by Minimizing the Description Length of the Weights (cs.toronto.edu)
7. Pointer Networks (arxiv.org)
8. ImageNet Classification with Deep CNNs (proceedings.neurips.cc)
9. Order Matters: Sequence to sequence for sets (arxiv.org)
10. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism (arxiv.org)
11. Deep Residual Learning for Image Recognition (arxiv.org)
12. Multi-Scale Context Aggregation by Dilated Convolutions (arxiv.org)
13. Neural Quantum Chemistry (arxiv.org)
14. Attention Is All You Need (arxiv.org)
15. Neural Machine Translation by Jointly Learning to Align and Translate (arxiv.org)
16. Identity Mappings in Deep Residual Networks (arxiv.org)
17. A Simple NN Module for Relational Reasoning (arxiv.org)
18. Variational Lossy Autoencoder (arxiv.org)
19. Relational RNNs (arxiv.org)
20. Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton (arxiv.org)
21. Neural Turing Machines (arxiv.org)
22. Deep Speech 2: End-to-End Speech Recognition in English and Mandarin (arxiv.org)
23. Scaling Laws for Neural LMs (arxiv.org)
24. A Tutorial Introduction to the Minimum Description Length Principle (arxiv.org)
25. Machine Super Intelligence Dissertation (vetta.org)
26. PAGE 434 onwards: Komogrov Complexity (lirmm.fr)
27. CS231n Convolutional Neural Networks for Visual Recognition (cs231n.github.io)
28. Improving Language Understanding by Generative Pre-Training
29. Language Models are Unsupervised Multitask Learners
30. Language Models are Few-Shot Learners
31. GPT-4 Technical Report
32. Llama 2: Open Foundation and Fine-Tuned Chat Models
33. What Are Tools Anyway? A Survey from the Language Model Perspective
34. Gemini 1.5: Unlocking multimodalunderstanding across millions of tokens of context