Sources & references

On attribution

Each work below is cited to its authors and linked to its canonical source — an arXiv page, a DOI, or the publishing venue's proceedings. The papers themselves remain the work and property of their respective authors; they are referenced here for scholarly commentary under normal academic citation. Where a paper is discussed by more than one essay it is listed once in the master bibliography and repeated in each relevant essay section, always with the same attribution. If you spot a citation that is wrong or incomplete, let me know at dattgoswami@gmail.com.

Master bibliography

74 unique works, ordered by first author.

1. Abbas, Z., Zhao, R., Modayil, J., White, A., & Machado, M. C. (2023). Loss of plasticity in continual deep reinforcement learning. Conference on Lifelong Learning Agents (CoLLAs). arXiv:2303.07507
2. Abel, D., Barreto, A., Van Roy, B., Schölkopf, B., Silver, D., & Singh, S. (2023). A definition of continual reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 36. arXiv:2307.11046
3. Aghajohari, M., Chitsaz, K., Kazemnejad, A., Chandar, S., Sordoni, A., Courville, A., & Reddy, S. (2025). The Markovian thinker: Architecture-agnostic linear scaling of reasoning. arXiv:2510.06557
4. Beel, J., Kan, M.-Y., & Baumgart, M. (2025). Evaluating Sakana's AI Scientist for autonomous research: Wishful thinking or an emerging reality towards 'Artificial Research Intelligence' (ARI)?. arXiv:2502.14297
5. Cetin, E., Zhao, T., & Tang, Y. (2025). Reinforcement learning teachers of test time scaling. Advances in Neural Information Processing Systems (NeurIPS). arXiv:2506.08388
6. Chandra, A., Agrawal, A., Hosseini, A., Fischmeister, S., Agarwal, R., Goyal, N., & Courville, A. (2025). Shape of thought: When distribution matters more than correctness in reasoning tasks. arXiv:2512.22255
7. Chandrasekar, S., & Machado, M. C. (2025). Towards an option basis to optimize all rewards. Reinforcement Learning Journal / Reinforcement Learning Conference (RLC). openreview.net/forum?id=93kcaZhYgG
8. Colelough, B. C., & Regli, W. (2024). Neuro-symbolic AI in 2024: A systematic review. arXiv:2501.05435
9. Creus Castanyer, R., Obando-Ceron, J., Li, L., Bacon, P.-L., Berseth, G., Courville, A., & Castro, P. S. (2025). Stable gradients for stable learning at scale in deep reinforcement learning. arXiv:2506.15544
10. Darlow, L., Regan, C., Risi, S., Seely, J., & Jones, L. (2025). Continuous thought machines. Advances in Neural Information Processing Systems (NeurIPS). arXiv:2505.05522
11. DeepSeek-AI (2025). DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv:2501.12948
12. Dohare, S., Hernandez-Garcia, J. F., Lan, Q., Rahman, P., Mahmood, A. R., & Sutton, R. S. (2024). Loss of plasticity in deep continual learning. Nature, 632, 768–774. doi:10.1038/s41586-024-07711-7
13. Dohare, S., Hernandez-Garcia, J. F., Rahman, P., Sutton, R. S., & Mahmood, A. R. (2023). Maintaining plasticity in deep continual learning via regenerative regularization. ICML 2023 Workshop on Continual Lifelong Learning. arXiv:2306.13812
14. Garcez, A. d'A., & Lamb, L. C. (2020). Neurosymbolic AI: The 3rd wave. arXiv:2012.05876
15. Gu, S., Knoll, A., & Jin, M. (2024). TeaMs-RL: Teaching LLMs to generate better instruction datasets via reinforcement learning. Transactions on Machine Learning Research (TMLR). arXiv:2403.08694
16. Guo, S., Darwiche Domingues, O., Avalos, R., Courville, A., & Strub, F. (2025). World modelling improves language model agents. arXiv:2506.02918
17. György, A., Lattimore, T., Lazić, N., & Szepesvári, C. (2025). Beyond statistical learning: Exact learning is essential for general intelligence. arXiv:2506.23908
18. Hafner, D., Pasukonis, J., Ba, J., & Lillicrap, T. (2023). Mastering diverse domains through world models (DreamerV3). arXiv:2301.04104
19. Han, S., Pari, J., Gershman, S. J., & Agrawal, P. (2025). General intelligence requires reward-based pretraining. Proceedings of the 42nd International Conference on Machine Learning (ICML). arXiv:2502.19402
20. Hansen, N., Wang, X., & Su, H. (2022). Temporal difference learning for model predictive control (TD-MPC). Proceedings of the 39th International Conference on Machine Learning (ICML), PMLR 162. arXiv:2203.04955
21. Huang, Z., Xia, X., Ren, Y., Zheng, J., Xiao, X., Xie, H., Li, H., Liang, S., Dai, Z., Zhuang, F., Li, J., Ban, Y., & Wang, D. (2026). Real-time aligned reward model beyond semantics (R2M). arXiv:2601.22664
22. Javed, K., & Sutton, R. S. (2024). The big world hypothesis and its ramifications for artificial intelligence. Finding the Frame Workshop, Reinforcement Learning Conference (RLC). openreview.net/forum?id=Sv7DazuCn8
23. Jiang, L., Chai, Y., Li, M., Liu, M., Fok, R., Dziri, N., Tsvetkov, Y., Sap, M., Albalak, A., & Choi, Y. (2025). Artificial hivemind: The open-ended homogeneity of language models (and beyond). Advances in Neural Information Processing Systems (NeurIPS). arXiv:2510.22954
24. Khetarpal, K., Riemer, M., Rish, I., & Precup, D. (2022). Towards continual reinforcement learning: A review and perspectives. Journal of Artificial Intelligence Research (JAIR), 75, 1401–1476. arXiv:2012.13490
25. Kim, J., Lai, S., Scherrer, N., Agüera y Arcas, B., & Evans, J. (2026). Reasoning models generate societies of thought. arXiv:2601.10825
26. Kim, J., Wu, D., Lee, J. D., & Suzuki, T. (2025). Metastable dynamics of chain-of-thought reasoning: Provable benefits of search, RL and distillation. arXiv:2502.01694
27. Klissarov, M., Bagaria, A., Luo, Z., Konidaris, G., Precup, D., & Machado, M. C. (2025). Discovering temporal structure: An overview of hierarchical reinforcement learning. arXiv:2506.14045
28. Koishekenov, Y., Lipani, A., & Cancedda, N. (2025). Encode, think, decode: Scaling test-time reasoning with recursive latent thoughts. arXiv:2510.07358
29. Kotamreddy, H., & Machado, M. C. (2025). A study of value-aware eigenoptions. Workshop on Inductive Biases in RL, Reinforcement Learning Conference (RLC). arXiv:2507.09127
30. Kumar, S., Jeon, H. J., Lewandowski, A., et al. (2024). The need for a big world simulator: A scientific challenge for continual learning. arXiv:2408.02930
31. Kumar, S., Marklund, H., Rao, A., Zhu, Y., Jeon, H. J., Liu, Y., & Van Roy, B. (2023). Continual learning as computationally constrained reinforcement learning. Reinforcement Learning Journal. arXiv:2307.04345
32. Lange, R. T., Imajuku, Y., & Cetin, E. (2025). ShinkaEvolve: Towards open-ended and sample-efficient program evolution. arXiv:2509.19349
33. Lawsen, A. (2025). The illusion of the illusion of thinking: A comment on Shojaee et al. (2025). arXiv:2506.09250
34. Lewandowski, A., et al. (2024). Plastic learning with deep Fourier features. arXiv:2410.20634
35. Lewandowski, A., Limbacher, J., Precup, D., & Courville, A. (2024). Continual learning by spectral regularization. arXiv:2406.06811
36. Lewandowski, A., Ramesh, A. A., Meyer, E., Schuurmans, D., & Machado, M. C. (2025). The world is bigger: A computationally-embedded perspective on the big world hypothesis. Advances in Neural Information Processing Systems (NeurIPS). arXiv:2512.23419
37. Li, C., Elmahdy, A., Boyd, A., Wang, Z., Zeng, S., Garcia, A., Bhatia, P., Kass-Hout, T., Xiao, C., & Hong, M. (2025). Stabilizing off-policy training for long-horizon LLM agents via turn-level importance sampling and clipping-triggered normalization (SORL). arXiv:2511.20718
38. Lin, Z., Nikishin, E., He, X. O., & Courville, A. (2025). Forgetting Transformer: Softmax attention with a forget gate. International Conference on Learning Representations (ICLR). arXiv:2503.02130
39. Lin, Z., Obando-Ceron, J., He, X. O., & Courville, A. (2025). Adaptive computation pruning for the Forgetting Transformer. Conference on Language Modeling (COLM). arXiv:2504.06949
40. Liu, B., Guertler, L., Yu, S., Liu, Z., Qi, P., Balcells, D., Liu, M., Tan, C., Shi, W., Lin, M., Lee, W. S., & Jaques, N. (2025). SPIRAL: Self-play on zero-sum games incentivizes reasoning via multi-agent multi-turn reinforcement learning. arXiv:2506.24119
41. Liu, J., Obando-Ceron, J., Lu, H., He, Y., Wang, W., Su, W., Zheng, B., Castro, P. S., Courville, A., & Pan, L. (2025). Asymmetric proximal policy optimization: Mini-critics boost LLM reasoning. arXiv:2510.01656
42. Liu, J., Wu, Z., Obando-Ceron, J., Castro, P. S., Courville, A., & Pan, L. (2025). Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learning. arXiv:2505.24061
43. Liu, M., Diao, S., Lu, X., Hu, J., Dong, X., Choi, Y., Kautz, J., & Dong, Y. (2025). ProRL: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv:2505.24864
44. Mayor, W., Obando-Ceron, J., Courville, A., & Castro, P. S. (2025). The impact of on-policy parallelized data collection on deep reinforcement learning networks. arXiv:2506.03404
45. Micheli, V., Alonso, E., & Fleuret, F. (2023). Transformers are sample-efficient world models (IRIS). International Conference on Learning Representations (ICLR). arXiv:2209.00588
46. MiniMax (2025). MiniMax-M1: Scaling test-time compute efficiently with lightning attention. arXiv:2506.13585
47. Nekoei, H., Badrinaaraayanan, A., Courville, A., & Chandar, S. (2021). Continuous coordination as a realistic scenario for lifelong learning. Proceedings of the International Conference on Machine Learning (ICML), 139, 8016–8024. arXiv:2103.03216
48. Pan, M., Zhang, W., Chen, G., Zhu, X., Gao, S., Wang, Y., & Yang, X. (2023). Continual visual reinforcement learning with a life-long world model. arXiv:2303.06572
49. Qiu, Z., Wang, Z., Zheng, B., Huang, Z., Wen, K., Yang, S., Men, R., Yu, L., Huang, F., Huang, S., Liu, D., Zhou, J., & Lin, J. (2025). Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free. Advances in Neural Information Processing Systems (NeurIPS). arXiv:2505.06708
50. Saheb, A., Obando-Ceron, J., Courville, A., Bashivan, P., & Castro, P. S. (2026). Stable deep reinforcement learning via isotropic Gaussian representations. arXiv:2602.19373
51. Schaul, T., Barreto, A., Quan, J., & Ostrovski, G. (2022). The phenomenon of policy churn. Advances in Neural Information Processing Systems (NeurIPS). arXiv:2206.00730
52. Schlegel, M., Jacobsen, A., Abbas, Z., Patterson, A., White, A., & White, M. (2021). General value function networks. Journal of Artificial Intelligence Research (JAIR), 70, 497–543. arXiv:1807.06763
53. Shah, V., Obando-Ceron, J., Jain, V., Bartoldson, B., Kailkhura, B., Mittal, S., Berseth, G., Castro, P. S., Bengio, Y., Malkin, N., Jain, M., & Venkatraman, S. (2025). A comedy of estimators: On KL regularization in RL training of LLMs. arXiv:2512.21852
54. Shalev-Shwartz, S., & Shashua, A. (2025). From reasoning to super-intelligence: A search-theoretic perspective. arXiv:2507.15865
55. Simonds, T., & Yoshiyama, A. (2025). LADDER: Self-improving LLMs through recursive problem decomposition. arXiv:2503.00735
56. Sokar, G., Agarwal, R., Castro, P. S., & Evci, U. (2023). The dormant neuron phenomenon in deep reinforcement learning. Proceedings of the International Conference on Machine Learning (ICML). arXiv:2302.12902
57. Stojanovski, Z., Stanley, O., Sharratt, J., Jones, R., Adefioye, A., Kaddour, J., & Köpf, A. (2025). Reasoning Gym: Reasoning environments for reinforcement learning with verifiable rewards. arXiv:2505.24760
58. Sun, Q., Cetin, E., & Tang, Y. (2025). Transformer²: Self-adaptive LLMs. International Conference on Learning Representations (ICLR). arXiv:2501.06252
59. Sun, Y., Cao, Y., Huang, P., Bai, H., Hajishirzi, H., Dziri, N., & Song, D. (2025). RL grokking recipe: How does RL unlock and transfer new algorithms in LLMs?. arXiv:2509.21016
60. Sutton, R. S., Modayil, J., Delp, M., Degris, T., Pilarski, P. M., White, A., & Precup, D. (2011). Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. Proceedings of the 10th International Conference on Autonomous Agents and Multiagent Systems (AAMAS). dl.acm.org/doi/10.5555/2031678.2031726
61. Tang, H., & Berseth, G. (2024). Improving deep reinforcement learning by reducing the chain effect of value and policy churn (CHAIN). Advances in Neural Information Processing Systems (NeurIPS). arXiv:2409.04792
62. Tang, H., Obando-Ceron, J., Castro, P. S., Courville, A., & Berseth, G. (2025). Mitigating plasticity loss in continual reinforcement learning by reducing churn. Proceedings of the 42nd International Conference on Machine Learning (ICML). arXiv:2506.00592
63. Tang, Y., Obando-Ceron, J. S., et al. (2025). AgarCL: The cell must go on — a benchmark for continual reinforcement learning. arXiv:2505.18347
64. Tur, Y., Naghiyev, J., Fang, H., Tsai, W.-C., Duan, J., Fox, D., & Krishna, R. (2026). Recurrent-Depth VLA: Implicit test-time compute scaling of vision–language–action models via latent iterative reasoning. arXiv:2602.07845
65. Verwimp, E., Aljundi, R., Ben-David, S., Bethge, M., Cossu, A., Gepperth, A., Hayes, T. L., et al. (2023). Continual learning: Applications and the road forward. Transactions on Machine Learning Research (TMLR). arXiv:2311.11908
66. Wang, K., Javali, I., Bortkiewicz, M., Trzciński, T., & Eysenbach, B. (2025). 1000 layer networks for self-supervised RL: Scaling depth can enable new goal-reaching capabilities. Advances in Neural Information Processing Systems (NeurIPS). arXiv:2503.14858
67. Wołczyk, M., Zając, M., Pascanu, R., Kuciński, Ł., & Miłoś, P. (2021). Continual World: A robotic benchmark for continual reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 34, 28496–28510. arXiv:2105.10919
68. Wu, F., Xuan, W., Lu, X., Liu, M., Dong, Y., Harchaoui, Z., & Choi, Y. (2025). The invisible leash? Why RLVR may or may not escape its origin. arXiv:2507.14843
69. Yeo, E., Tong, Y., Niu, M., Neubig, G., & Yue, X. (2025). Demystifying long chain-of-thought reasoning in LLMs. Proceedings of the International Conference on Machine Learning (ICML). arXiv:2502.03373
70. Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., & Levine, S. (2020). Meta-World: A benchmark and evaluation for multi-task and meta reinforcement learning. Conference on Robot Learning (CoRL). arXiv:1910.10897
71. Yuan, L., Chen, W., Zhang, Y., Cui, G., Wang, H., You, Z., Ding, N., Liu, Z., Sun, M., & Peng, H. (2025). From f(x) and g(x) to f(g(x)): LLMs learn new skills in RL by composing old ones. arXiv:2509.25123
72. Yue, Y., Chen, Z., Lu, R., Zhao, A., Wang, Z., Song, S., & Huang, G. (2025). Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model?. arXiv:2504.13837
73. Zhang, C., Neubig, G., & Yue, X. (2025). On the interplay of pre-training, mid-training, and RL on reasoning language models. arXiv:2512.07783
74. Zhang, J., Hu, S., Lu, C., Lange, R., & Clune, J. (2025). Darwin Gödel Machine: Open-ended evolution of self-improving agents. arXiv:2505.22954

References by essay

The papers behind each of the twelve essays, in the order they are cited.