Writing
Blog
Hyper-dual numbers and higher-order derivatives
Extending dual numbers with multiple independent infinitesimals gives a clean, recursive route to higher-order automatic differentiation, connecting to the labeled rooted trees of Faà di Bruno's formula.
On rooted trees and differentiation
The chain rule for higher order derivatives boasts a wealth of beautiful mathematical structure touching the theory of special rooted trees, group theory, combinatorics of integer partitions, order theory, and many others.
Dual numbers
There is a generalisation of the complex numbers where i²=0 instead of i²=−1. Functions extended to this dual number system have the curious property that we can read off their derivatives if we evaluate them at the dual number x + i.
Reversible optimisers
Reversible neural architectures have been a popular research area in the last few years, but reversibility is also built into many modern day neural optimisers, perhaps serendipitously.
The invention of randomness
Of all the many inventions, concepts, and innovations that have shaped our society today and that we take for granted, there is one that stands out as being somewhat overlooked: randomness.