--------------------------------------------------------------------------------------------------------------------------
Name : Muhammad Uzair
Reg No : FA20-BSE-081
Subject : Differential Equations
To: Sir Haider Ali
--------------------------------------------------------------------------------------------------------------------------
Application of Differential Equation in Computer Science
In this documentation we will discuss two applications of differential equations in computer science:
Neural Network.
Deep Learning.
Neural Network [1][2]
Differential Equations as a Neural Network Layers
The main idea of artificial neural networks (ANN) is to build up representations for complicated functions
using compositions of relatively simple functions called layers.
ANN ( x)=Ln ∘ L1−1 ∘ … L2 ∘ L1 (x )
A deep neural network is one that has many layers, or many functions composed together.
ANN ( x)=Ln ⋅ L1−1 0 … L2 ∘ L1( x)
Although layers are typically simple functions (e.g. rely (Wx+b)) in general they could be any differentiable
functions.
⃗ Rn ⟶ R n
Lk ( ⃗x ; θ):
⃗ n
X ∈ R input dimensions
n
⃗y=L K ( ⃗x ) ∈ R output dimensions
θ⃗ is a vector
The layer is specified by some finite vector of parameters θ ∈ ℝᵖ. To be practically useful we need to be
able to fit this layer (or layers) to data. This involves defining a cost or loss function which measures the
closeness of the model prediction to the data.
P
C ( θ ) : R → R cost loss function
If our layers are differentiable then we can find the gradient of this cost function ∇C(θ) and use this to find
a local minimum of the cost in an efficient manner.
Here I am considering differential equations models. These systems describe the time evolution of the
state of a system (x) in time, using an expression which involves the derivative of the current state. In
general, they take the form:
d ⃗x
=F ( ⃗x , tj θ⃗ )
dt
¿⃗
n
X∈ R
⃗ p
¿ θ ∈ R state vector∧ parameters
A differential equation fits into our neural network framework, as it takes in some parameters and
produces the solution as output and it is differentiable.
Thus, we can use a differential equation as a layer in a neural network. This is really neat for a few reasons:
Differential equations are the fundamental language of all physical laws.
Outside of physics and chemistry differential equations are an important tool in describing the behavior of
complex systems. Using differential equations models in our neural networks allows these models to be
combined with neural networks approaches.
Building effective neural networks involves choosing a good underlying structure for the network. It is
often easier to think of describing how a function changes in time, then it is to write down a function for
the phenomena. Relatively simple rate laws can turn into very complex behavior.
Building Neural Networks
A fundamental technique in applied mathematics is to find a change of coordinates that converts a difficult
or impossible problem into a simpler one. Perhaps my favorite example of this are the equations for a
circle. If we write down the equation of the unit circle in Cartesian coordinates we can express the
geometric concept of a circle as an implicit function:
x 2+ y 2=1
This gets even worse when we try to get an explicit expression by solving for y.
y=± √ 1−x
2
Now we have to piece this simple function together from two branches of the square root (top and bottom half of
the circle). Beyond aesthetics using this representation of a circle can prevent us from seeing the geometrical
simplicity inherent in a perfect circle. By contrast, if we represent this same mathematical object using polar
coordinates, then the unit circle becomes very simple to work with.
In polar coordinates our unit circle takes the simple form r=1, θ ∈ [0,2π). So in polar coordinates a circle takes the
form of a rectangle (height=1, width=2π). Cartesian coordinates are the natural coordinates for rectangles and polar
coordinates are the natural coordinates for circles. In this article I will apply this line of thinking to constructing
simple neural networks, and build a illustrative example of the importance of choosing the correct coordinates.
Deep Learning [3]
Deep Learning and Differential Equations.
The starting point for our connection between neural networks and differential equations is the neural
differential equation. If we look at a recurrent neural network:
x n+1=x n+ NN ( x n)
In its most general form, then we can think of pulling out a multiplication factor h out of the neural
network, where tn+1=tn+h, and see
x n+1=x n+ ΔtNN (x n)
x n+1−x n h=NN ( x n)
and if we send h→0h→0 then we get:
x ' =NN ( x)
The Augmented Neural Ordinary Differential Equation
Note that not every function can be represented by an ordinary differential equation. Specifically, u(t) is an
R→Rn function which cannot loop over itself except when the solution is cyclic. The reason is because the
flow of the ODE's solution is unique from every time point, and for it to have "two directions" at a point ui
in phase space would have two solutions to the problem.
u ' =f (u , p , t)
The Universal Ordinary Differential Equation
This formulation of the nueral differential equation in terms of a "knowledge-embedded" structure is
leading. If we already knew something about the differential equation, could we use that information in
the differential equation definition itself? This leads us to the idea of the universal differential equation,
which is a differential equation that embeds universal approximates in its definition to allow for learning
arbitrary functions as pieces of the differential equation.
The best way to describe this object is to code up an example. As our example, let's say that we have a
two-state system and know that the second state is defined by a linear ODE. This mean we want to write:
x '=NN ( x , y )
y '= p 1 x+ p 2 y
Discretization’s of Partial Differential Equations
Now let's investigate discretization of partial differential equations. A canonical differential equation to
start with is the Poisson equation. This is the equation:
u xx=f (x)
Where here we have that subscripts correspond to partial derivatives, i.e. this syntax stands for the partial
differential equation:
d 2udx 2=f ( x)
In this case, f is some given data and the goal is to find the u that satisfies this equation. There are two
ways this is generally done:
Expand out the derivative in terms of Taylor series approximations.
Expand out u in terms of some function basis.
Finite Difference Discretization’s
Let's start by looking at Taylor series approximations to the derivative. In this case, we will use what's
known as finite differences. The simplest finite difference approximation is known as the first order
forward difference. This is commonly denoted as:
δ +u=u ( x+ Δx)−u( x )Δx
This looks like a derivative, and we think it's a derivative as Δx→0, but let's show that this approximation is
meaningful. Assume that u is sufficiently nice. Then from a Taylor series we have that:
u( x + Δx)=u ( x)+ Δxu '(x )+ O(Δx 2)
(Here I write (Ax) 2 as Δx2 out of convenience, note that those two terms are not necessarily the same)?
That term on the end is called “Big-O Notation”. What is means is that those terms are asymptotically like
Δx2. If ax is small, then Δx2≪Δx and so we can think of those terms as smaller than any of the terms we
show in the expansion. By simplification notice that we get:
u( x + Δx)−u ( x) Δx=u' ( x)+O( Δx)
This means that δ+ is correct up to first order, where the O (ax) portion that we dropped is the error. Thus
δ+ is a first order approximation.
Notice that the same proof shows that the backwards difference,
δ−u=u(x )−u(x− Δx) Δx
References
1. https://towardsdatascience.com/differential-equations-as-a-neural-network-layer-ac3092632255#:~:text=A
%20differential%20equation%20fits%20into,layer%20in%20a%20neural%20network.
2. https://towardsdatascience.com/work-smarter-not-harder-when-building-neural-networks-6f4aa7c5ee61
3. https://mitmath.github.io/18S096SciML/lecture3/diffeq_ml
