Feedforward Propagation
Contents
Feedforward Propagation#
What is Feedforward Propagation?#
It is a first step in the training of a neural network (after initialization of the weights, which will be covered in the next lecture). The forward direction means going from input to output nodes.
Definition 57
The Feedforward Propagation, also called Forward Pass, is the process consisting of computing and storing all network nodes’ output values, starting with the first hidden layer until the last output layer, using at start either a subset or the entire dataset samples.
Forward propagation thus leads to a list of the neural network predictions for each data instance row used as input. At each node, the computation is the key equation (50) we saw in the previous Section Model Representation, written again for convenience:
Let’s define everything in the next subsection.
Notations#
Let’s say we have the following network with

Fig. 54 . A feedforward neural network with the notation we will use for the forward propagation equations (more in text).
Image from the author#
There are lots of subscripts and upperscripts here. Let’s explain the conventions we will use.
Input data
We saw in Lecture 2 that the dataset in supervised learning can be represented as a matrix
The vector elements are all the features in the data. The upperscript indicates the sample index
Activation units
In a given layer
where subscript is the row of the activation unit in the layer, starting from the top. The upperscript indicates the sample index
Biases
The biases are also column vectors, one for each layer it connects to and of dimension the number of nodes in that layer:
If the last layer is only made of one node like in our example above, then
Weights
Now the weights. You may see in the literature different ways to represent them. In here we use a convention we could write as:
In other words, the weights from the layer
We can actually represent each weight from layer
Note that we do not have an index
Let’s now see how we calculate all the values of the activation units!
Step by step calculations#
General rule for Forward Propagation#
If we rewrite the first layer of inputs for a given sample
then we can write a general rule for computing the outputs of a fully connected layer
This is the general rule for computing all outputs of a fully connected feedforward neural network.
Summary#
Feedforward propagation is the computation of the values of all activation units of a fully connected feedforward neural network.
As the process includes the last layer (output), feedforward propagation also leads to predictions.
These predictions will be compared to the observed values.
Feedforward propagation is a step in the training of a neural network.
The next step of the training is to go ‘backward’, from the output error
Learn More
Very nice animations here illustrating the forward propagation process.
Source: Xinyu You’s course An online deep learning course for humanists