Initialisers

Xavier Glorot

class DLL.DeepLearning.Initialisers.Xavier_Normal

Bases: Initialiser

The Xavier Glorot normal initialiser. Xavier Glorot initialiser should be used for tanh, sigmoid, softmax or other activations, which are approximately linear close to origin.

initialise(shape, data_type=torch.float32, device=device(type='cpu'))

Initialises a tensor of the wanted shape with values in \(N(0, \sigma^2)\), where \(\sigma = \sqrt{\frac{2}{d_{\text{input}} + d_{\text{output}}}}\).

Parameters:

• shape (torch.Size) – The shape of the wanted tensor.

• data_type (torch.dtype, optional) – The data type used in the returned tensor. Defaults to torch.float32.

• device (torch.device, optional) – The device of the tensor. Determines if the computation is made using the gpu or the cpu. Defaults to torch.device(“cpu”).

class DLL.DeepLearning.Initialisers.Xavier_Uniform

Bases: Initialiser

The Xavier Glorot uniform initialiser. Xavier Glorot initialiser should be used for tanh, sigmoid, softmax or other activations, which are approximately linear close to origin.

initialise(shape, data_type=torch.float32, device=device(type='cpu'))

Initialises a tensor of the wanted shape with values in \(U(-a, a)\), where \(a = \sqrt{\frac{6}{d_{\text{input}} + d_{\text{output}}}}\).

Parameters:

• shape (torch.Size) – The shape of the wanted tensor.

• data_type (torch.dtype, optional) – The data type used in the returned tensor. Defaults to torch.float32.

• device (torch.device, optional) – The device of the tensor. Determines if the computation is made using the gpu or the cpu. Defaults to torch.device(“cpu”).

Kaiming He

class DLL.DeepLearning.Initialisers.Kaiming_Normal(mode='input_dim')

Bases: Initialiser

The Kaiming He normal initialiser. Kaiming He initialiser should be used for the ReLU or any other activation, which is nonlinear close to origin.

Parameters:

mode (str, optional) – Determines if the variance is constant in the forward or backward propagation. If “input_dim”, variance is constant in forward propagation, while if “output_dim”, the variance is constant in back propagation. Defaults to “input_dim”.

initialise(shape, data_type=torch.float32, device=device(type='cpu'))

Initialises a tensor of the wanted shape with values in \(N(0, \sigma^2)\), where \(\sigma = \sqrt{\frac{2}{d_{\text{input}} + d_{\text{output}}}}\).

Parameters:

• shape (torch.Size) – The shape of the wanted tensor.

• data_type (torch.dtype, optional) – The data type used in the returned tensor. Defaults to torch.float32.

• device (torch.device, optional) – The device of the tensor. Determines if the computation is made using the gpu or the cpu. Defaults to torch.device(“cpu”).

class DLL.DeepLearning.Initialisers.Kaiming_Uniform(mode='input_dim')

Bases: Initialiser

The Kaiming He uniform initialiser. Kaiming He initialiser should be used for the ReLU or any other activation, which is nonlinear close to origin.

Parameters:

mode (str, optional) – Determines if the variance is constant in the forward or backward propagation. If “input_dim”, variance is constant in forward propagation, while if “output_dim”, the variance is constant in back propagation. Defaults to “input_dim”.

initialise(shape, data_type=torch.float32, device=device(type='cpu'))

Initialises a tensor of the wanted shape with values in \(U(-a, a)\), where \(a = \sqrt{\frac{6}{d}}\).

Parameters:

• shape (torch.Size) – The shape of the wanted tensor.

• data_type (torch.dtype, optional) – The data type used in the returned tensor. Defaults to torch.float32.

• device (torch.device, optional) – The device of the tensor. Determines if the computation is made using the gpu or the cpu. Defaults to torch.device(“cpu”).