Layers#

Collection of Ivy neural network layers as stateful classes.

class ivy.stateful.layers.AvgPool2D(kernel_size, stride, padding, /, *, data_format='NHWC', device=None, v=None, dtype=None)[source]#

Bases: Module

__init__(kernel_size, stride, padding, /, *, data_format='NHWC', device=None, v=None, dtype=None)[source]#

Class for applying Average Pooling over a mini-batch of inputs.

Parameters:

  • kernel_size – The size of the window to take a max over.

  • stride – The stride of the window. Default value: 1

  • padding – Implicit zero padding to be added on both sides.

  • device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’

class ivy.stateful.layers.AvgPool3D(kernel_size, strides, padding, /, *, data_format='NDHWC', count_include_pad=False, ceil_mode=False, divisor_override=None)[source]#

Bases: Module

__init__(kernel_size, strides, padding, /, *, data_format='NDHWC', count_include_pad=False, ceil_mode=False, divisor_override=None)[source]#

Class for applying Average Pooling over a mini-batch of inputs.

Parameters:

  • kernel_size – The size of the window to take a max over.

  • stride – The stride of the window. Default value: 1

  • padding – Implicit zero padding to be added on both sides.

  • data_format – NDHWC” or “NCDHW”. Defaults to “NDHWC”.

  • count_include_pad – Whether to include padding in the averaging calculation.

  • ceil_mode – Whether to use ceil or floor for creating the output shape.

  • divisor_override – If specified, it will be used as divisor, otherwise kernel_size will be used.

class ivy.stateful.layers.Conv1D(input_channels, output_channels, filter_size, strides, padding, /, *, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, with_bias=True, data_format='NWC', dilations=1, device=None, v=None, dtype=None)[source]#

Bases: Module

__init__(input_channels, output_channels, filter_size, strides, padding, /, *, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, with_bias=True, data_format='NWC', dilations=1, device=None, v=None, dtype=None)[source]#

1D convolutional layer.

Parameters:

  • input_channels – Number of input channels for the layer.

  • output_channels – Number of output channels for the layer.

  • filter_size – Size of the convolutional filter.

  • strides – The stride of the sliding window for each dimension of input.

  • padding – SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.

  • weight_initializer – Initializer for the weights. Default is GlorotUniform.

  • bias_initializer – Initializer for the bias. Default is Zeros.

  • with_bias – Whether or not to include a bias term, default is True.

  • data_format – NWC” or “NCW”. Defaults to “NWC”.

  • dilations – The dilation factor for each dimension of input. (Default value = 1)

  • device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.

  • v – the variables for each of the conv layer, as a container, constructed internally by default.

  • dtype –

    the desired data type of the internal variables to be created if not

    provided. Default is None.

class ivy.stateful.layers.Conv1DTranspose(input_channels, output_channels, filter_size, strides, padding, /, *, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, with_bias=True, output_shape=None, data_format='NWC', dilations=1, device=None, v=None, dtype=None)[source]#

Bases: Module

__init__(input_channels, output_channels, filter_size, strides, padding, /, *, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, with_bias=True, output_shape=None, data_format='NWC', dilations=1, device=None, v=None, dtype=None)[source]#

1D transpose convolutional layer.

Parameters:

  • input_channels – Number of input channels for the layer.

  • output_channels – Number of output channels for the layer.

  • filter_size – Size of the convolutional filter.

  • strides – The stride of the sliding window for each dimension of input.

  • padding – SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.

  • weight_initializer – Initializer for the weights. Default is GlorotUniform.

  • bias_initializer – Initializer for the bias. Default is Zeros.

  • with_bias – Whether or not to include a bias term, default is True.

  • output_shape – Shape of the output (Default value = None)

  • data_format – NWC” or “NCW”. Defaults to “NWC”.

  • dilations – The dilation factor for each dimension of input. (Default value = 1)

  • device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.

  • v – the variables for each of the conv layer, as a container, constructed internally by default.

  • dtype –

    the desired data type of the internal variables to be created if not

    provided. Default is None.

class ivy.stateful.layers.Conv2D(input_channels, output_channels, filter_shape, strides, padding, /, *, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, with_bias=True, data_format='NHWC', dilations=1, device=None, v=None, dtype=None)[source]#

Bases: Module

__init__(input_channels, output_channels, filter_shape, strides, padding, /, *, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, with_bias=True, data_format='NHWC', dilations=1, device=None, v=None, dtype=None)[source]#

2D convolutional layer.

Parameters:

  • input_channels – Number of input channels for the layer.

  • output_channels – Number of output channels for the layer.

  • filter_shape – Shape of the convolutional filter.

  • strides – The stride of the sliding window for each dimension of input.

  • padding – SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.

  • weight_initializer – Initializer for the weights. Default is GlorotUniform.

  • bias_initializer – Initializer for the bias. Default is Zeros.

  • with_bias – Whether or not to include a bias term, default is True.

  • data_format – NHWC” or “NCHW”. Defaults to “NHWC”.

  • dilations – The dilation factor for each dimension of input. (Default value = 1)

  • device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.

  • v – the variables for each of the conv layer, as a container, constructed internally by default.

  • dtype –

    the desired data type of the internal variables to be created if not

    provided. Default is None.

class ivy.stateful.layers.Conv2DTranspose(input_channels, output_channels, filter_shape, strides, padding, /, *, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, with_bias=True, output_shape=None, data_format='NHWC', dilations=1, device=None, v=None, dtype=None)[source]#

Bases: Module

__init__(input_channels, output_channels, filter_shape, strides, padding, /, *, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, with_bias=True, output_shape=None, data_format='NHWC', dilations=1, device=None, v=None, dtype=None)[source]#

2D convolutional transpose layer.

Parameters:

  • input_channels – Number of input channels for the layer.

  • output_channels – Number of output channels for the layer.

  • filter_shape – Shape of the convolutional filter.

  • strides – The stride of the sliding window for each dimension of input.

  • padding – SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.

  • weight_initializer – Initializer for the weights. Default is GlorotUniform.

  • bias_initializer – Initializer for the bias. Default is Zeros.

  • with_bias – Whether or not to include a bias term, default is True.

  • output_shape – Shape of the output (Default value = None)

  • data_format – NHWC” or “NCHW”. Defaults to “NHWC”.

  • dilations – The dilation factor for each dimension of input. (Default value = 1)

  • device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.

  • v – the variables for each of the conv layer, as a container, constructed internally by default.

  • dtype –

    the desired data type of the internal variables to be created if not

    provided. Default is None.

class ivy.stateful.layers.Conv3D(input_channels, output_channels, filter_shape, strides, padding, /, *, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, with_bias=True, data_format='NDHWC', dilations=1, device=None, v=None, dtype=None)[source]#

Bases: Module

__init__(input_channels, output_channels, filter_shape, strides, padding, /, *, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, with_bias=True, data_format='NDHWC', dilations=1, device=None, v=None, dtype=None)[source]#

3D convolutional layer.

Parameters:

  • input_channels – Number of input channels for the layer.

  • output_channels – Number of output channels for the layer.

  • filter_shape – Shape of the convolutional filter.

  • strides – The stride of the sliding window for each dimension of input.

  • padding – SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.

  • weight_initializer – Initializer for the weights. Default is GlorotUniform.

  • bias_initializer – Initializer for the bias. Default is Zeros.

  • with_bias – Whether or not to include a bias term, default is True.

  • data_format – NDHWC” or “NCDHW”. Defaults to “NDHWC”.

  • dilations – The dilation factor for each dimension of input. (Default value = 1)

  • device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.

  • v – the variables for each of the conv layer, as a container, constructed internally by default.

  • dtype –

    the desired data type of the internal variables to be created if not

    provided. Default is None.

class ivy.stateful.layers.Conv3DTranspose(input_channels, output_channels, filter_shape, strides, padding, /, *, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, with_bias=True, output_shape=None, data_format='NDHWC', dilations=1, device=None, v=None, dtype=None)[source]#

Bases: Module

__init__(input_channels, output_channels, filter_shape, strides, padding, /, *, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, with_bias=True, output_shape=None, data_format='NDHWC', dilations=1, device=None, v=None, dtype=None)[source]#

3D convolutional transpose layer.

Parameters:

  • input_channels – Number of input channels for the layer.

  • output_channels – Number of output channels for the layer.

  • filter_shape – Shape of the convolutional filter.

  • strides – The stride of the sliding window for each dimension of input.

  • padding – SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.

  • weight_initializer – Initializer for the weights. Default is GlorotUniform.

  • bias_initializer – Initializer for the bias. Default is Zeros.

  • with_bias – Whether or not to include a bias term, default is True.

  • output_shape – Shape of the output (Default value = None)

  • data_format – NDHWC” or “NCDHW”. Defaults to “NDHWC”.

  • dilations – The dilation factor for each dimension of input. (Default value = 1)

  • device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.

  • v – the variables for each of the conv layer, as a container, constructed internally by default.

  • dtype –

    the desired data type of the internal variables to be created if not

    provided. Default is None.

class ivy.stateful.layers.DepthwiseConv2D(num_channels, filter_shape, strides, padding, /, *, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, with_bias=True, data_format='NHWC', dilations=1, device=None, v=None, dtype=None)[source]#

Bases: Module

__init__(num_channels, filter_shape, strides, padding, /, *, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, with_bias=True, data_format='NHWC', dilations=1, device=None, v=None, dtype=None)[source]#

Depthwise 2D convolutional layer.

Parameters:

  • num_channels – Number of input channels for the layer.

  • filter_shape – Shape of the convolutional filter.

  • strides – The stride of the sliding window for each dimension of input.

  • padding – SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.

  • weight_initializer – Initializer for the weights. Default is GlorotUniform.

  • bias_initializer – Initializer for the bias. Default is Zeros.

  • with_bias – Whether or not to include a bias term, default is True.

  • data_format – NHWC” or “NCHW”. Defaults to “NHWC”.

  • dilations – The dilation factor for each dimension of input. (Default value = 1)

  • device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.

  • v – the variables for each of the conv layer, as a container, constructed internally by default.

  • dtype –

    the desired data type of the internal variables to be created if not

    provided. Default is None.

class ivy.stateful.layers.Dropout(prob, scale=True, dtype=None, training=True)[source]#

Bases: Module

__init__(prob, scale=True, dtype=None, training=True)[source]#

Dropout layer. The layer randomly zeroes some of the elements of the input tensor with probability p using samples from a Bernoull distribution.

Parameters:

  • prob – The probability of zeroing out each array element.

  • scale (bool) – Whether to scale the output by 1/(1-prob), default is True. (default: True)

  • dtype – the desired data type of the internal variables to be created. Default is None.

  • training (bool) – Turn on dropout if training, turn off otherwise. Default is True. (default: True)

class ivy.stateful.layers.LSTM(input_channels, output_channels, /, *, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, num_layers=1, return_sequence=True, return_state=True, device=None, v=None, dtype=None)[source]#

Bases: Module

__init__(input_channels, output_channels, /, *, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, num_layers=1, return_sequence=True, return_state=True, device=None, v=None, dtype=None)[source]#

LSTM layer, which is a set of stacked lstm cells.

Parameters:

  • input_channels – Number of input channels for the layer

  • output_channels – Number of output channels for the layer

  • weight_initializer – Initializer for the weights. Default is GlorotUniform.

  • num_layers – Number of lstm cells in the lstm layer, default is 1.

  • return_sequence – Whether or not to return the entire output sequence, or just the latest timestep. Default is True.

  • return_state – Whether or not to return the latest hidden and cell states. Default is True.

  • device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.

  • v – the variables for each of the lstm cells, as a container, constructed internally by default.

  • dtype –

    the desired data type of the internal variables to be created if not

    provided. Default is None.

get_initial_state(batch_shape, dtype=None)[source]#

Get the initial state of the hidden and cell states, if not provided explicitly.

Parameters:

  • batch_shape –

  • dtype –

    the desired data type of the internal variables to be created if not

    provided. Default is None.

class ivy.stateful.layers.Linear(input_channels, output_channels, /, *, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, with_bias=True, device=None, v=None, dtype=None)[source]#

Bases: Module

__init__(input_channels, output_channels, /, *, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, with_bias=True, device=None, v=None, dtype=None)[source]#

Linear layer, also referred to as dense or fully connected. The layer receives tensors with input_channels last dimension and returns a new tensor with output_channels last dimension, following matrix multiplication with the weight matrix and addition with the bias vector.

Parameters:

  • input_channels – Number of input channels for the layer.

  • output_channels – Number of output channels for the layer.

  • weight_initializer – Initializer for the weights. Default is GlorotUniform.

  • bias_initializer – Initializer for the bias. Default is Zeros.

  • with_bias – Whether or not to include a bias term, default is True.

  • device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.

  • v – the variables for the linear layer, as a container, constructed internally by default.

  • dtype –

    the desired data type of the internal variables to be created if not

    provided. Default is None.

class ivy.stateful.layers.MaxPool1D(kernel_size, stride, padding, /, *, data_format='NWC', device=None, v=None, dtype=None)[source]#

Bases: Module

__init__(kernel_size, stride, padding, /, *, data_format='NWC', device=None, v=None, dtype=None)[source]#

Class for applying Max Pooling over a mini-batch of inputs.

Parameters:

  • kernel_size – The size of the window to take a max over.

  • stride – The stride of the window. Default value: 1

  • padding – Implicit zero padding to be added on both sides.

  • device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’

class ivy.stateful.layers.MaxPool2D(kernel_size, stride, padding, /, *, data_format='NHWC', device=None, v=None, dtype=None)[source]#

Bases: Module

__init__(kernel_size, stride, padding, /, *, data_format='NHWC', device=None, v=None, dtype=None)[source]#

Class for applying Max Pooling over a mini-batch of inputs.

Parameters:

  • kernel_size – The size of the window to take a max over.

  • stride – The stride of the window. Default value: 1

  • padding – Implicit zero padding to be added on both sides.

  • device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’

class ivy.stateful.layers.MaxPool3D(kernel_size, stride, padding, /, *, data_format='NDHWC', device=None, dtype=None)[source]#

Bases: Module

__init__(kernel_size, stride, padding, /, *, data_format='NDHWC', device=None, dtype=None)[source]#

Class for applying 3D Max Pooling over 5D inputs.

Parameters:

  • kernel_size – The size of the window to take a max over.

  • stride – The stride of the window.

  • padding – Implicit zero padding to be added on both sides.

class ivy.stateful.layers.MultiHeadAttention(query_dim, /, *, num_heads=8, head_dim=64, dropout_rate=0.0, context_dim=None, scale=None, with_to_q_fn=True, with_to_kv_fn=True, with_to_out_fn=True, device=None, v=None, build_mode='on_init', dtype=None)[source]#

Bases: Module

__init__(query_dim, /, *, num_heads=8, head_dim=64, dropout_rate=0.0, context_dim=None, scale=None, with_to_q_fn=True, with_to_kv_fn=True, with_to_out_fn=True, device=None, v=None, build_mode='on_init', dtype=None)[source]#

Multi Head Attention layer.

Parameters:

  • query_dim – The dimension of the attention queries.

  • num_heads – Number of attention heads. Default is 8.

  • head_dim – The dimension of each of the heads. Default is 64.

  • dropout_rate – The rate of dropout. Default is 0.

  • context_dim – The dimension of the context array. Default is None, in which case the query dim is used.

  • scale – The value by which to scale the query-key similarity measure. Default is head_dim^-0.5

  • with_to_q_fn – Whether to include fully connected mapping from input x to queries. Default is True.

  • with_to_kv_fn – Whether to include fully connected mapping from input context to keys and values. Default is True.

  • with_to_out_fn – Whether to include fully connected mapping from output scaled dot-product attention to final output. Default is True.

  • device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.

  • v – the variables for the attention layer, as a container, constructed internally by default.

  • build_mode – How the Module is built, either on initialization (now), explicitly by the user by calling build(), or the first time the __call__ method is run. Default is on initialization.

  • dtype –

    the desired data type of the internal variables to be created if not

    provided. Default is None.

This should have hopefully given you an overview of the layers submodule, if you have any questions, please feel free to reach out on our discord in the layers channel or in the layers forum!