Layers

Collection of Ivy neural network layers as stateful classes.

class ivy.stateful.layers.Conv1D(input_channels, output_channels, filter_size, strides, padding, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, data_format='NWC', dilations=1, device=None, v=None, dtype=None)[source]

Bases: Module

__init__(input_channels, output_channels, filter_size, strides, padding, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, data_format='NWC', dilations=1, device=None, v=None, dtype=None)[source]

1D convolutional layer.

Parameters

  • input_channels – Number of input channels for the layer.

  • output_channels – Number of output channels for the layer.

  • filter_size – Size of the convolutional filter.

  • strides – The stride of the sliding window for each dimension of input.

  • padding – SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.

  • weight_initializer – Initializer for the weights. Default is GlorotUniform.

  • bias_initializer – Initializer for the bias. Default is Zeros.

  • data_format – NWC” or “NCW”. Defaults to “NWC”.

  • dilations – The dilation factor for each dimension of input. (Default value = 1)

  • device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.

  • v – the variables for each of the linear layer, as a container, constructed internally by default.

  • dtype –

    the desired data type of the internal variables to be created if not

    provided. Default is None.

class ivy.stateful.layers.Conv1DTranspose(input_channels, output_channels, filter_size, strides, padding, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, output_shape=None, data_format='NWC', dilations=1, device=None, v=None, dtype=None)[source]

Bases: Module

__init__(input_channels, output_channels, filter_size, strides, padding, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, output_shape=None, data_format='NWC', dilations=1, device=None, v=None, dtype=None)[source]

1D transpose convolutional layer.

Parameters

  • input_channels – Number of input channels for the layer.

  • output_channels – Number of output channels for the layer.

  • filter_size – Size of the convolutional filter.

  • strides – The stride of the sliding window for each dimension of input.

  • padding – SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.

  • weight_initializer – Initializer for the weights. Default is GlorotUniform.

  • bias_initializer – Initializer for the bias. Default is Zeros.

  • output_shape – Shape of the output (Default value = None)

  • data_format – NWC” or “NCW”. Defaults to “NWC”.

  • dilations – The dilation factor for each dimension of input. (Default value = 1)

  • device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.

  • v – the variables for each of the linear layer, as a container, constructed internally by default.

  • dtype –

    the desired data type of the internal variables to be created if not

    provided. Default is None.

class ivy.stateful.layers.Conv2D(input_channels, output_channels, filter_shape, strides, padding, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, data_format='NHWC', dilations=1, device=None, v=None, dtype=None)[source]

Bases: Module

__init__(input_channels, output_channels, filter_shape, strides, padding, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, data_format='NHWC', dilations=1, device=None, v=None, dtype=None)[source]

2D convolutional layer.

Parameters

  • input_channels – Number of input channels for the layer.

  • output_channels – Number of output channels for the layer.

  • filter_shape – Shape of the convolutional filter.

  • strides – The stride of the sliding window for each dimension of input.

  • padding – SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.

  • weight_initializer – Initializer for the weights. Default is GlorotUniform.

  • bias_initializer – Initializer for the bias. Default is Zeros.

  • data_format – NHWC” or “NCHW”. Defaults to “NHWC”.

  • dilations – The dilation factor for each dimension of input. (Default value = 1)

  • device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.

  • v – the variables for each of the linear layer, as a container, constructed internally by default.

  • dtype –

    the desired data type of the internal variables to be created if not

    provided. Default is None.

class ivy.stateful.layers.Conv2DTranspose(input_channels, output_channels, filter_shape, strides, padding, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, output_shape=None, data_format='NHWC', dilations=1, device=None, v=None, dtype=None)[source]

Bases: Module

__init__(input_channels, output_channels, filter_shape, strides, padding, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, output_shape=None, data_format='NHWC', dilations=1, device=None, v=None, dtype=None)[source]

2D convolutional transpose layer.

Parameters

  • input_channels – Number of input channels for the layer.

  • output_channels – Number of output channels for the layer.

  • filter_shape – Shape of the convolutional filter.

  • strides – The stride of the sliding window for each dimension of input.

  • padding – SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.

  • weight_initializer – Initializer for the weights. Default is GlorotUniform.

  • bias_initializer – Initializer for the bias. Default is Zeros.

  • output_shape – Shape of the output (Default value = None)

  • data_format – NHWC” or “NCHW”. Defaults to “NHWC”.

  • dilations – The dilation factor for each dimension of input. (Default value = 1)

  • device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.

  • v – the variables for each of the linear layer, as a container, constructed internally by default.

  • dtype –

    the desired data type of the internal variables to be created if not

    provided. Default is None.

class ivy.stateful.layers.Conv3D(input_channels, output_channels, filter_shape, strides, padding, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, data_format='NDHWC', dilations=1, device=None, v=None, dtype=None)[source]

Bases: Module

__init__(input_channels, output_channels, filter_shape, strides, padding, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, data_format='NDHWC', dilations=1, device=None, v=None, dtype=None)[source]

3D convolutional layer.

Parameters

  • input_channels – Number of input channels for the layer.

  • output_channels – Number of output channels for the layer.

  • filter_shape – Shape of the convolutional filter.

  • strides – The stride of the sliding window for each dimension of input.

  • padding – SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.

  • weight_initializer – Initializer for the weights. Default is GlorotUniform.

  • bias_initializer – Initializer for the bias. Default is Zeros.

  • data_format – NDHWC” or “NCDHW”. Defaults to “NDHWC”.

  • dilations – The dilation factor for each dimension of input. (Default value = 1)

  • device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.

  • v – the variables for each of the linear layer, as a container, constructed internally by default.

  • dtype –

    the desired data type of the internal variables to be created if not

    provided. Default is None.

class ivy.stateful.layers.Conv3DTranspose(input_channels, output_channels, filter_shape, strides, padding, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, output_shape=None, data_format='NDHWC', dilations=1, device=None, v=None, dtype=None)[source]

Bases: Module

__init__(input_channels, output_channels, filter_shape, strides, padding, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, output_shape=None, data_format='NDHWC', dilations=1, device=None, v=None, dtype=None)[source]

3D convolutional transpose layer.

Parameters

  • input_channels – Number of input channels for the layer.

  • output_channels – Number of output channels for the layer.

  • filter_shape – Shape of the convolutional filter.

  • strides – The stride of the sliding window for each dimension of input.

  • padding – SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.

  • weight_initializer – Initializer for the weights. Default is GlorotUniform.

  • bias_initializer – Initializer for the bias. Default is Zeros.

  • output_shape – Shape of the output (Default value = None)

  • data_format – NDHWC” or “NCDHW”. Defaults to “NDHWC”.

  • dilations – The dilation factor for each dimension of input. (Default value = 1)

  • device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.

  • v – the variables for each of the linear layer, as a container, constructed internally by default.

  • dtype –

    the desired data type of the internal variables to be created if not

    provided. Default is None.

class ivy.stateful.layers.DepthwiseConv2D(num_channels, filter_shape, strides, padding, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, data_format='NHWC', dilations=1, device=None, v=None, dtype=None)[source]

Bases: Module

__init__(num_channels, filter_shape, strides, padding, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, data_format='NHWC', dilations=1, device=None, v=None, dtype=None)[source]

Depthwise 2D convolutional layer.

Parameters

  • num_channels – Number of input channels for the layer.

  • filter_shape – Shape of the convolutional filter.

  • strides – The stride of the sliding window for each dimension of input.

  • padding – SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.

  • weight_initializer – Initializer for the weights. Default is GlorotUniform.

  • bias_initializer – Initializer for the bias. Default is Zeros.

  • data_format – NHWC” or “NCHW”. Defaults to “NHWC”.

  • dilations – The dilation factor for each dimension of input. (Default value = 1)

  • device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.

  • v – the variables for each of the linear layer, as a container, constructed internally by default.

  • dtype –

    the desired data type of the internal variables to be created if not

    provided. Default is None.

class ivy.stateful.layers.Dropout(prob, scale=True, dtype=None)[source]

Bases: Module

__init__(prob, scale=True, dtype=None)[source]

Dropout layer. The layer randomly zeroes some of the elements of the input tensor with probability p using samples from a Bernoull distribution.

Parameters

  • prob – The probability of zeroing out each array element.

  • scale – Whether to scale the output by 1/(1-prob), default is True.

  • device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.

  • dtype – the desired data type of the internal variables to be created. Default is None.

class ivy.stateful.layers.LSTM(input_channels, output_channels, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, num_layers=1, return_sequence=True, return_state=True, device=None, v=None, dtype=None)[source]

Bases: Module

__init__(input_channels, output_channels, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, num_layers=1, return_sequence=True, return_state=True, device=None, v=None, dtype=None)[source]

LSTM layer, which is a set of stacked lstm cells.

Parameters

  • input_channels – Number of input channels for the layer

  • output_channels – Number of output channels for the layer

  • weight_initializer – Initializer for the weights. Default is GlorotUniform.

  • num_layers – Number of lstm cells in the lstm layer, default is 1.

  • return_sequence – Whether or not to return the entire output sequence, or just the latest timestep. Default is True.

  • return_state – Whether or not to return the latest hidden and cell states. Default is True.

  • device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.

  • v – the variables for each of the lstm cells, as a container, constructed internally by default.

  • dtype –

    the desired data type of the internal variables to be created if not

    provided. Default is None.

get_initial_state(batch_shape, dtype=None)[source]

Get the initial state of the hidden and cell states, if not provided explicitly

Parameters

  • batch_shape –

  • dtype –

    the desired data type of the internal variables to be created if not

    provided. Default is None.

class ivy.stateful.layers.Linear(input_channels, output_channels, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, with_bias=True, device=None, v=None, dtype=None)[source]

Bases: Module

__init__(input_channels, output_channels, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, with_bias=True, device=None, v=None, dtype=None)[source]

Linear layer, also referred to as dense or fully connected. The layer receives tensors with input_channels last dimension and returns a new tensor with output_channels last dimension, following matrix multiplication with the weight matrix and addition with the bias vector.

Parameters

  • input_channels – Number of input channels for the layer.

  • output_channels – Number of output channels for the layer.

  • weight_initializer – Initializer for the weights. Default is GlorotUniform.

  • bias_initializer – Initializer for the bias. Default is Zeros.

  • with_bias – Whether or not to include a bias term, default is True.

  • device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.

  • v – the variables for the linear layer, as a container, constructed internally by default.

  • dtype –

    the desired data type of the internal variables to be created if not

    provided. Default is None.

class ivy.stateful.layers.MultiHeadAttention(query_dim, num_heads=8, head_dim=64, dropout_rate=0.0, context_dim=None, scale=None, with_to_q_fn=True, with_to_kv_fn=True, with_to_out_fn=True, device=None, v=None, build_mode='on_init', dtype=None)[source]

Bases: Module

__init__(query_dim, num_heads=8, head_dim=64, dropout_rate=0.0, context_dim=None, scale=None, with_to_q_fn=True, with_to_kv_fn=True, with_to_out_fn=True, device=None, v=None, build_mode='on_init', dtype=None)[source]

Multi Head Attention layer.

Parameters

  • query_dim – The dimension of the attention queries.

  • num_heads – Number of attention heads. Default is 8.

  • head_dim – The dimension of each of the heads. Default is 64.

  • dropout_rate – The rate of dropout. Default is 0.

  • context_dim – The dimension of the context array. Default is None, in which case the query dim is used.

  • scale – The value by which to scale the query-key similarity measure. Default is head_dim^-0.5

  • with_to_q_fn – Whether to include fully connected mapping from input x to queries. Default is True.

  • with_to_kv_fn – Whether to include fully connected mapping from input context to keys and values. Default is True.

  • with_to_out_fn – Whether to include fully connected mapping from output scaled dot-product attention to final output. Default is True.

  • device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.

  • v – the variables for the attention layer, as a container, constructed internally by default.

  • build_mode – How the Module is built, either on initialization (now), explicitly by the user by calling build(), or the first time the __call__ method is run. Default is on initialization.

  • dtype –

    the desired data type of the internal variables to be created if not

    provided. Default is None.