Layers
Collection of Ivy neural network layers as stateful classes.
- class ivy.stateful.layers.Conv1D(input_channels, output_channels, filter_size, strides, padding, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, data_format='NWC', dilations=1, device=None, v=None, dtype=None)[source]
Bases:
Module
- __init__(input_channels, output_channels, filter_size, strides, padding, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, data_format='NWC', dilations=1, device=None, v=None, dtype=None)[source]
1D convolutional layer.
- Parameters
input_channels – Number of input channels for the layer.
output_channels – Number of output channels for the layer.
filter_size – Size of the convolutional filter.
strides – The stride of the sliding window for each dimension of input.
padding – SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.
weight_initializer – Initializer for the weights. Default is GlorotUniform.
bias_initializer – Initializer for the bias. Default is Zeros.
data_format – NWC” or “NCW”. Defaults to “NWC”.
dilations – The dilation factor for each dimension of input. (Default value = 1)
device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.
v – the variables for each of the linear layer, as a container, constructed internally by default.
dtype –
- the desired data type of the internal variables to be created if not
provided. Default is None.
- class ivy.stateful.layers.Conv1DTranspose(input_channels, output_channels, filter_size, strides, padding, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, output_shape=None, data_format='NWC', dilations=1, device=None, v=None, dtype=None)[source]
Bases:
Module
- __init__(input_channels, output_channels, filter_size, strides, padding, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, output_shape=None, data_format='NWC', dilations=1, device=None, v=None, dtype=None)[source]
1D transpose convolutional layer.
- Parameters
input_channels – Number of input channels for the layer.
output_channels – Number of output channels for the layer.
filter_size – Size of the convolutional filter.
strides – The stride of the sliding window for each dimension of input.
padding – SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.
weight_initializer – Initializer for the weights. Default is GlorotUniform.
bias_initializer – Initializer for the bias. Default is Zeros.
output_shape – Shape of the output (Default value = None)
data_format – NWC” or “NCW”. Defaults to “NWC”.
dilations – The dilation factor for each dimension of input. (Default value = 1)
device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.
v – the variables for each of the linear layer, as a container, constructed internally by default.
dtype –
- the desired data type of the internal variables to be created if not
provided. Default is None.
- class ivy.stateful.layers.Conv2D(input_channels, output_channels, filter_shape, strides, padding, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, data_format='NHWC', dilations=1, device=None, v=None, dtype=None)[source]
Bases:
Module
- __init__(input_channels, output_channels, filter_shape, strides, padding, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, data_format='NHWC', dilations=1, device=None, v=None, dtype=None)[source]
2D convolutional layer.
- Parameters
input_channels – Number of input channels for the layer.
output_channels – Number of output channels for the layer.
filter_shape – Shape of the convolutional filter.
strides – The stride of the sliding window for each dimension of input.
padding – SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.
weight_initializer – Initializer for the weights. Default is GlorotUniform.
bias_initializer – Initializer for the bias. Default is Zeros.
data_format – NHWC” or “NCHW”. Defaults to “NHWC”.
dilations – The dilation factor for each dimension of input. (Default value = 1)
device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.
v – the variables for each of the linear layer, as a container, constructed internally by default.
dtype –
- the desired data type of the internal variables to be created if not
provided. Default is None.
- class ivy.stateful.layers.Conv2DTranspose(input_channels, output_channels, filter_shape, strides, padding, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, output_shape=None, data_format='NHWC', dilations=1, device=None, v=None, dtype=None)[source]
Bases:
Module
- __init__(input_channels, output_channels, filter_shape, strides, padding, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, output_shape=None, data_format='NHWC', dilations=1, device=None, v=None, dtype=None)[source]
2D convolutional transpose layer.
- Parameters
input_channels – Number of input channels for the layer.
output_channels – Number of output channels for the layer.
filter_shape – Shape of the convolutional filter.
strides – The stride of the sliding window for each dimension of input.
padding – SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.
weight_initializer – Initializer for the weights. Default is GlorotUniform.
bias_initializer – Initializer for the bias. Default is Zeros.
output_shape – Shape of the output (Default value = None)
data_format – NHWC” or “NCHW”. Defaults to “NHWC”.
dilations – The dilation factor for each dimension of input. (Default value = 1)
device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.
v – the variables for each of the linear layer, as a container, constructed internally by default.
dtype –
- the desired data type of the internal variables to be created if not
provided. Default is None.
- class ivy.stateful.layers.Conv3D(input_channels, output_channels, filter_shape, strides, padding, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, data_format='NDHWC', dilations=1, device=None, v=None, dtype=None)[source]
Bases:
Module
- __init__(input_channels, output_channels, filter_shape, strides, padding, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, data_format='NDHWC', dilations=1, device=None, v=None, dtype=None)[source]
3D convolutional layer.
- Parameters
input_channels – Number of input channels for the layer.
output_channels – Number of output channels for the layer.
filter_shape – Shape of the convolutional filter.
strides – The stride of the sliding window for each dimension of input.
padding – SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.
weight_initializer – Initializer for the weights. Default is GlorotUniform.
bias_initializer – Initializer for the bias. Default is Zeros.
data_format – NDHWC” or “NCDHW”. Defaults to “NDHWC”.
dilations – The dilation factor for each dimension of input. (Default value = 1)
device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.
v – the variables for each of the linear layer, as a container, constructed internally by default.
dtype –
- the desired data type of the internal variables to be created if not
provided. Default is None.
- class ivy.stateful.layers.Conv3DTranspose(input_channels, output_channels, filter_shape, strides, padding, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, output_shape=None, data_format='NDHWC', dilations=1, device=None, v=None, dtype=None)[source]
Bases:
Module
- __init__(input_channels, output_channels, filter_shape, strides, padding, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, output_shape=None, data_format='NDHWC', dilations=1, device=None, v=None, dtype=None)[source]
3D convolutional transpose layer.
- Parameters
input_channels – Number of input channels for the layer.
output_channels – Number of output channels for the layer.
filter_shape – Shape of the convolutional filter.
strides – The stride of the sliding window for each dimension of input.
padding – SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.
weight_initializer – Initializer for the weights. Default is GlorotUniform.
bias_initializer – Initializer for the bias. Default is Zeros.
output_shape – Shape of the output (Default value = None)
data_format – NDHWC” or “NCDHW”. Defaults to “NDHWC”.
dilations – The dilation factor for each dimension of input. (Default value = 1)
device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.
v – the variables for each of the linear layer, as a container, constructed internally by default.
dtype –
- the desired data type of the internal variables to be created if not
provided. Default is None.
- class ivy.stateful.layers.DepthwiseConv2D(num_channels, filter_shape, strides, padding, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, data_format='NHWC', dilations=1, device=None, v=None, dtype=None)[source]
Bases:
Module
- __init__(num_channels, filter_shape, strides, padding, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, data_format='NHWC', dilations=1, device=None, v=None, dtype=None)[source]
Depthwise 2D convolutional layer.
- Parameters
num_channels – Number of input channels for the layer.
filter_shape – Shape of the convolutional filter.
strides – The stride of the sliding window for each dimension of input.
padding – SAME” or “VALID” indicating the algorithm, or list indicating the per-dimension paddings.
weight_initializer – Initializer for the weights. Default is GlorotUniform.
bias_initializer – Initializer for the bias. Default is Zeros.
data_format – NHWC” or “NCHW”. Defaults to “NHWC”.
dilations – The dilation factor for each dimension of input. (Default value = 1)
device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.
v – the variables for each of the linear layer, as a container, constructed internally by default.
dtype –
- the desired data type of the internal variables to be created if not
provided. Default is None.
- class ivy.stateful.layers.Dropout(prob, scale=True, dtype=None)[source]
Bases:
Module
- __init__(prob, scale=True, dtype=None)[source]
Dropout layer. The layer randomly zeroes some of the elements of the input tensor with probability p using samples from a Bernoull distribution.
- Parameters
prob – The probability of zeroing out each array element.
scale – Whether to scale the output by 1/(1-prob), default is True.
device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.
dtype – the desired data type of the internal variables to be created. Default is None.
- class ivy.stateful.layers.LSTM(input_channels, output_channels, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, num_layers=1, return_sequence=True, return_state=True, device=None, v=None, dtype=None)[source]
Bases:
Module
- __init__(input_channels, output_channels, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, num_layers=1, return_sequence=True, return_state=True, device=None, v=None, dtype=None)[source]
LSTM layer, which is a set of stacked lstm cells.
- Parameters
input_channels – Number of input channels for the layer
output_channels – Number of output channels for the layer
weight_initializer – Initializer for the weights. Default is GlorotUniform.
num_layers – Number of lstm cells in the lstm layer, default is 1.
return_sequence – Whether or not to return the entire output sequence, or just the latest timestep. Default is True.
return_state – Whether or not to return the latest hidden and cell states. Default is True.
device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.
v – the variables for each of the lstm cells, as a container, constructed internally by default.
dtype –
- the desired data type of the internal variables to be created if not
provided. Default is None.
- class ivy.stateful.layers.Linear(input_channels, output_channels, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, with_bias=True, device=None, v=None, dtype=None)[source]
Bases:
Module
- __init__(input_channels, output_channels, weight_initializer=<ivy.stateful.initializers.GlorotUniform object>, bias_initializer=<ivy.stateful.initializers.Zeros object>, with_bias=True, device=None, v=None, dtype=None)[source]
Linear layer, also referred to as dense or fully connected. The layer receives tensors with input_channels last dimension and returns a new tensor with output_channels last dimension, following matrix multiplication with the weight matrix and addition with the bias vector.
- Parameters
input_channels – Number of input channels for the layer.
output_channels – Number of output channels for the layer.
weight_initializer – Initializer for the weights. Default is GlorotUniform.
bias_initializer – Initializer for the bias. Default is Zeros.
with_bias – Whether or not to include a bias term, default is True.
device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.
v – the variables for the linear layer, as a container, constructed internally by default.
dtype –
- the desired data type of the internal variables to be created if not
provided. Default is None.
- class ivy.stateful.layers.MultiHeadAttention(query_dim, num_heads=8, head_dim=64, dropout_rate=0.0, context_dim=None, scale=None, with_to_q_fn=True, with_to_kv_fn=True, with_to_out_fn=True, device=None, v=None, build_mode='on_init', dtype=None)[source]
Bases:
Module
- __init__(query_dim, num_heads=8, head_dim=64, dropout_rate=0.0, context_dim=None, scale=None, with_to_q_fn=True, with_to_kv_fn=True, with_to_out_fn=True, device=None, v=None, build_mode='on_init', dtype=None)[source]
Multi Head Attention layer.
- Parameters
query_dim – The dimension of the attention queries.
num_heads – Number of attention heads. Default is 8.
head_dim – The dimension of each of the heads. Default is 64.
dropout_rate – The rate of dropout. Default is 0.
context_dim – The dimension of the context array. Default is None, in which case the query dim is used.
scale – The value by which to scale the query-key similarity measure. Default is head_dim^-0.5
with_to_q_fn – Whether to include fully connected mapping from input x to queries. Default is True.
with_to_kv_fn – Whether to include fully connected mapping from input context to keys and values. Default is True.
with_to_out_fn – Whether to include fully connected mapping from output scaled dot-product attention to final output. Default is True.
device – device on which to create the layer’s variables ‘cuda:0’, ‘cuda:1’, ‘cpu’ etc. Default is cpu.
v – the variables for the attention layer, as a container, constructed internally by default.
build_mode – How the Module is built, either on initialization (now), explicitly by the user by calling build(), or the first time the __call__ method is run. Default is on initialization.
dtype –
- the desired data type of the internal variables to be created if not
provided. Default is None.