Ctrl+K

Layers#

Collection of Ivy neural network layers in functional form.

ivy.conv(x, filters, strides, padding, /, *, transpose=False, dims=2, output_shape=None, data_format='channel_last', filter_format='channel_last', feature_group_count=1, x_dilations=1, dilations=1, bias=None, out=None)[source]#

Compute a 1-D, 2-D, and 3-D transpose or dilated convolution given 3-D, 4-D and 5-D input x respectively and filters arrays.

Parameters:

  • x (Union[Array, NativeArray]) – Input image [batch_size,d,h,w,d_in] or [batch_size,d_in,d,h,w].

  • filters (Union[Array, NativeArray]) – Convolution filters [fd,fh,fw,d_in/feature_group_count,d_out].

  • strides (Union[int, Tuple[int], Tuple[int, int], Tuple[int, int, int]]) – The stride of the sliding window for each dimension of input.

  • padding (Union[str, Sequence[Tuple[int, int]]]) – either the string ‘SAME’ (padding with zeros evenly), the string ‘VALID’ (no padding), or a sequence of n (low, high) integer pairs that give the padding to apply before and after each spatial dimension.

  • transpose (bool, default: False) – True for computing transpose convolution, and False for dilated convolution. When True, x_dilations must be 1 (the default).

  • dims (int, default: 2) – Either 1, 2, or 3 corresponding to 1-D, 2-D, and 3-D convolution.

  • output_shape (Optional[Union[Shape, NativeShape]], default: None) – Shape of the output (Default value = None)

  • data_format (str, default: 'channel_last') – Either “channel_first” or “channel_last”. “channel_first” corresponds to “NCW”, “NCHW”, “NCDHW” input data formatS for 1-D, 2-D, 3-D convolution respectively, while “channel_last” corresponds to “NWC”, “NHWC”, “NDHWC” respectively.

  • filter_format (str, default: 'channel_last') – Either “channel_first” or “channel_last”. “channel_first” corresponds to “OIW”, “OIHW”, “OIDHW” input data formats for 1-D, 2-D, 3-D convolution respectively, while “channel_last” corresponds to “WIO”, “HWIO”, “DHWIO” respectively.

  • feature_group_count (int, default: 1) – split input into groups, d_in should be divisible by the number of groups. (Default value = 1)

  • x_dilations (Union[int, Tuple[int], Tuple[int, int], Tuple[int, int, int]], default: 1) – The dilation factor for each dimension of input. (Default value = 1)

  • dilations (Union[int, Tuple[int], Tuple[int, int], Tuple[int, int, int]], default: 1) – The dilation factor for each dimension of input. (Default value = 1)

  • bias (Optional[Union[Array, NativeArray]], default: None) – Bias array of shape [d_out].

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

ret – The result of the transpose or dilated convolution operation.

ivy.conv1d(x, filters, strides, padding, /, *, data_format='NWC', filter_format='channel_last', x_dilations=1, dilations=1, bias=None, out=None)[source]#

Compute a 1-D convolution given 3-D input x and filters arrays.

Parameters:

  • x (Union[Array, NativeArray]) – Input image [batch_size,w,d_in] or [batch_size,d_in,w].

  • filters (Union[Array, NativeArray]) – Convolution filters [fw,d_in,d_out].

  • strides (Union[int, Tuple[int]]) – The stride of the sliding window for each dimension of input.

  • padding (Union[str, int, Sequence[Tuple[int, int]]]) – either the string ‘SAME’ (padding with zeros evenly), the string ‘VALID’ (no padding), or a sequence of n (low, high) integer pairs that give the padding to apply before and after each spatial dimension.

  • data_format (str, default: 'NWC') – The ordering of the dimensions in the input, one of “NWC” or “NCW”. “NWC” corresponds to input with shape (batch_size, width, channels), while “NCW” corresponds to input with shape (batch_size, channels, width).

  • filter_format (str, default: 'channel_last') –

    Either “channel_first” or “channel_last”. “channel_first” corresponds to “OIW”,

    input data formats, while “channel_last” corresponds to “WIO”, “HWIO”, “DHWIO”.

    x_dilations

    The dilation factor for each dimension of input. (Default value = 1)

  • dilations (Union[int, Tuple[int]], default: 1) – The dilation factor for each dimension of input. (Default value = 1)

  • bias (Optional[Array], default: None) – Bias array of shape [d_out].

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

  • ret – The result of the convolution operation.

  • Both the description and the type hints above assumes an array input for simplicity,

  • but this function is nestable, and therefore also accepts ivy.Container

  • instances in place of any of the arguments.

Examples

With ivy.Array input:

>>> x = ivy.asarray([[[0.], [3.], [0.]]]) #NWC
>>> filters = ivy.array([[[0.]], [[1.]], [[0.]]]) #WIO
>>> result = ivy.conv1d(x, filters, (1,), 'SAME', data_format='NWC',dilations= (1,))
>>> print(result)
ivy.array([[[0.], [3.], [0.]]])

With ivy.NativeArray input:

>>> x = ivy.native_array([[[1., 3.], [2., 4.], [5., 7]]])
>>> filters = ivy.native_array([[[0., 1.], [1., 0.]]])
>>> result = ivy.conv1d(x, filters, (2,),'VALID')
>>> print(result)
ivy.array([[[3., 1.],
...         [7., 5.]]])

With a mix of ivy.Array and ivy.Container inputs:

>>> x = ivy.Container(a=ivy.array([[[1.2, 3.1, 4.8], [5.9, 2.2, 3.3],
...                                 [10.8, 7.6, 4.9], [6.1, 2.2, 9.5]]]),
...                   b=ivy.array([[[8.8, 7.7, 6.6], [1.1, 2.2, 3.5]]]))
>>> filters = ivy.array([[[1., 0., 1.], [0., 1., 0.], [1., 1., 0.]]])
>>> result  = ivy.conv1d(x, filters, 3, 'VALID')
>>> print(result)
{
        a: ivy.array([[[6., 7.9, 1.2],
...                    [15.6, 11.7, 6.1]]]),
...     b: ivy.array([[[15.4, 14.3, 8.8]]])
}
ivy.conv1d_transpose(x, filters, strides, padding, /, *, output_shape=None, filter_format='channel_last', data_format='NWC', dilations=1, bias=None, out=None)[source]#

Compute a 1-D transpose convolution given 3-D input x and filters arrays.

Parameters:

  • x (Union[Array, NativeArray]) – Input image [batch_size,w,d_in] or [batch_size,d_in,w].

  • filters (Union[Array, NativeArray]) – Convolution filters [fw,d_out,d_in].

  • strides (Union[int, Tuple[int]]) – The stride of the sliding window for each dimension of input.

  • padding (str) – Either ‘SAME’ (padding so that the output’s shape is the same as the input’s), or ‘VALID’ (padding so that the output’s shape is output_shape).

  • output_shape (Optional[Union[Shape, NativeShape]], default: None) – Shape of the output (Default value = None)

  • filter_format (str, default: 'channel_last') – Either “channel_first” or “channel_last”. “channel_first” corresponds to “IOW”,input data formats, while “channel_last” corresponds to “WOI”.

  • data_format (str, default: 'NWC') – The ordering of the dimensions in the input, one of “NWC” or “NCW”. “NWC” corresponds to input with shape (batch_size, width, channels), while “NCW” corresponds to input with shape (batch_size, channels, width).

  • dilations (Union[int, Tuple[int]], default: 1) – The dilation factor for each dimension of input. (Default value = 1)

  • bias (Optional[Array], default: None) – Bias array of shape [d_out].

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

  • ret – The result of the transpose convolution operation.

  • Both the description and the type hints above assumes an array input for simplicity,

  • but this function is nestable, and therefore also accepts ivy.Container

  • instances in place of any of the arguments.

Examples

With ivy.Array input:

>>> x = ivy.random_normal(mean=0, std=1, shape=[1, 28, 3])
>>> filters = ivy.random_normal(mean=0, std=1, shape=[3, 6, 3])
>>> y = ivy.conv1d_transpose(x, filters, 2, 'SAME')
>>> print(y.shape)
ivy.Shape(1, 56, 6)

>>> x = ivy.random_normal(mean=0, std=1, shape=[1, 128, 64])
>>> filters = ivy.random_normal(mean=0, std=1, shape=[1, 64, 64])
>>> ivy.conv1d_transpose(x, filters, 1, 'VALID', out=x)
>>> print(x.shape)
ivy.Shape(1, 128, 64)

>>> x = ivy.random_normal(mean=0, std=1, shape=[1, 256, 64])
>>> y = ivy.zeros((1, 258, 32))
>>> filters = ivy.random_normal(mean=0, std=1, shape=[3, 32, 64])
>>> ivy.conv1d_transpose(x, filters, 1, 'VALID', out=y)
>>> print(y.shape)
ivy.Shape(1, 258, 32)

With ivy.NativeArray input:

>>> x = ivy.native_array(
...         ivy.random_normal(mean=0, std=1, shape=[1, 256, 128]))
>>> filters = ivy.native_array(
...         ivy.random_normal(mean=0, std=1, shape=[3, 32, 128]))
>>> y = ivy.conv1d_transpose(x, filters, 2, 'SAME')
>>> print(y.shape)
ivy.Shape(1, 512, 32)

With one ivy.Container input:

>>> x = ivy.full((1, 6, 1), 2.7)
>>> a = ivy.random_normal(mean=0, std=1, shape=[3, 1, 1])
>>> b = ivy.random_normal(mean=0, std=1, shape=[3, 1, 1])
>>> filters = ivy.Container(a=a, b=b)
>>> y = ivy.conv1d_transpose(x, filters, 1, 'VALID', dilations=2)
>>> print(y.shape)
{
    a: ivy.Shape(1, 10, 1),
    b: ivy.Shape(1, 10, 1)
}

With multiple ivy.Container inputs:

>>> a = ivy.random_normal(mean=0, std=1, shape=[1, 14, 3])
>>> b = ivy.random_normal(mean=0, std=1, shape=[1, 28, 3])
>>> c = ivy.random_normal(mean=0, std=1, shape=[6, 3, 3])
>>> d = ivy.random_normal(mean=0, std=1, shape=[6, 3, 3])
>>> x = ivy.Container(a=a, b=b)
>>> filters = ivy.Container(c=c, d=d)
>>> y = ivy.conv1d_transpose(x, filters, 2, 'SAME')
>>> print(y.shape)
{
    a: {
        c: ivy.Shape(1, 28, 3),
        d: ivy.Shape(1, 28, 3)
    },
    b: {
        c: ivy.Shape(1, 56, 3),
        d: ivy.Shape(1, 56, 3)
    },
    c: {
        c: ivy.Shape(6, 6, 3),
        d: ivy.Shape(6, 6, 3)
    },
    d: {
        c: ivy.Shape(6, 6, 3),
        d: ivy.Shape(6, 6, 3)
    }
}
ivy.conv2d(x, filters, strides, padding, /, *, data_format='NHWC', filter_format='channel_last', x_dilations=1, dilations=1, bias=None, out=None)[source]#

Compute a 2-D convolution given 4-D input x and filters arrays.

Parameters:

  • x (Union[Array, NativeArray]) – Input image [batch_size,h,w,d_in] or [batch_size,d_in,h,w].

  • filters (Union[Array, NativeArray]) – Convolution filters [fh,fw,d_in,d_out].

  • strides (Union[int, Tuple[int, int]]) – The stride of the sliding window for each dimension of input.

  • padding (Union[str, int, Sequence[Tuple[int, int]]]) – either the string ‘SAME’ (padding with zeros evenly), the string ‘VALID’ (no padding), or a sequence of n (low, high) integer pairs that give the padding to apply before and after each spatial dimension.

  • data_format (str, default: 'NHWC') – The ordering of the dimensions in the input, one of “NHWC” or “NCHW”. “NHWC” corresponds to inputs with shape (batch_size, height, width, channels), while “NCHW” corresponds to input with shape (batch_size, channels, height, width).

  • filter_format (str, default: 'channel_last') –

    Either “channel_first” or “channel_last”. “channel_first” corresponds to “OIHW”,

    input data formats, while “channel_last” corresponds to “HWIO”.

    x_dilations

    The dilation factor for each dimension of input. (Default value = 1)

  • dilations (Union[int, Tuple[int, int]], default: 1) – The dilation factor for each dimension of input. (Default value = 1)

  • bias (Optional[Array], default: None) – Bias array of shape [d_out].

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

  • ret – The result of the convolution operation.

  • Both the description and the type hints above assumes an array input for simplicity,

  • but this function is nestable, and therefore also accepts ivy.Container

  • instances in place of any of the arguments.

Examples

With ivy.Array input:

>>> x = ivy.array([[[[1.], [2.0],[3.]],
...                 [[1.], [2.0],[3.]],
...                 [[1.], [2.0],[3.]]]])
>>> filters = ivy.array([[[[0.]],[[1.]],[[0.]]],
...                      [[[0.]],[[1.]], [[0.]]],
...                      [[[0.]],[[1.]], [[0.]]]])
>>> result = ivy.conv2d(x, filters, 1, 'SAME', data_format='NHWC', dilations=1)
>>> print(result)
ivy.array([[
          [[2.],[4.],[6.]],
          [[3.],[6.],[9.]],
          [[2.],[4.],[6.]]
          ]])

With one ivy.Container input:

>>> x = ivy.Container(a=ivy.array([[[[1.], [2.0],[3.]],
...                                 [[1.], [2.0],[3.]],
...                                 [[1.], [2.0],[3.]]]]))
>>> filters = ivy.eye(3, 3).reshape((3, 3, 1, 1)).astype(ivy.float32)
>>> result = ivy.conv2d(x, filters, 2, 'SAME', data_format='NHWC', dilations= 1)
>>> print(result)
{
    a:ivy.array([[[[3.], [3.]], [[1.], [5.]]]])
}

With multiple ivy.Container inputs:

>>> x = ivy.Container(a = ivy.eye(3, 3).reshape((1, 3, 3, 1)),
...                   b = ivy.eye(4, 4).reshape((1, 4, 4, 1)),
...                   c = ivy.eye(5, 5).reshape((1, 5, 5, 1)))
>>> filters = ivy.array([[1, 1, 1],
...                      [0, 1, 1],
...                      [0, 0, 1]], dtype = ivy.float32).reshape((3, 3, 1, 1))
>>> result = ivy.conv2d(x, filters, 2, 'SAME')
>>> print(result)
{
    a:ivy.array([[[[2.], [0.]], [[1.], [2.]]]]),
    b:ivy.array([[[[3.], [0.]], [[1.], [2.]]]]),
    c:ivy.array([[[[2.], [0.], [0.]],
                  [[1.], [3.], [0.]],
                  [[0.], [1.], [2.]]
                ]])
}

With a mix of ivy.Array and ivy.Container inputs:

>>> x = ivy.Container(a = ivy.eye(3, 3).reshape((1, 3, 3, 1)),
...                   b = ivy.eye(5, 5).reshape((1, 5, 5, 1)))
>>> filters = ivy.array([[2, 0, 1],
...                      [1, 3, 1],
...                      [0, 1, 1]], dtype = ivy.float32).reshape((3, 3, 1, 1))
>>> result = ivy.conv2d(x, filters, 2, 'SAME')
>>> print(result)
{
    a:ivy.array([[[[4.],[0.]],[[1.],[5.]]]]),
    b:ivy.array([[[[4.],[0.],[0.]],[[1.],[6.],[0.]],[[0.],[1.],[5.]]]])
}
ivy.conv2d_transpose(x, filters, strides, padding, /, *, output_shape=None, filter_format='channel_last', data_format='NHWC', dilations=1, bias=None, out=None)[source]#

Compute a 2-D transpose convolution given 4-D input x and filters arrays.

Parameters:

  • x (Union[Array, NativeArray]) – Input image [batch_size,h,w,d_in] or [batch_size,d_in,h,w].

  • filters (Union[Array, NativeArray]) – Convolution filters [fh,fw,d_out,d_in].

  • strides (Union[int, Tuple[int, int]]) – The stride of the sliding window for each dimension of input.

  • padding (str) – Either ‘SAME’ (padding so that the output’s shape is the same as the input’s), or ‘VALID’ (padding so that the output’s shape is output_shape).

  • output_shape (Optional[Union[Shape, NativeShape]], default: None) – Shape of the output (Default value = None)

  • data_format (str, default: 'NHWC') – The ordering of the dimensions in the input, one of “NHWC” or “NCHW”. “NHWC” corresponds to inputs with shape (batch_size, height, width, channels), while “NCHW” corresponds to input with shape (batch_size, channels, height, width).

  • filter_format (str, default: 'channel_last') – Either “channel_first” or “channel_last”. “channel_first” corresponds to “IOHW”,input data formats, while “channel_last” corresponds to “HWOI”.

  • x_dilations – The dilation factor for each dimension of input. (Default value = 1)

  • dilations (Union[int, Tuple[int, int]], default: 1) – The dilation factor for each dimension of input. (Default value = 1)

  • bias (Optional[Array], default: None) – Bias array of shape [d_out].

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

  • ret – The result of the transpose convolution operation.

  • Both the description and the type hints above assumes an array input for simplicity,

  • but this function is nestable, and therefore also accepts ivy.Container

  • instances in place of any of the arguments.

Examples

With ivy.Array input: >>> x = ivy.random_normal(mean=0, std=1, shape=[1, 28, 28, 3]) >>> filters = ivy.random_normal(mean=0, std=1, shape=[3, 3, 6, 3]) >>> y = ivy.conv2d_transpose(x,filters,2,’SAME’) >>> print(y.shape) ivy.Shape(1, 56, 56, 6)

>>> x = ivy.random_normal(mean=0, std=1, shape=[1, 128, 128, 64])
>>> filters = ivy.random_normal(mean=0, std=1, shape=[1, 1, 64, 64])
>>> ivy.conv2d_transpose(x,filters,1,'VALID',out=x)
>>> print(x.shape)
ivy.Shape(1, 128, 128, 64)

>>> x = ivy.random_normal(mean=0, std=1, shape=[1, 256, 256, 64])
>>> y = ivy.zeros((1, 258, 258, 32))
>>> filters = ivy.random_normal(mean=0, std=1, shape=[3, 3, 32, 64])
>>> ivy.conv2d_transpose(x,filters,[1, 1, 1],'VALID',out=y)
>>> print(y.shape)
ivy.Shape(1, 258, 258, 32)

With one ivy.Container inputs: >>> x = ivy.full((1, 6, 6, 1), 2.7) >>> a = ivy.random_normal(mean=0, std=1, shape=[3, 3, 1, 1]) >>> b = ivy.random_normal(mean=0, std=1, shape=[3, 3, 1, 1]) >>> filters = ivy.Container(a=a, b=b) >>> y = ivy.conv2d_transpose(x,filters,1,’VALID’,dilations=2) >>> print(y.shape) {

a: ivy.Shape(1, 10, 10, 1), b: ivy.Shape(1, 10, 10, 1)

}

With multiple ivy.Container inputs: >>> a = ivy.random_normal(mean=0, std=1, shape=[1, 14, 14, 3]) >>> b = ivy.random_normal(mean=0, std=1, shape=[1, 28, 28, 3]) >>> c = ivy.random_normal(mean=0, std=1, shape=[6, 3, 3, 3]) >>> d = ivy.random_normal(mean=0, std=1, shape=[6, 3, 3, 3]) >>> x = ivy.Container(a=a, b=b) >>> filters = ivy.Container(c=c, d=d) >>> y = ivy.conv2d_transpose(x,filters,2,’SAME’) >>> print(y.shape) {

a: {

c: ivy.Shape(1, 28, 28, 3), d: ivy.Shape(1, 28, 28, 3)

}, b: {

c: ivy.Shape(1, 56, 56, 3), d: ivy.Shape(1, 56, 56, 3)

}, c: {

c: ivy.Shape(6, 6, 6, 3), d: ivy.Shape(6, 6, 6, 3)

}, d: {

c: ivy.Shape(6, 6, 6, 3), d: ivy.Shape(6, 6, 6, 3)

}

}

ivy.conv3d(x, filters, strides, padding, /, *, data_format='NDHWC', filter_format='channel_last', x_dilations=1, dilations=1, bias=None, out=None)[source]#

Compute a 3-D convolution given 5-D input x and filters arrays.

Parameters:

  • x (Union[Array, NativeArray, Container]) – Input volume [batch_size,d,h,w,d_in] or [batch_size,d_in,d,h,w].

  • filters (Union[Array, NativeArray, Container]) – Convolution filters [fd,fh,fw,d_in,d_out].

  • strides (Union[int, Tuple[int, int, int]]) – The stride of the sliding window for each dimension of input.

  • padding (Union[str, int, Sequence[Tuple[int, int]]]) – either the string ‘SAME’ (padding with zeros evenly), the string ‘VALID’ (no padding), or a sequence of n (low, high) integer pairs that give the padding to apply before and after each spatial dimension.

  • data_format (str, default: 'NDHWC') – The ordering of the dimensions in the input, one of “NDHWC” or “NCDHW”. “NDHWC” corresponds to inputs with shape (batch_size, depth, height, width, channels), while “NCDHW” corresponds to input with shape (batch_size, channels, depth, height, width).

  • filter_format (str, default: 'channel_last') –

    Either “channel_first” or “channel_last”. “channel_first” corresponds

    to “OIDHW”,input data formats, while “channel_last” corresponds to “DHWIO”.

    x_dilations

    The dilation factor for each dimension of input. (Default value = 1)

  • dilations (Union[int, Tuple[int, int, int]], default: 1) – The dilation factor for each dimension of input. (Default value = 1)

  • bias (Optional[Array], default: None) – Bias array of shape [d_out]

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

  • ret – The result of the convolution operation.

  • Both the description and the type hints above assumes an array input for simplicity,

  • but this function is nestable, and therefore also accepts ivy.Container

  • instances in place of any of the arguments.

Examples

With ivy.Array input:

>>> x = ivy.array([[[1., 2. ,1.], [1., 2. ,1.], [1., 2. ,1.]],
...         [[1., 2. ,1.], [1., 2. ,1.], [1., 2. ,1.]],
...         [[1., 2. ,1.], [1., 2. ,1.], [1., 2. ,1.]]]).reshape((1, 3, 3, 3, 1))
>>> filters = ivy.array([[[0.,1.,0.],
...                       [0.,1.,0.],
...                       [0.,1.,0.]]]).reshape((1,3,3,1,1))
>>> result = ivy.conv3d(x, filters, 1, 'SAME', data_format='NDHWC', dilations=1)
>>> print(result)
ivy.array([[[[[2.],[4.],[2.]],[[3.],[6.],[3.]],[[2.],[4.],[2.]]],
            [[[2.],[4.],[2.]],[[3.],[6.],[3.]],[[2.],[4.],[2.]]],
            [[[2.],[4.],[2.]],[[3.],[6.],[3.]],[[2.],[4.],[2.]]]]])

With one ivy.Container input:

>>> x = ivy.Container(a = ivy.ones((1, 3, 3, 3, 1)).astype(ivy.float32))
>>> filters = ivy.ones((3, 3, 3, 1, 1)).astype(ivy.float32)
>>> result = ivy.conv3d(x, filters, 2, 'SAME')
>>> print(result)
{
    a: ivy.array([[[[[8.],[8.]],[[8.],[8.]]],[[[8.],[8.]],[[8.],[8.]]]]])
}

With multiple ivy.Container input:

>>> x = ivy.Container( a = ivy.random_normal(mean = 0, std = 1,
...                        shape = [1, 3, 5, 5, 1]),
...                    b = ivy.random_normal(mean = 0, std = 1,
...                        shape = [1, 5, 32 ,32, 1]),
...                    c = ivy.random_normal(mean = 0, std = 1,
...                        shape = [1, 32, 32, 32, 1]))
>>> filters = ivy.ones((3, 5, 5, 1, 3)).astype(ivy.float32)
>>> result = ivy.conv3d(x, filters, 1, 'SAME')
>>> print(result.cont_shapes)
{
    a: ivy.Shape(1, 3, 5, 5, 3),
    b: ivy.Shape(1, 5, 32, 32, 3),
    c: ivy.Shape(1, 32, 32, 32, 3)
}
ivy.conv3d_transpose(x, filters, strides, padding, /, *, output_shape=None, filter_format='channel_last', data_format='NDHWC', dilations=1, bias=None, out=None)[source]#

Compute a 3-D transpose convolution given 5-D input x and filters arrays.

Parameters:

  • x (Union[Array, NativeArray]) – Input volume [batch_size,d,h,w,d_in] or [batch_size,d_in,d,h,w].

  • filters (Union[Array, NativeArray]) – Convolution filters [fd,fh,fw,d_out,d_in].

  • strides (Union[int, Tuple[int, int, int]]) – The stride of the sliding window for each dimension of input.

  • padding (str) – Either ‘SAME’ (padding so that the output’s shape is the same as the input’s), or ‘VALID’ (padding so that the output’s shape is output_shape).

  • output_shape (Optional[Union[Shape, NativeShape]], default: None) – Shape of the output (Default value = None)

  • filter_format (str, default: 'channel_last') – Either “channel_first” or “channel_last”. “channel_first” corresponds to “IODHW”,input data formats, while “channel_last” corresponds to “DHWOI”.

  • data_format (str, default: 'NDHWC') – The ordering of the dimensions in the input, one of “NDHWC” or “NCDHW”. “NDHWC” corresponds to inputs with shape (batch_size, depth, height, width, channels), while “NCDHW” corresponds to input with shape (batch_size, channels, depth, height, width).

  • dilations (Union[int, Tuple[int, int, int]], default: 1) – The dilation factor for each dimension of input. (Default value = 1)

  • bias (Optional[Array], default: None) – Bias array of shape [d_out]

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

ret – The result of the transpose convolution operation.

Examples

With ivy.Array input:

>>> x = ivy.random_normal(mean=0, std=1, shape=[1, 3, 28, 28, 3])
>>> filters = ivy.random_normal(mean=0, std=1, shape=[3, 3, 3, 6, 3])
>>> y = ivy.conv3d_transpose(x, filters, [2, 2, 2], 'SAME')
>>> print(y.shape)
ivy.Shape(1, 6, 56, 56, 6)

>>> x = ivy.random_normal(mean=0, std=1, shape=[1, 3, 64, 64, 3])
>>> filters = ivy.random_normal(mean=0, std=1, shape=[3, 3, 3, 6, 3])
>>> y = ivy.conv3d_transpose(x, filters, [2, 2, 2], 'VALID', dilations=[1, 1, 1])
>>> print(y.shape)
ivy.Shape(1, 7, 129, 129, 6)

With ivy.Container inputs:

>>> a = ivy.random_normal(mean=0, std=1, shape=[1, 3, 14, 14, 3])
>>> b = ivy.random_normal(mean=0, std=1, shape=[1, 3, 28, 28, 3])
>>> c = ivy.random_normal(mean=0, std=1, shape=[6, 3, 3, 3, 3])
>>> d = ivy.random_normal(mean=0, std=1, shape=[6, 3, 3, 3, 3])
>>> x = ivy.Container(a=a, b=b)
>>> filters = ivy.Container(c=c, d=d)
>>> y = ivy.conv3d_transpose(x, filters, [2, 2, 2], 'SAME')
>>> print(y.shape)
{
    a: {
        c: ivy.Shape(1, 6, 28, 28, 3),
        d: ivy.Shape(1, 6, 28, 28, 3)
    },
    b: {
        c: ivy.Shape(1, 6, 56, 56, 3),
        d: ivy.Shape(1, 6, 56, 56, 3)
    },
    c: {
        c: ivy.Shape(6, 6, 6, 6, 3),
        d: ivy.Shape(6, 6, 6, 6, 3)
    },
    d: {
        c: ivy.Shape(6, 6, 6, 6, 3),
        d: ivy.Shape(6, 6, 6, 6, 3)
    }
}

With a mix of ivy.Array and ivy.Container inputs:

>>> x = ivy.full((1, 6, 6, 6, 1), 2.7)
>>> a = ivy.random_normal(mean=0, std=1, shape=[3, 3, 3, 1, 1])
>>> b = ivy.random_normal(mean=0, std=1, shape=[3, 3, 3, 1, 1])
>>> filters = ivy.Container(a=a, b=b)
>>> y = ivy.conv3d_transpose(x, filters, [1, 1, 1], 'VALID', dilations=[1, 1, 1])
>>> print(y.shape)
{
    a: ivy.Shape(1, 8, 8, 8, 1),
    b: ivy.Shape(1, 8, 8, 8, 1)
}

>>> x = ivy.full((1, 6, 6, 6, 1), 1.23)
>>> a = ivy.array(ivy.random_normal(mean=0, std=1, shape=[3, 3, 3, 1, 1]))
>>> b = ivy.array(ivy.random_normal(mean=0, std=1, shape=[3, 3, 3, 1, 1]))
>>> filters = ivy.Container(a=a, b=b)
>>> y = ivy.conv3d_transpose(x, filters, [1, 1, 1], 'VALID', dilations=[1, 1, 1])
>>> print(y.shape)
{
    a: ivy.Shape(1, 8, 8, 8, 1),
    b: ivy.Shape(1, 8, 8, 8, 1)
}
ivy.conv_general_dilated(x, filters, strides, padding, /, *, dims=2, data_format='channel_last', filter_format='channel_last', feature_group_count=1, x_dilations=1, dilations=1, bias=None, out=None)[source]#

Compute a 1-D, 2-D, and 3-D convolution given 3-D, 4-D and 5-D input x respectively and filters arrays.

Parameters:

  • x (Union[Array, NativeArray]) – Input image [batch_size,d,h,w,d_in] or [batch_size,d_in,d,h,w].

  • filters (Union[Array, NativeArray]) – Convolution filters [fd,fh,fw,d_in/feature_group_count,d_out].

  • strides (Union[int, Tuple[int], Tuple[int, int], Tuple[int, int, int]]) – The stride of the sliding window for each dimension of input.

  • padding (Union[str, int, Sequence[Tuple[int, int]]]) – either the string ‘SAME’ (padding with zeros evenly), the string ‘VALID’ (no padding), or a sequence of n (low, high) integer pairs that give the padding to apply before and after each spatial dimension.

  • dims (int, default: 2) – Either 1, 2, or 3 corresponding to 1-D, 2-D, and 3-D convolution.

  • data_format (str, default: 'channel_last') – Either “channel_first” or “channel_last”. “channel_first” corresponds to “NCW”, “NCHW”, “NCDHW” input data formatS for 1-D, 2-D, 3-D convolution respectively, while “channel_last” corresponds to “NWC”, “NHWC”, “NDHWC” respectively.

  • filter_format (str, default: 'channel_last') – Either “channel_first” or “channel_last”. “channel_first” corresponds to “OIW”, “OIHW”, “OIDHW” input data formats for 1-D, 2-D, 3-D convolution respectively, while “channel_last” corresponds to “WIO”, “HWIO”, “DHWIO” respectively.

  • feature_group_count (int, default: 1) – split input into groups, d_in should be divisible by the number of groups. (Default value = 1)

  • x_dilations (Union[int, Tuple[int], Tuple[int, int], Tuple[int, int, int]], default: 1) – The dilation factor for each dimension of input. (Default value = 1)

  • dilations (Union[int, Tuple[int], Tuple[int, int], Tuple[int, int, int]], default: 1) – The dilation factor for each dimension of filter. (Default value = 1)

  • bias (Optional[Union[Array, NativeArray]], default: None) – Bias array of shape [d_out].

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

ret – The result of the transpose convolution operation.

ivy.conv_general_transpose(x, filters, strides, padding, /, *, dims=2, output_shape=None, filter_format='channel_last', data_format='channel_last', dilations=1, feature_group_count=1, bias=None, out=None)[source]#

Compute a 1-D, 2-D, and 3-D transpose convolution given 3-D, 4-D and 5-D input x respectively and filters arrays.

Parameters:

  • x (Union[Array, NativeArray]) – Input image [batch_size,d,h,w,d_in] or [batch_size,d_in,d,h,w].

  • filters (Union[Array, NativeArray]) – Convolution filters [fd,fh,fw,d_out,d_in].

  • strides (Union[int, Tuple[int], Tuple[int, int], Tuple[int, int, int]]) – The stride of the sliding window for each dimension of input.

  • padding (str) – Either ‘SAME’ (padding so that the output’s shape is the same as the input’s), or ‘VALID’ (padding so that the output’s shape is output_shape).

  • dims (int, default: 2) – Either 1, 2, or 3 corresponding to 1-D, 2-D, and 3-D convolution.

  • output_shape (Optional[Union[Shape, NativeShape]], default: None) – Shape of the output.

  • filter_format (str, default: 'channel_last') – Either “channel_first” or “channel_last”. “channel_first” corresponds to “IODHW”,input data formats, while “channel_last” corresponds to “DHWOI”.

  • data_format (str, default: 'channel_last') – Either “channel_first” or “channel_last”. “channel_first” corresponds to “NCW”, “NCHW”, “NCDHW” input data formatS for 1-D, 2-D, 3-D convolution respectively, while “channel_last” corresponds to “NWC”, “NHWC”, “NDHWC” respectively.

  • dilations (Union[int, Tuple[int], Tuple[int, int], Tuple[int, int, int]], default: 1) – The dilation factor for each dimension of input. (Default value = 1)

  • feature_group_count (int, default: 1) – split input into groups, d_in should be divisible by the number of groups.

  • bias (Optional[Union[Array, NativeArray]], default: None) – Bias array of shape [d_out].

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

ret – The result of the transpose convolution operation.

Examples

With ivy.Array input: >>> x = ivy.random_normal(mean=0, std=1, shape=[1, 3, 28, 28, 3]) >>> filters = ivy.random_normal(mean=0, std=1, shape=[3, 3, 3, 6, 3]) >>> y = ivy.conv3d_transpose(x, filters, [2, 2, 2], ‘SAME’) >>> print(y.shape) ivy.Shape(1, 6, 56, 56, 6) >>> x = ivy.random_normal(mean=0, std=1, shape=[1, 3, 64, 64, 3]) >>> filters = ivy.random_normal(mean=0, std=1, shape=[3, 3, 3, 6, 3]) >>> y = ivy.conv3d_transpose(x, filters, [2, 2, 2], ‘VALID’, dilations=[1, 1, 1]) >>> print(y.shape) ivy.Shape(1, 7, 129, 129, 6) With :class: ‘ivy.Container’ inputs: >>> a = ivy.random_normal(mean=0, std=1, shape=[1, 3, 14, 14, 3]) >>> b = ivy.random_normal(mean=0, std=1, shape=[1, 3, 28, 28, 3]) >>> c = ivy.random_normal(mean=0, std=1, shape=[6, 3, 3, 3, 3]) >>> d = ivy.random_normal(mean=0, std=1, shape=[6, 3, 3, 3, 3]) >>> x = ivy.Container(a=a, b=b) >>> filters = ivy.Container(c=c, d=d) >>> y = ivy.conv3d_transpose(x, filters, [2, 2, 2], ‘SAME’) >>> print(y.shape) {

a: {

c: ivy.Shape(1, 6, 28, 28, 3), d: ivy.Shape(1, 6, 28, 28, 3)

}, b: {

c: ivy.Shape(1, 6, 56, 56, 3), d: ivy.Shape(1, 6, 56, 56, 3)

}, c: {

c: ivy.Shape(6, 6, 6, 6, 3), d: ivy.Shape(6, 6, 6, 6, 3)

}, d: {

c: ivy.Shape(6, 6, 6, 6, 3), d: ivy.Shape(6, 6, 6, 6, 3)

}

} With a mix of ivy.Array and ivy.Container inputs: >>> x = ivy.full((1, 6, 6, 6, 1), 2.7) >>> a = ivy.random_normal(mean=0, std=1, shape=[3, 3, 3, 1, 1]) >>> b = ivy.random_normal(mean=0, std=1, shape=[3, 3, 3, 1, 1]) >>> filters = ivy.Container(a=a, b=b) >>> y = ivy.conv3d_transpose(x, filters, [1, 1, 1], ‘VALID’, dilations=[1, 1, 1]) >>> print(y.shape) {

a: ivy.Shape(1, 8, 8, 8, 1), b: ivy.Shape(1, 8, 8, 8, 1)

} >>> x = ivy.full((1, 6, 6, 6, 1), 1.23) >>> a = ivy.array(ivy.random_normal(mean=0, std=1, shape=[3, 3, 3, 1, 1])) >>> b = ivy.array(ivy.random_normal(mean=0, std=1, shape=[3, 3, 3, 1, 1])) >>> filters = ivy.Container(a=a, b=b) >>> y = ivy.conv3d_transpose(x, filters, [1, 1, 1], ‘VALID’, dilations=[1, 1, 1]) >>> print(y.shape) {

a: ivy.Shape(1, 8, 8, 8, 1), b: ivy.Shape(1, 8, 8, 8, 1)

}

ivy.depthwise_conv2d(x, filters, strides, padding, /, *, data_format='NHWC', dilations=1, out=None)[source]#

Compute a 2-D depthwise convolution given 4-D input x and filters arrays.

Parameters:

  • x (Union[Array, NativeArray]) – Input image [batch_size,h,w,d_in] or [batch_size,d_in,h,w].

  • filters (Union[Array, NativeArray]) – Convolution filters [fh,fw,d_in]. (d_in must be the same as d from x)

  • strides (Union[int, Tuple[int, int]]) – The stride of the sliding window for each dimension of input.

  • padding (Union[str, Sequence[Tuple[int, int]]]) – either the string ‘SAME’ (padding with zeros evenly), the string ‘VALID’ (no padding), or a sequence of n (low, high) integer pairs that give the padding to apply before and after each spatial dimension.

  • data_format (str, default: 'NHWC') – The ordering of the dimensions in the input, one of “NHWC” or “NCHW”. “NHWC” corresponds to inputs with shape (batch_size, height, width, channels), while “NCHW” corresponds to input with shape (batch_size, channels, height, width).

  • dilations (Union[int, Tuple[int, int]], default: 1) – The dilation factor for each dimension of input. (Default value = 1)

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

  • ret – The result of the convolution operation.

  • Both the description and the type hints above assumes an array input for simplicity,

  • but this function is nestable, and therefore also accepts ivy.Container

  • instances in place of any of the arguments.

Examples

With ivy.Array input:

>>> x = ivy.random_normal(mean=0, std=1, shape=[1, 28, 28, 3])
>>> filters = ivy.random_normal(mean=0, std=1, shape=[3, 3, 3])
>>> y = ivy.depthwise_conv2d(x, filters, (1, 1), 'VALID')
>>> print(y.shape)
ivy.Shape(1, 26, 26, 3)

>>> x = ivy.random_normal(mean=0, std=1, shape=[1, 32, 32, 3])
>>> y = ivy.zeros((1, 16, 16, 3))
>>> filters = ivy.random_normal(mean=0, std=1, shape=[5, 5, 3])
>>> ivy.depthwise_conv2d(x, filters, [2, 2], 'SAME', out=y)
>>> print(y.shape)
ivy.Shape(1, 16, 16, 3)

>>> x = ivy.random_normal(mean=0, std=1, shape=[1, 64, 64, 32])
>>> y = ivy.zeros((1, 61, 61, 32))
>>> filters = ivy.random_normal(mean=0, std=1, shape=[4, 4, 32])
>>> ivy.depthwise_conv2d(x, filters, [1, 1], 'VALID', out=y)
>>> print(x.shape)
ivy.Shape(1, 64, 64, 32)

With ivy.NativeArray input:

>>> x = ivy.native_array(ivy.random_normal(mean=0, std=1, shape=[1, 7, 7, 64]))
>>> filters = ivy.native_array(ivy.random_normal(mean=0, std=1, shape=[3, 3, 64]))
>>> y = ivy.depthwise_conv2d(x, filters, [1, 1], 'SAME')
>>> print(y.shape)
ivy.Shape(1, 7, 7, 64)

With a mix of ivy.Array and ivy.Container inputs:

>>> x = ivy.eye(6, 6).reshape((1, 6, 6, 1)) #NHWC
>>> a = ivy.array([[1., 1., 1.], [1., -8., 1.], [1., 1., 1.]]).expand_dims(axis=-1)
>>> b = ivy.array([[1., 1., 1.],
...                [1., 1., 1.],
...                [1., 1., 1.]]).expand_dims(axis=-1) / 9.0
>>> filters = ivy.Container(a = a, b = b)
>>> y = ivy.depthwise_conv2d(x, filters, 1, 'VALID', dilations=2)
>>> print(y)
{
    a: ivy.array([[[[-6.],
                    [0.]],
                   [[0.],
                    [-6.]]]]),
    b: ivy.array([[[[0.33333334],
                    [0.]],
                   [[0.],
                    [0.33333334]]]])
}

With a mix of ivy.Array, code:ivy.NativeArray and ivy.Container inputs:

>>> x = ivy.eye(6, 6).reshape((1, 6, 6, 1)) #NHWC
>>> y = ivy.native_array(ivy.eye(6, 6).reshape((1, 6, 6, 1)))
>>> inp = ivy.Container(x = x, y = y)
>>> filter = ivy.array([[1., 1., 1.],
...                     [1., -8., 1.],
...                     [1., 1., 1.]]).expand_dims(axis=-1)
>>> y = ivy.depthwise_conv2d(inp, filter, 1, 'VALID', dilations=2)
>>> print(y)
{
    x: ivy.array([[[[-6.],
                    [0.]],
                   [[0.],
                    [-6.]]]]),
    y: ivy.array([[[[-6.],
                    [0.]],
                   [[0.],
                    [-6.]]]])
}
ivy.dropout(x, prob, /, *, scale=True, dtype=None, training=True, seed=None, noise_shape=None, out=None)[source]#

Randomly setting a fraction of input tensor to zeroes with probability.

prob at each update during training time to prevent possible overfitting. The inputs not set to 0 are scaled up 1 / (1 - prob) by default, so that overall sum is unchanged at training time and inference time.

Parameters:

  • x (Union[Array, NativeArray]) – The input array x to perform dropout on.

  • prob (float) – The probability of zeroing out each array element, float between 0 and 1.

  • scale (bool, default: True) – Whether to scale the output by 1/(1-prob). Default is True.

  • dtype (Optional[Union[Dtype, NativeDtype]], default: None) – output array data type. If dtype is None, the output array data type must be inferred from x. Default is None.

  • training (bool, default: True) – Turn on dropout if training, turn off otherwise. Default is True.

  • seed (Optional[int], default: None) – Set a default seed for random number generating (for reproducibility). Default is None.

  • noise_shape (Optional[Sequence[int]], default: None) – a sequence representing the shape of the binary dropout mask that will be multiplied with the input. A shape dimension set to None means that a different mask value will be applied to each element of the input across that dimension. A dimension set to 1 means the same mask value will be applied to all elements of the input across that dimension.

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

  • ret – Result array after dropout is performed.

  • Both the description and the type hints above assumes an array input for simplicity,

  • but this function is nestable, and therefore also accepts ivy.Container

  • instances in place of any of the arguments.

Examples

With ivy.Array input:

>>> x = ivy.array([[1., 2., 3.],
...                [4., 5., 6.],
...                [7., 8., 9.],
...                [10., 11., 12.]])
>>> y = ivy.dropout(x,0.3)
>>> print(y)
ivy.array([[ 1.42857146,  2.85714293,  4.28571415],
       [ 0.        ,  7.14285755,  8.5714283 ],
       [10.        , 11.4285717 ,  0.        ],
       [14.2857151 ,  0.        , 17.1428566 ]])

>>> x = ivy.array([[1.5, 2.6],
...                [4.9, 6.6],
...                [7.2, 8.7]])
>>> y = ivy.dropout(x,0.5)
>>> print(y)
ivy.array([[ 0.        ,  5.19999981],
           [ 0.        ,  0.        ],
           [ 0.        , 17.39999962]])

>>> x = ivy.array([[1., 2., 3.],
...                [4., 5., 6.],
...                [7., 8., 9.],
...                [10., 11., 12.]])
>>> y = ivy.dropout(x,0.3,scale=False)
>>> print(y)
ivy.array([[ 1.,  2., 3.],
           [ 4.,  5., 0.],
           [ 7.,  0., 9.],
           [10., 11., 0.]])

>>> x = ivy.array([[1.5, 2.6],
...                [4.9, 6.6],
...                [7.2, 8.7]])
>>> y = ivy.dropout(x,0.5,scale=False)
>>> print(y)
ivy.array([[0., 2.6],
           [0., 0. ],
           [0., 8.7]])

With ivy.Container input:

>>> x = ivy.Container(a=ivy.array([[1., 2., 3.], [4., 5., 6.]]),
...                   b=ivy.array([7., 8., 9.]))
>>> y = ivy.dropout(x,0.3)
>>> print(y)
{
a: ivy.array([[0., 0., 4.28571415],
              [5.71428585, 7.14285755, 0.]]),
b: ivy.array([0., 11.4285717, 12.8571434])
}

>>> x = ivy.Container(a=ivy.array([[1.1, 2.2, 3.3], [11., 22., 33.]]),
...                   b=ivy.array([[1.245, 0.278, 4.105], [7., 13., 17.]]))
>>> y = ivy.dropout(x,0.5)
>>> print(y)
{
    a: ivy.array([[0., 4.4000001, 6.5999999],
                  [22., 44., 0.]]),
    b: ivy.array([[2.49000001, 0.55599999, 8.21000004],
                  [14., 0., 0.]])
}

>>> x = ivy.Container(a=ivy.array([[1., 2., 3.], [4., 5., 6.]]),
...                   b=ivy.array([7., 8., 9.]))
>>> y = ivy.dropout(x,0.3)
>>> print(y)
{
    a: ivy.array([[0., 0., 3.],
                  [4., 5., 0.]]),
    b: ivy.array([0., 8., 9.])
}

>>> x = ivy.Container(a=ivy.array([[1.1, 2.2, 3.3], [11., 22., 33.]]),
...                   b=ivy.array([[1.245, 0.278, 4.105], [7., 13., 17.]]))
>>> y = ivy.dropout(x,0.5)
>>> print(y)
{
    a: ivy.array([[0., 2.2, 3.3],
                  [11., 22., 0.]]),
    b: ivy.array([[1.245, 0.278, 4.105],
                  [7., 0., 0.]])
}
ivy.linear(x, weight, /, *, bias=None, out=None)[source]#

Apply a linear transformation to the incoming data: y = x * t(weight) + bias. The operation also supports batching of the weight matrices. This is useful if a batch of different network parameters are to be represented.

Parameters:

  • x (Union[Array, NativeArray]) – The input x to compute linear transformation on. [outer_batch_shape,inner_batch_shape,in_features]

  • weight (Union[Array, NativeArray]) – The weight matrix. [outer_batch_shape,out_features,in_features]

  • bias (Optional[Union[Array, NativeArray]], default: None) – The bias vector, default is None. [outer_batch_shape,out_features]

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

  • ret – Result array of the linear transformation. [outer_batch_shape,inner_batch_shape,out_features]

  • Both the description and the type hints above assumes an array input for simplicity,

  • but this function is nestable, and therefore also accepts ivy.Container

  • instances in place of any of the arguments.

Examples

With ivy.Array input:

>>> x = ivy.array([1., 2., 3.])
>>> w = ivy.array([[1., 0., 0.]])
>>> y = ivy.linear(x, w)
>>> print(y)
ivy.array([1.])

>>> x = ivy.array([[0.666, -0.4269, 1.911]])
>>> w = ivy.array([[1., 0., 0.], [0., 0., 1.]])
>>> y = ivy.zeros((1, 2))
>>> ivy.linear(x, w, out=y)
>>> print(y)
ivy.array([[0.666, 1.91 ]])

>>> x = ivy.array([[1.546, 5.234, 6.487],
...                [0.157, 5.753, 4.52],
...                [5.165, 3.159, 7.101]])
>>> w = ivy.array([[1.545, 2.547, 3.124],
...                [5.852, 8.753, 6.963]])
>>> b = ivy.array([-1., 1.])
>>> y = ivy.zeros((3, 2))
>>> ivy.linear(x, w, bias=b, out=y)
>>> print(y)
ivy.array([[ 34.98495483, 101.0293808 ],
       [ 28.0159359 ,  83.74752808],
       [ 37.20942307, 108.3205719 ]])

With ivy.Container input:

>>> x = ivy.Container(a=ivy.array([[1., 2., 3.],
...                                [4., 5., 6.]]),
...                   b=ivy.array([1.1, 2.2, 3.3]))
>>> w = ivy.Container(a=ivy.array([[1., 2., 3.],
...                                [-1., 1., 2.]]),
...                   b=ivy.array([[0., -1., 1.],
...                                [0., 1., 1.]]))
>>> b = ivy.Container(a=ivy.array([1., -1.]), b=ivy.array([1., 1.]))
>>> y = ivy.linear(x, w, bias=b)
>>> print(y)
{
    a: ivy.array([[15., 6.],
                  [33., 12.]]),
    b: ivy.array([2.1, 6.5])
}

With a mix of ivy.Array and ivy.Container inputs:

>>> x = ivy.Container(a=ivy.array([[1.1, 2.2, 3.3],
...                                [11., 22., 33.]]),
...                   b=ivy.array([[1.245, 0.278, 4.105],
...                                [7., 13., 17.]]))
>>> w = ivy.array([[1., 2., 3.],
...                [4., 5., 6.],
...                [7., 8., 9.]])
>>> b = ivy.Container(a=ivy.array([1., 0., -1.]),
...                   b=ivy.array([1., 1., 0.]))
>>> ivy.linear(x, w, bias=b, out=x)
>>> print(x)
{
    a: ivy.array([[16.4, 35.2, 54.],
                  [155., 352., 549.]]),
    b: ivy.array([[15.1, 32., 47.9],
                  [85., 196., 306.]])
}
ivy.lstm(input, initial_states, all_weights, num_layers, dropout, train, bidirectional, batch_first=False, batch_sizes=None, weights_transposed=False, has_ih_bias=True, has_hh_bias=True)[source]#

Applies a multi-layer long-short term memory to an input sequence.

Parameters:

  • input (Array) – input array of shape (seq_len, batch, input_size) when batch_first is False or (batch, seq_len, input_size) when batch_first is True

  • initial_states (Tuple[Array]) –

    tuple of two arrays (h_0, c_0) where h_0 is the initial hidden state of shape (num_layers * num_directions, batch, hidden_size) and c_0 is the initial cell state of shape (num_layers * num_directions, batch, hidden_size)

    (num_directions being 2 when bidirectional, otherwise 1)

  • all_weights (Tuple[Array]) –

    tuple of arrays representing the learnable weights of the lstm, with each layer having up to four arrays (w_ih, w_hh, b_ih, b_hh) representing the weights and biases (if biases are being used)

    w_ih: weight of shape (4 * hidden_size, input_size) w_hh: weight of shape (4 * hidden_size, hidden_size) b_ih: bias of shape (4 * hidden_size,) b_hh: bias of shape (4 * hidden_size,)

  • num_layers (int) – number of layers for the lstm to use

  • dropout (float) – dropout rate

  • train (bool) – whether to run the lstm in train mode or eval mode

  • bidirectional (bool) – whether the lstm is bidirectional or unidirectional

  • batch_first (bool, default: False) – defines the data format of the input and output arrays

  • batch_sizes (Optional[Sequence], default: None) – specifies the batch size at each timestep, when the input is a packed sequence

  • weights_transposed (bool, default: False) – whether the weights are transposed compared to the format in which they are expected (input_size, 4 * hidden_size) rather than (4 * hidden_size, input_size)

  • has_ih_bias (bool, default: True) – whether the all_weights argument includes a input-hidden bias

  • has_hh_bias (bool, default: True) – whether the all_weights argument includes a hidden-hidden bias

Returns:

  • output – output array of shape (seq_len, batch, num_directions * hidden_size) or (batch, seq_len, num_directions * hidden_size), depending on batch_first

  • h_outs – final hidden state of shape (num_layers * num_directions, batch, hidden_size)

  • c_outs – final cell state of shape (num_layers * num_directions, batch, hidden_size)

ivy.lstm_update(x, init_h, init_c, kernel, recurrent_kernel, /, *, bias=None, recurrent_bias=None, time_major=False)[source]#

Perform long-short term memory update by unrolling time dimension of input array.

Parameters:

  • x (Union[Array, NativeArray]) – input tensor of LSTM layer [batch_shape, t, in] if time_major=False, else [t, batch_shape, in].

  • init_h (Union[Array, NativeArray]) – initial state tensor for the cell output [batch_shape, out].

  • init_c (Union[Array, NativeArray]) – initial state tensor for the cell hidden state [batch_shape, out].

  • kernel (Union[Array, NativeArray]) – weights for cell kernel [in, 4 x out].

  • recurrent_kernel (Union[Array, NativeArray]) – weights for cell recurrent kernel [out, 4 x out].

  • bias (Optional[Union[Array, NativeArray]], default: None) – bias for cell kernel [4 x out]. (Default value = None)

  • recurrent_bias (Optional[Union[Array, NativeArray]], default: None) – bias for cell recurrent kernel [4 x out]. (Default value = None)

  • time_major (bool, default: False) – whether or not the input tensor x has the time dimension before batch dim.

Return type:

Tuple[Array, Tuple[Array, Array]]

Returns:

ret – hidden state for all timesteps of shape [batch_shape,t,out] if time_major is False, else [t, batch_shape, out], and a tuple containing the final cell states, both of shape [batch_shape,out].

ivy.multi_head_attention(query, /, *, key=None, value=None, batch_first=True, num_heads=8, scale=None, attention_mask=None, in_proj_weights=None, q_proj_weights=None, k_proj_weights=None, v_proj_weights=None, out_proj_weights=None, in_proj_bias=None, out_proj_bias=None, is_causal=False, key_padding_mask=None, bias_k=None, bias_v=None, static_k=None, static_v=None, add_zero_attn=False, return_attention_weights=False, average_attention_weights=True, dropout=0.0, training=False, out=None)[source]#

Apply multi-head attention to inputs x. This is an implementation of multi-headed attention as described in the paper “Attention is all you Need” (Vaswani et al., 2017). If query, key, value are the same, then this is self-attention. Each timestep in query attends to the corresponding sequence in key, and returns a fixed-width vector. This layer first projects query, key and value. These are (effectively) a list of tensors of length num_attention_heads, where the corresponding shapes are (batch_size, <query dimensions>, key_dim), (batch_size, <key/value dimensions>, key_dim), (batch_size, <key/value dimensions>, value_dim). Then, the query and key tensors are dot-producted and scaled. These are softmaxed to obtain attention probabilities. The value tensors are then interpolated by these probabilities, then concatenated back to a single tensor. Finally, the result tensor with the last dimension as value_dim can take a linear projection and return.

Parameters:

  • query (Union[Array, NativeArray]) – The query embeddings. Shape: (L, Q) or (N, L, Q), where L is the number of queries, N is the batch size, Q is the query embedding dimension.

  • key (Optional[Union[Array, NativeArray]], default: None) – The key embeddings. Shape: (S, K) or (N, S, K), where S is the number of keys, N is the batch size, K is the key embedding dimension.

  • value (Optional[Union[Array, NativeArray]], default: None) – The value embeddings. Shape (S, V) or (N, S, V), where S is the number of keys, N is the batch size, V is the value embedding dimension.

  • batch_first (bool, default: True) – If False, query, key and value will have shapes (L, N, Q), (S, N, K) and (S, N, V) respectively (if batched).

  • num_heads (int, default: 8) – The number of attention heads to use.

  • scale (Optional[float], default: None) – The value by which to scale the query-key similarity measure before softmax.

  • attention_mask (Optional[Union[Array, NativeArray]], default: None) – The mask to apply to the query-key values. Shape: (L, S) or (N*num_heads, L, S).

  • in_proj_weights (Optional[Union[Array, NativeArray]], default: None) – The weights used to project query, key and value. Shape: (3*E, E’), where E is the new embedding dimension and E’ is the input embedding dimension, i.e. E’ = Q = K = V.

  • q_proj_weights (Optional[Union[Array, NativeArray]], default: None) – The weights used to project query if in_proj_weights is None. Shape: (E, Q).

  • k_proj_weights (Optional[Union[Array, NativeArray]], default: None) – The weights used to project key if in_proj_weights is None. Shape: (E, K).

  • v_proj_weights (Optional[Union[Array, NativeArray]], default: None) – The weights used to project value if in_proj_weights is None. Shape: (E, V).

  • out_proj_weights (Optional[Union[Array, NativeArray]], default: None) – The weights used to project the attention output. Shape: (O, E), where O is the output embedding dimension.

  • in_proj_bias (Optional[Union[Array, NativeArray]], default: None) – The bias used when projecting query, key and value. Shape: (3*E,).

  • out_proj_bias (Optional[Union[Array, NativeArray]], default: None) – The bias used when projecting the output. Shape: (O,).

  • is_causal (bool, default: False) – If True, use a causal attention mask and ignore the provided attention_mask.

  • key_padding_mask (Optional[Union[Array, NativeArray]], default: None) – A binary mask to apply to the key sequence. Shape: (S,) or (N, S).

  • bias_k (Optional[Union[Array, NativeArray]], default: None) – An additional bias added to the key sequence. Shape: (E,).

  • bias_v (Optional[Union[Array, NativeArray]], default: None) – An additional bias added to the value sequence. Shape: (E,).

  • static_k (Optional[Union[Array, NativeArray]], default: None) – A static key to be used in the attention operators. Shape: (N*num_heads, S, E//num_heads).

  • static_v (Optional[Union[Array, NativeArray]], default: None) – A static value to be used in the attention operators. Shape: (N*num_heads, S, E//num_heads).

  • add_zero_attn (bool, default: False) – A boolean flag indicating whether to add a batch of zeros to key and value.

  • return_attention_weights (bool, default: False) – If True, return the attention weights alongside the attention output.

  • average_attention_weights (bool, default: True) – If True, the returned attention weights will be averaged across heads. Otherwise, the attention weights will be provided separately per head. Note that this flag only has an effect when return_attention_weights=True.

  • dropout (float, default: 0.0) – Specifies the dropout probability. Dropout is applied on the attention weights.

  • training (bool, default: False) – If True, dropout is used, otherwise dropout is not activated.

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Union[Array, NativeArray]

Returns:

  • ret – The output following the application of multi-head attention. Either output or (output, attention_weights). output will have shape (L, E) if the inputs were unbatched or (N, L, E) otherwise, and attention_weights will have shape (L, S) or (N, L, S) respectively. If batch_first is False and the inputs were batched, the output will have shape (L, N, E).

  • Both the description and the type hints above assumes an array input for simplicity,

  • but this function is nestable, and therefore also accepts ivy.Container

  • instances in place of any of the arguments.

ivy.nms(boxes, scores=None, iou_threshold=0.5, max_output_size=None, score_threshold=-inf)[source]#
ivy.roi_align(input, boxes, output_size, spatial_scale=1.0, sampling_ratio=-1, aligned=False)[source]#
ivy.scaled_dot_product_attention(query, key, value, /, *, scale=None, mask=None, dropout_p=0.0, is_causal=False, training=False, out=None)[source]#

Apply scaled dot product attention to inputs x using optional mask.

Parameters:

  • query (Union[Array, NativeArray]) – The queries input array. The shape of queries input array should be in [batch_shape,num_queries,feat_dim]. The queries input array should have the same size as keys and values.

  • key (Union[Array, NativeArray]) – The keys input array. The shape of keys input array should be in [batch_shape,num_keys,feat_dim]. The keys input array should have the same size as queries and values.

  • value (Union[Array, NativeArray]) – The values input array. The shape of values input should be in [batch_shape,num_keys,feat_dim]. The values input array should have the same size as queries and keys.

  • scale (Optional[float], default: None) – The scale float value. The scale float value is used to scale the query-key pairs before softmax.

  • mask (Optional[Union[Array, NativeArray]], default: None) – The mask input array. The mask to apply to the query-key values. Default is None. The shape of mask input should be in [batch_shape,num_queries,num_keys].

  • dropout_p (Optional[float], default: 0.0) – Specifies the dropout probability, if greater than 0.0, dropout is applied

  • is_causal (Optional[bool], default: False) – If true, assumes causal attention masking and errors if both mask and is_causal are set.

  • training (Optional[bool], default: False) – If True, dropout is used, otherwise dropout is not activated.

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

  • ret – The output following application of scaled dot-product attention. The output array is the weighted sum produced by the attention score and value. The shape of output array is [batch_shape,num_queries,feat_dim] .

  • Both the description and the type hints above assumes an array input for simplicity,

  • but this function is nestable, and therefore also accepts ivy.Container

  • instances in place of any of the arguments.

Examples

With ivy.Array input:

>>> q = ivy.array([[[0.2, 1.], [2.2, 3.],[4.4, 5.6]]])
>>> k = ivy.array([[[0.6, 1.5], [2.4, 3.3],[4.2, 5.1]]])
>>> v = ivy.array([[[0.4, 1.3], [2.2, 3.1],[4.3, 5.3]]])
>>> result = ivy.scaled_dot_product_attention(q,
...                                           k,
...                                           v,
...                                           scale=1,
...                                           dropout_p=0.1,
...                                           is_causal=True,
...                                           training=True)
>>> print(result)

ivy.array([[[0.40000001, 1.29999995], … [2.19994521, 3.09994531], … [4.30000019, 5.30000019]]])

>>> q = ivy.array([[[0.2, 1.], [2.2, 3.],[4.4, 5.6]]])
>>> k = ivy.array([[[0.6, 1.5], [2.4, 3.3],[4.2, 5.1]]])
>>> v = ivy.array([[[0.4, 1.3], [2.2, 3.1],[4.3, 5.3]]])
>>> mask = ivy.array([[[0.0, 0.0, 0.0], [0.0, 0.0, 0.0],[0.0, 0.0, 0.0]]])
>>> result = ivy.scaled_dot_product_attention(q,k,v,scale=1,mask=mask)
>>> print(result)
ivy.array([[[2.30000019, 3.23333359],

[2.30000019, 3.23333359], [2.30000019, 3.23333359]]])

>>> q = ivy.array([[[0.2, 1.], [2.2, 3.], [4.4, 5.6]]])
>>> k = ivy.array([[[0.6, 1.5], [2.4, 3.3], [4.2, 5.1]]])
>>> v = ivy.array([[[0.4, 1.3], [2.2, 3.1], [4.3, 5.3]]])
>>> out = ivy.zeros(shape=(1, 3, 2))
>>> ivy.scaled_dot_product_attention(q,
...                                  k,
...                                  v,
...                                  scale=1,
...                                  dropout_p=0.1,
...                                  is_causal=True,
...                                  training=True,
...                                  out=out)
>>> print(out)

ivy.array([[[0.40000001, 1.29999995], … [2.19994521, 3.09994531], … [4.30000019, 5.30000019]]])

>>> q = ivy.native_array([[[0.2, 1.], [2.2, 3.],[4.4, 5.6]]])
>>> k = ivy.native_array([[[0.6, 1.5], [2.4, 3.3],[4.2, 5.1]]])
>>> v = ivy.native_array([[[0.4, 1.3], [2.2, 3.1],[4.3, 5.3]]])
>>> mask = ivy.native_array([[[0.0, 0.0, 0.0], [0.0, 0.0, 0.0],[0.0, 0.0, 0.0]]])
>>> result = ivy.scaled_dot_product_attention(q,k,v,scale=1,mask=mask)
>>> print(result)

ivy.array([[[2.30000019, 3.23333359], … [2.30000019, 3.23333359], … [2.30000019, 3.23333359]]])

>>> q = ivy.native_array([[[0.2, 1.], [2.2, 3.], [4.4, 5.6]]])
>>> k = ivy.native_array([[[0.6, 1.5], [2.4, 3.3], [4.2, 5.1]]])
>>> v = ivy.native_array([[[0.4, 1.3], [2.2, 3.1], [4.3, 5.3]]])
>>> out = ivy.zeros(shape=(1, 3, 2))
>>> ivy.scaled_dot_product_attention(q,
...                                  k,
...                                  v,
...                                  scale=1,
...                                  dropout_p=0.1,
...                                  is_causal=True,
...                                  training=True,
...                                  out=out)
>>> print(out)

ivy.array([[[0.40000001, 1.29999995], … [2.19994521, 3.09994531], … [4.30000019, 5.30000019]]])

With ivy.Container input:

>>> q = ivy.Container(a=ivy.array([[[0.2, 1.], [2.7, 3.], [4.4, 5.6]]]),
...                   b=ivy.array([[[1.2, 1.], [2.2, 3.], [4.4, 5.6]]]))
>>> k = ivy.Container(a=ivy.array([[[4.2, 1.], [2.2, 3.3], [4.4, 5.6]]]),
...                   b=ivy.array([[[3.2, 1.], [2.2, 3.6], [4.0, 5.6]]]))
>>> v = ivy.Container(a=ivy.array([[[5.2, 1.], [2.1, 3.], [4.4, 5.6]]]),
...                   b=ivy.array([[[0.2, 1.], [2.2, 3.], [4.4, 5.6]]]))
>>> result = ivy.scaled_dot_product_attention(q,
...                                           k,
...                                           v,
...                                           scale=1,
...                                           dropout_p=0.1,
...                                           is_causal=True,
...                                           training=True)
>>> print(result)
{
    a: ivy.array([[[5.19999981, 1.],
    ...            [2.59249449, 2.68226194],
    ...            [4.4000001, 5.5999999]]]),
    b: ivy.array([[[0.2, 1.],
    ...            [2.19603825, 2.9960382],
    ...            [4.4000001, 5.5999999]]])
}

>>> q = ivy.Container(a=ivy.array([[[0.2, 1.], [2.7, 3.], [4.4, 5.6]]]),
...                   b=ivy.array([[[1.2, 1.], [2.2, 3.], [4.4, 5.6]]]))
>>> k = ivy.Container(a=ivy.array([[[4.2, 1.], [2.2, 3.3], [4.4, 5.6]]]),
...                   b=ivy.array([[[3.2, 1.], [2.2, 3.6], [4.0, 5.6]]]))
>>> v = ivy.Container(a=ivy.array([[[5.2, 1.], [2.1, 3.], [4.4, 5.6]]]),
...                   b=ivy.array([[[0.2, 1.], [2.2, 3.], [4.4, 5.6]]]))
>>> mask = ivy.Container(
...     a=ivy.array([[[1.0, 1.0, 1.0],[1.0, 1.0, 1.0],[1.0, 1.0, 1.0]]]),
...     b=ivy.array([[[1.0, 1.0, 1.0], [1.0, 1.0, 1.0], [1.0, 1.0,1.0]]])
... )
>>> result = ivy.scaled_dot_product_attention(q,k,v,scale=1,mask=mask)
>>> print(result)
{
    a: ivy.array([[[4.26894283, 5.40236187],
    ...            [4.39999437, 5.59999037],
    ...            [4.4000001, 5.5999999]]]),
    b: ivy.array([[[4.35046196, 5.54282808],
    ...            [4.39989519, 5.5998764],
    ...            [4.4000001, 5.5999999]]])
}

With a mix of ivy.Array and ivy.NativeArray inputs:

>>> q = ivy.array([[[0.2, 1.], [2.2, 3.],[4.4, 5.6]]])
>>> k = ivy.native_array([[[0.6, 1.5], [2.4, 3.3],[4.2, 5.1]]])
>>> v = ivy.native_array([[[0.4, 1.3], [2.2, 3.1],[4.3, 5.3]]])
>>> result = ivy.scaled_dot_product_attention(q,
...                                            k,
...                                            v,
...                                            scale=1,
...                                            dropout_p=0.1,
...                                            is_causal=True,
...                                            training=True)
>>> print(result)

ivy.array([[[0.40000001, 1.29999995], … [2.19994521, 3.09994531], … [4.30000019, 5.30000019]]])

>>> q = ivy.array([[[0.2, 1.], [2.2, 3.], [4.4, 5.6]]])
>>> k = ivy.native_array([[[0.6, 1.5], [2.4, 3.3], [4.2, 5.1]]])
>>> v = ivy.native_array([[[0.4, 1.3], [2.2, 3.1], [4.3, 5.3]]])
>>> out = ivy.zeros(shape=(1, 3, 2))
>>> ivy.scaled_dot_product_attention(q,k,v,scale=1,out=out)
>>> print(out)
ivy.array([[[4.03946018, 5.0280633 ],
...         [4.29981947, 5.29981089],
...         [4.30000019, 5.30000019]]])

With a mix of ivy.Array and ivy.Container inputs:

>>> q = ivy.array([[[0.2, 1.], [2.2, 3.],[4.4, 5.6]]])
>>> k = ivy.Container(a=ivy.array([[[4.2, 1.], [2.2, 3.3], [4.4, 5.6]]]),
...                   b=ivy.array([[[3.2, 1.], [2.2, 3.6], [4.0, 5.6]]]))
>>> v = ivy.array([[[0.4, 1.3], [2.2, 3.1], [4.3, 5.3]]])
>>> result = ivy.scaled_dot_product_attention(q,k,v,scale=1,is_causal=True)
>>> print(result)
{
    a: ivy.array([[[0.40000001, 1.29999995],
    ...            [2.06345534, 2.9634552],
    ...            [4.30000019, 5.30000019]]]),
    b: ivy.array([[[0.40000001, 1.29999995],
    ...            [2.19336844, 3.09336829],
    ...            [4.30000019, 5.30000019]]])
}
With a mix of :class:`ivy.Array` and :class:`ivy.Container` inputs:

>>> q = ivy.array([[[0.2, 1.], [2.2, 3.],[4.4, 5.6]]])
>>> k = ivy.Container(a=ivy.array([[[4.2, 1.], [2.2, 3.3],[4.4, 5.6]]]),
...                   b=ivy.array([[[3.2, 1.], [2.2, 3.6],[4.0, 5.6]]]))
>>> v = ivy.array([[[0.4, 1.3], [2.2, 3.1],[4.3, 5.3]]])
>>> mask = ivy.native_array([[[0.0, 0.0, 0.0], [0.0, 0.0, 0.0], [0.0, 0.0, 0.0]]])
>>> result = ivy.scaled_dot_product_attention(q,
...                                           k,
...                                           v,
...                                           scale=1,
...                                           mask=mask,
...                                           dropout_p=0.1,
...                                           training=True)
>>> print(result)
{
    a: ivy.array([[[2.30000019, 3.23333359],
    ...            [2.30000019, 3.23333359],
    ...            [2.30000019, 3.23333359]]]),
    b: ivy.array([[[2.30000019, 3.23333359],
    ...            [2.30000019, 3.23333359],
    ...            [2.30000019, 3.23333359]]])
}

This should have hopefully given you an overview of the layers submodule, if you have any questions, please feel free to reach out on our discord in the layers channel!

On this page