Activations

Collection of Ivy activation functions.

ivy.gelu(x, approximate=True, *, out=None)[source]

Applies the Gaussian error linear unit (GELU) activation function.

Parameters
  • x (Union[Array, NativeArray]) – Input array.

  • approximate – Whether to approximate, default is True.

  • out (Optional[Array]) – optional output array, for writing the result to. It must have a shape that the (default: None) inputs broadcast to.

Returns

ret – The input array with gelu applied element-wise.

ivy.leaky_relu(x, alpha=0.2, *, out=None)[source]

Applies the leaky rectified linear unit function element-wise.

Parameters
  • x (Union[Array, NativeArray]) – Input array.

  • alpha (Optional[float]) – Negative slope for ReLU. (default: 0.2)

  • out (Optional[Array]) – optional output array, for writing the result to. It must have a shape that the (default: None) inputs broadcast to.

Return type

Array

Returns

ret – The input array with leaky relu applied element-wise.

Functional Examples

With :code: ivy.Array input:

>>> x = ivy.array([0.39, -0.85])
>>> y = ivy.leaky_relu(x)
>>> print(y)
ivy.array([ 0.39, -0.17])
>>> x = ivy.array([1.5, 0.7, -2.4])
>>> y = ivy.zeros(3)
>>> ivy.leaky_relu(x, out = y)
>>> print(y)
ivy.array([ 1.5 ,  0.7 , -0.48])
>>> x = ivy.array([[1.1, 2.2, 3.3],                        [-4.4, -5.5, -6.6]])
>>> ivy.leaky_relu(x, out = x)
>>> print(x)
ivy.array([[ 1.1 ,  2.2 ,  3.3 ],
   [-0.88, -1.1 , -1.32]])

With :code: ivy.NativeArray input:

>>> x = ivy.native_array([0., -1., 2.])
>>> y = ivy.leaky_relu(x)
>>> print(y)
ivy.array([ 0. , -0.2,  2. ])

Instance Method Examples

Using :code: ivy.Array instance method:

>>> x = ivy.array([-0.5, 1., -2.5])
>>> y = x.leaky_relu()
>>> print(y)
ivy.array([-0.1,  1. , -0.5])
ivy.relu(x, *, out=None)[source]

Applies the rectified linear unit function element-wise.

Parameters
  • x (Union[Array, NativeArray]) – input array

  • out (Optional[Array]) – optional output array, for writing the result to. It must have a shape that the (default: None) inputs broadcast to.

Return type

Array

Returns

ret – an array containing the rectified linear unit activation of each element in x.

Functional Examples

With :code: ivy.Array input:

>>> x = ivy.array([-1., 0., 1.])
>>> y = ivy.relu(x)
>>> print(y)
ivy.array([0., 0., 1.])
>>> x = ivy.array([1.5, 0.7, -2.4])
>>> y = ivy.zeros(3)
>>> ivy.relu(x, out = y)
>>> print(y)
ivy.array([1.5, 0.7, 0.])
>>> x = ivy.array([[1.1, 2.2, 3.3],                        [-4.4, -5.5, -6.6]])
>>> ivy.relu(x, out = x)
>>> print(x)
ivy.array([[1.1, 2.2, 3.3],
           [0., 0., 0.]])

With :code: ivy.NativeArray input:

>>> x = ivy.native_array([0., -1., 2.])
>>> y = ivy.relu(x)
>>> print(y)
ivy.array([0., 0., 2.])

Instance Method Examples

Using :code: ivy.Array instance method:

>>> x = ivy.array([-0.5, 1., -2.5])
>>> y = x.relu()
>>> print(y)
ivy.array([0., 1., 0.])
ivy.sigmoid(x, *, out=None)[source]

Applies the sigmoid function element-wise.

Parameters
  • x (Union[Array, NativeArray]) – input array.

  • out (Optional[Array]) – optional output array, for writing the result to. It must have a shape that the (default: None) inputs broadcast to.

Return type

Array

Returns

ret – an array containing the sigmoid activation of each element in x.

Functional Examples

With :code: ivy.Array input:

>>> x = ivy.array([-1., 1., 2.])
>>> y = ivy.sigmoid(x)
>>> print(y)
ivy.array([0.269, 0.731, 0.881])

With :code: ivy.NativeArray input:

>>> x = ivy.native_array([-1.3, 3.8, 2.1])
>>> y = ivy.sigmoid(x)
>>> print(y)
ivy.array([0.214, 0.978, 0.891])

Instance Method Example

Using :code: ivy.Array instance method:

>>> x = ivy.array([-1., 1., 2.])
>>> y = x.sigmoid()
>>> print(y)
ivy.array([0.269, 0.731, 0.881])
ivy.softmax(x, axis=-1, *, out=None)[source]

Applies the softmax function element-wise.

Parameters
  • x (Union[Array, NativeArray]) – Input array.

  • axis (Optional[int]) – The dimension softmax would be performed on. The default is -1 which indicates (default: -1) the last dimension.

  • out (Optional[Array]) – optional output array, for writing the result to. It must have a shape that the (default: None) inputs broadcast to.

Return type

Array

Returns

ret – The input array with softmax applied element-wise.

Functional Examples

With :code: ivy.Array input:

>>> x = ivy.array([1.0, 0, 1.0])
>>> y = ivy.softmax(x)
>>> print(y)
ivy.array([0.422, 0.155, 0.422])
>>> x = ivy.array([[1.1, 2.2, 3.3],                        [4.4, 5.5, 6.6]])
>>> y = ivy.softmax(x, axis = 1)
>>> print(y)
ivy.array([[0.0768, 0.231 , 0.693 ],
           [0.0768, 0.231 , 0.693 ]])

With :code: ivy.NativeArray input:

>>> x = ivy.native_array([1.5, 0.3, 1.2])
>>> y = ivy.softmax(x)
>>> print(y)
ivy.array([0.49 , 0.147, 0.363])

Instance Method Example

Using :code: ivy.Array instance method:

>>> x = ivy.array([1.0, 0, 1.0])
>>> y = x.softmax()
>>> print(y)
ivy.array([0.422, 0.155, 0.422])
ivy.softplus(x, *, out=None)[source]

Applies the softplus function element-wise.

Parameters
  • x (Union[Array, NativeArray]) – input array.

  • out (Optional[Array]) – optional output array, for writing the result to. It must have a shape that the (default: None) inputs broadcast to.

Return type

Array

Returns

ret – an array containing the softplus activation of each element in x.

Functional Examples

With :code: ivy.Array input:

>>> x = ivy.array([-0.3461, -0.6491])
>>> y = ivy.softplus(x)
>>> print(y)
ivy.array([0.535,0.42])

With :code: ivy.NativeArray input:

>>> x = ivy.native_array([-0.3461, -0.6491])
>>> y = ivy.softplus(x)
>>> print(y)
ivy.array([0.535,0.42])

Instance Method Example

Using :code: ivy.Array instance method:

>>> x = ivy.array([-0.3461, -0.6491])
>>> y = x.softplus()
>>> print(y)
ivy.array([0.535,0.42])
ivy.tanh(x, *, out=None)[source]

Calculates an implementation-dependent approximation to the hyperbolic tangent, having domain [-infinity, +infinity] and codomain [-1, +1], for each element x_i of the input array x.

Special cases

For floating-point operands,

  • If x_i is NaN, the result is NaN.

  • If x_i is +0, the result is +0.

  • If x_i is -0, the result is -0.

  • If x_i is +infinity, the result is +1.

  • If x_i is -infinity, the result is -1.

Parameters
  • x (Union[Array, NativeArray]) – input array whose elements each represent a hyperbolic angle. Should have a real-valued floating-point data type.

  • out (Optional[Array]) – optional output, for writing the result to. It must have a shape that the inputs (default: None) broadcast to.

Return type

Array

Returns

ret – an array containing the hyperbolic tangent of each element in x. The returned array must have a real-valued floating-point data type determined by type-promotion.

This method conforms to the Array API Standard. This docstring is an extension of the docstring in the standard. The descriptions above assume an array input for simplicity, but the method also accepts ivy.Container instances in place of ivy.Array or ivy.NativeArray instances, as shown in the type hints and also the examples below.

Examples

With ivy.Array input:

>>> x = ivy.array([0., 1., 2.])
>>> y = ivy.tanh(x)
>>> print(y)
ivy.array([0., 0.762, 0.964])
>>> x = ivy.array([0.5, -0.7, 2.4])
>>> y = ivy.zeros(3)
>>> ivy.tanh(x, out=y)
>>> print(y)
ivy.array([0.462, -0.604, 0.984])
>>> x = ivy.array([[1.1, 2.2, 3.3],                      [-4.4, -5.5, -6.6]])
>>> ivy.tanh(x, out=x)
>>> print(x)
ivy.array([[0.8, 0.976, 0.997],
          [-1., -1., -1.]])

With ivy.NativeArray input:

>>> x = ivy.native_array([0., 1., 2.])
>>> y = ivy.tanh(x)
>>> print(y)
ivy.array([0., 0.762, 0.964])

With ivy.Container input:

>>> x = ivy.Container(a=ivy.array([0., 1., 2.]),                          b=ivy.array([3., 4., 5.]))
>>> y = ivy.tanh(x)
>>> print(y)
{
    a: ivy.array([0., 0.762, 0.964]),
    b: ivy.array([0.995, 0.999, 1.])
}