Activations

Collection of Ivy activation functions.

ivy.deserialize(name, /, *, custom_objects=typing.Union[typing.Dict, NoneType])[source]

Returns activation function given a string identifier.

Parameters
  • name (Optional[str]) – The name of the activation function.

  • custom_objects – Optional dictionary listing user-provided activation functions.

Return type

Optional[Callable]

Returns

ret – Corresponding activation function.

Examples

With :str: input:

>>> name = "sigmoid"
>>> sigmoid = ivy.deserialize(name)
>>> print(sigmoid)
<function sigmoid at XXXXXXXXXXXXXX>

With :str and dict: input:

>>> name = "custom_fn"
>>> objects = {"custom_fn": lambda x: x}
>>> custom_fn = ivy.deserialize(name, custom_objects=objects)
>>> print(custom_fn)
<function custom_fn at XXXXXXXXXXXXXX>
ivy.gelu(x, /, *, approximate=False, out=None)[source]

Applies the Gaussian error linear unit (GELU) activation function.

Parameters
  • x (Union[Array, NativeArray]) – Input array.

  • approximate (bool) – Whether to approximate, default is True. (default: False)

  • out (Optional[Array]) – optional output array, for writing the result to. It must have a shape that the (default: None) inputs broadcast to.

Return type

Array

Returns

ret – The input array with gelu applied element-wise.

Examples

With ivy.Array input:

>>> x = ivy.array([-1.2, -0.6, 1.5])
>>> y = ivy.gelu(x)
>>> y
ivy.array([-0.138, -0.165, 1.4])

With ivy.NativeArray input:

>>> x = ivy.native_array([-1.3, 3.8, 2.1])
>>> y = ivy.gelu(x)
>>> y
ivy.array([-0.126, 3.8, 2.06])

With ivy.Container input:

>>> x = ivy.Container(a=ivy.array([1., 2.]), b=ivy.array([-0.9, -1.]))
>>> y = ivy.gelu(x)
>>> y
{
    a: ivy.array([0.841, 1.95]),
    b: ivy.array([-0.166, -0.159])
}
ivy.get(name, /, *, custom_objects=typing.Union[typing.Dict, NoneType])[source]

Returns activation function given a string identifier.

Parameters
  • name (Optional[str]) – The name of the activation function.

  • custom_objects – Optional dictionary listing user-provided activation functions.

Return type

Optional[Callable]

Returns

ret – Corresponding activation function.

Examples

With :str: input:

>>> name = "sigmoid"
>>> sigmoid = ivy.get(name)
>>> print(sigmoid)
<function sigmoid at XXXXXXXXXXXXXX>
>>> name = None
>>> linear = ivy.get(name)
>>> print(linear)
<function linear at XXXXXXXXXXXXXX>

With :str and dict: input:

>>> name = "custom_fn"
>>> objects = {"custom_fn": lambda x: x}
>>> custom_fn = ivy.get(name, custom_objects=objects)
>>> print(custom_fn)
<function custom_fn at XXXXXXXXXXXXXX>
ivy.leaky_relu(x, /, *, alpha=0.2, out=None)[source]

Applies the leaky rectified linear unit function element-wise.

Parameters
  • x (Union[Array, NativeArray]) – Input array.

  • alpha (float) – Negative slope for ReLU. (default: 0.2)

  • out (Optional[Array]) – optional output array, for writing the result to. It must have a shape that the (default: None) inputs broadcast to.

Return type

Array

Returns

ret – The input array with leaky relu applied element-wise.

Examples

With ivy.Array input:

>>> x = ivy.array([0.39, -0.85])
>>> y = ivy.leaky_relu(x)
>>> print(y)
ivy.array([ 0.39, -0.17])
>>> x = ivy.array([1.5, 0.7, -2.4])
>>> y = ivy.zeros(3)
>>> ivy.leaky_relu(x, out = y)
>>> print(y)
ivy.array([ 1.5 ,  0.7 , -0.48])
>>> x = ivy.array([[1.1, 2.2, 3.3],
...                [-4.4, -5.5, -6.6]])
>>> ivy.leaky_relu(x, out = x)
>>> print(x)
ivy.array([[ 1.1 ,  2.2 ,  3.3 ],
   [-0.88, -1.1 , -1.32]])

With ivy.NativeArray input:

>>> x = ivy.native_array([0., -1., 2.])
>>> y = ivy.leaky_relu(x)
>>> print(y)
ivy.array([ 0. , -0.2,  2. ])
ivy.log_softmax(x, /, *, axis=None, out=None)[source]

Applies the log_softmax function element-wise.

Parameters
  • x (Union[Array, NativeArray]) – Input array.

  • axis (Optional[int]) – The dimension log_softmax would be performed on. The default is None. (default: None)

  • out (Optional[Array]) – optional output array, for writing the result to. It must have a shape that the (default: None) inputs broadcast to.

Return type

Array

Returns

ret – The output array with log_softmax applied element-wise to input.

Examples

With ivy.Array input:

>>> x = ivy.array([-1.0, -0.98])
>>> y = ivy.log_softmax(x)
>>> print(y)
ivy.array([-0.703, -0.683])
>>> x = ivy.array([1.0, 2.0, 3.0])
>>> y = ivy.log_softmax(x)
>>> print(y)
ivy.array([-2.41, -1.41, -0.408])

With ivy.NativeArray input:

>>> x = ivy.native_array([1.5, 0.5, 1.0])
>>> y = ivy.log_softmax(x)
>>> print(y)
ivy.array([-0.68, -1.68, -1.18])

With ivy.Container input:

>>> x = ivy.Container(a=ivy.array([1.5, 0.5, 1.0]))
>>> y = ivy.log_softmax(x)
>>> print(y)
{
    a: ivy.array([-0.68, -1.68, -1.18])
}
>>> x = ivy.Container(a=ivy.array([1.0, 2.0]), b=ivy.array([0.4, -0.2]))
>>> y = ivy.log_softmax(x)
>>> print(y)
{
    a: ivy.array([-1.31, -0.313]),
    b: ivy.array([-0.437, -1.04])
}
ivy.relu(x, /, *, out=None)[source]

Applies the rectified linear unit function element-wise.

Parameters
  • x (Union[Array, NativeArray]) – input array

  • out (Optional[Array]) – optional output array, for writing the result to. It must have a shape that the (default: None) inputs broadcast to.

Return type

Array

Returns

ret – an array containing the rectified linear unit activation of each element in x.

Examples

With ivy.Array input:

>>> x = ivy.array([-1., 0., 1.])
>>> y = ivy.relu(x)
>>> print(y)
ivy.array([0., 0., 1.])
>>> x = ivy.array([1.5, 0.7, -2.4])
>>> y = ivy.zeros(3)
>>> ivy.relu(x, out = y)
>>> print(y)
ivy.array([1.5, 0.7, 0.])

With ivy.NativeArray input:

>>> x = ivy.native_array([0., -1., 2.])
>>> y = ivy.relu(x)
>>> print(y)
ivy.array([0., 0., 2.])

With ivy.Container input:

>>> x = ivy.Container(a=ivy.array([1.0, -1.2]), b=ivy.array([0.4, -0.2]))
>>> x = ivy.relu(x, out = x)
>>> print(x)
{
a: ivy.array([1., 0.]),
b: ivy.array([0.40000001, 0.])
}
ivy.sigmoid(x, /, *, out=None)[source]

Applies the sigmoid function element-wise.

Parameters
  • x (Union[Array, NativeArray]) – input array.

  • out (Optional[Array]) – optional output array, for writing the result to. It must have a shape that the (default: None) inputs broadcast to.

Return type

Array

Returns

ret – an array containing the sigmoid activation of each element in x.

Functional Examples

With ivy.Array input:

>>> x = ivy.array([-1., 1., 2.])
>>> y = ivy.sigmoid(x)
>>> print(y)
ivy.array([0.269, 0.731, 0.881])

With ivy.NativeArray input:

>>> x = ivy.native_array([-1.3, 3.8, 2.1])
>>> y = ivy.sigmoid(x)
>>> print(y)
ivy.array([0.214, 0.978, 0.891])

Instance Method Example

Using ivy.Array instance method:

>>> x = ivy.array([-1., 1., 2.])
>>> y = x.sigmoid()
>>> print(y)
ivy.array([0.269, 0.731, 0.881])
ivy.softmax(x, /, *, axis=None, out=None)[source]

Applies the softmax function element-wise.

Parameters
  • x (Union[Array, NativeArray]) – Input array.

  • axis (Optional[int]) – The dimension softmax would be performed on. The default is None. (default: None)

  • out (Optional[Array]) – optional output array, for writing the result to. It must have a shape that the (default: None) inputs broadcast to.

Return type

Array

Returns

ret – The input array with softmax applied element-wise.

Functional Examples

With ivy.Array input:

>>> x = ivy.array([1.0, 0, 1.0])
>>> y = ivy.softmax(x)
>>> print(y)
ivy.array([0.422, 0.155, 0.422])
>>> x = ivy.array([[1.1, 2.2, 3.3],
...                [4.4, 5.5, 6.6]])
>>> y = ivy.softmax(x, axis = 1)
>>> print(y)
ivy.array([[0.0768, 0.231 , 0.693 ],
           [0.0768, 0.231 , 0.693 ]])

With ivy.NativeArray input:

>>> x = ivy.native_array([1.5, 0.3, 1.2])
>>> y = ivy.softmax(x)
>>> print(y)
ivy.array([0.49 , 0.147, 0.363])

Instance Method Example

Using ivy.Array instance method:

>>> x = ivy.array([1.0, 0, 1.0])
>>> y = x.softmax()
>>> print(y)
ivy.array([0.422, 0.155, 0.422])
ivy.softplus(x, /, *, beta=None, threshold=None, out=None)[source]

Applies the softplus function element-wise.

Parameters
  • x (Union[Array, NativeArray]) – input array.

  • beta (Optional[Union[int, float]]) – The beta value for the softplus formation. Default: None. (default: None)

  • threshold (Optional[Union[int, float]]) – values above this revert to a linear function. Default: None. (default: None)

  • out (Optional[Array]) – optional output array, for writing the result to. It must have a shape that the (default: None) inputs broadcast to.

Return type

Array

Returns

ret – an array containing the softplus activation of each element in x.

Functional Examples

With ivy.Array input:

>>> x = ivy.array([-0.3461, -0.6491])
>>> y = ivy.softplus(x)
>>> print(y)
ivy.array([0.535,0.42])
>>> x = ivy.array([-0.3461, -0.6491])
>>> y = ivy.softplus(x, beta=0.5)
>>> print(y)
ivy.array([1.22, 1.09])
>>> x = ivy.array([1., 2., 3.])
>>> y = ivy.softplus(x, threshold=2)
>>> print(y)
ivy.array([1.31, 2.13, 3.  ])

With ivy.NativeArray input:

>>> x = ivy.native_array([-0.3461, -0.6491])
>>> y = ivy.softplus(x)
>>> print(y)
ivy.array([0.535,0.42])

Instance Method Example

Using ivy.Array instance method:

>>> x = ivy.array([-0.3461, -0.6491])
>>> y = x.softplus()
>>> print(y)
ivy.array([0.535,0.42])

This should have hopefully given you an overview of the activations submodule,If you have any questions, please feel free to reach out on our discord in the activations channel or in the activations forum!