Activations#

Collection of Ivy activation functions.

ivy.gelu(x, /, *, approximate=False, complex_mode='jax', out=None)[source]#

Apply the Gaussian error linear unit (GELU) activation function.

Parameters:

  • x (Union[Array, NativeArray]) – Input array.

  • approximate (bool, default: False) – Whether to approximate, default is True. An approximation is always used if the input array is complex.

  • complex_mode (Literal['split', 'magnitude', 'jax'], default: 'jax') – optional specifier for how to handle complex data types. See ivy.func_wrapper.handle_complex_input for more detail.

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

ret – The input array with gelu applied element-wise.

Examples

With ivy.Array input:

>>> x = ivy.array([-1.2, -0.6, 1.5])
>>> y = ivy.gelu(x)
>>> y
ivy.array([-0.138, -0.165, 1.4])

With ivy.NativeArray input:

>>> x = ivy.native_array([-1.3, 3.8, 2.1])
>>> y = ivy.gelu(x)
>>> y
ivy.array([-0.126, 3.8, 2.06])

With ivy.Container input:

>>> x = ivy.Container(a=ivy.array([1., 2.]), b=ivy.array([-0.9, -1.]))
>>> y = ivy.gelu(x)
>>> y
{
    a: ivy.array([0.841, 1.95]),
    b: ivy.array([-0.166, -0.159])
}
ivy.hardswish(x, /, *, complex_mode='jax', out=None)[source]#

Apply the hardswish activation function element-wise.

Parameters:

  • x (Union[Array, NativeArray]) – input array

  • complex_mode (Literal['split', 'magnitude', 'jax'], default: 'jax') – optional specifier for how to handle complex data types. See ivy.func_wrapper.handle_complex_input for more detail.

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

ret – an array containing the hardswish activation of each element in x.

Examples

With ivy.Array input:

>>> x = ivy.array([0., 0., 4.])
>>> y = ivy.hardswish(x)
>>> y
ivy.array([0., 0., 4.])

With ivy.Container input:

>>> x = ivy.Container(a=ivy.array([-3., 4., 5.]), b=ivy.array([0., 5.]))
>>> x = ivy.hardswish(x, out=x)
>>> x
{
    a: ivy.array([-0.,  4.,  5.]),
    b: ivy.array([0., 5.])
}
ivy.leaky_relu(x, /, *, alpha=0.2, complex_mode='jax', out=None)[source]#

Apply the leaky rectified linear unit function element-wise.

If the input is complex, then by default each element is scaled by alpha if either its real part is strictly negative or if its real part is zero and its imaginary part is negative. This behaviour can be changed by specifying a different complex_mode.

Parameters:

  • x (Union[Array, NativeArray]) – Input array.

  • alpha (float, default: 0.2) – Negative slope for ReLU.

  • complex_mode (Literal['split', 'magnitude', 'jax'], default: 'jax') – optional specifier for how to handle complex data types. See ivy.func_wrapper.handle_complex_input for more detail.

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

ret – The input array with leaky relu applied element-wise.

Examples

With ivy.Array input:

>>> x = ivy.array([0.39, -0.85])
>>> y = ivy.leaky_relu(x)
>>> print(y)
ivy.array([ 0.39, -0.17])

>>> x = ivy.array([1.5, 0.7, -2.4])
>>> y = ivy.zeros(3)
>>> ivy.leaky_relu(x, out=y)
>>> print(y)
ivy.array([ 1.5 ,  0.7 , -0.48])

>>> x = ivy.array([[1.1, 2.2, 3.3],
...                [-4.4, -5.5, -6.6]])
>>> ivy.leaky_relu(x, out=x)
>>> print(x)
ivy.array([[ 1.1 ,  2.2 ,  3.3 ],
   [-0.88, -1.1 , -1.32]])

With ivy.Container input:

>>> x = ivy.Container(a=ivy.array([0.0, -1.2]), b=ivy.array([0.4, -0.2]))
>>> x = ivy.leaky_relu(x, out=x)
>>> print(x)
{
    a: ivy.array([0., -0.24000001]),
    b: ivy.array([0.40000001, -0.04])
}
ivy.log_softmax(x, /, *, axis=None, complex_mode='jax', out=None)[source]#

Apply the log_softmax function element-wise.

Parameters:

  • x (Union[Array, NativeArray]) – Input array.

  • axis (Optional[int], default: None) – The dimension log_softmax would be performed on. The default is None.

  • complex_mode (Literal['split', 'magnitude', 'jax'], default: 'jax') – optional specifier for how to handle complex data types. See ivy.func_wrapper.handle_complex_input for more detail.

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

ret – The output array with log_softmax applied element-wise to input.

Examples

With ivy.Array input:

>>> x = ivy.array([-1.0, -0.98])
>>> y = ivy.log_softmax(x)
>>> print(y)
ivy.array([-0.703, -0.683])

>>> x = ivy.array([1.0, 2.0, 3.0])
>>> y = ivy.log_softmax(x)
>>> print(y)
ivy.array([-2.41, -1.41, -0.408])

With ivy.NativeArray input:

>>> x = ivy.native_array([1.5, 0.5, 1.0])
>>> y = ivy.log_softmax(x)
>>> print(y)
ivy.array([-0.68, -1.68, -1.18])

With ivy.Container input:

>>> x = ivy.Container(a=ivy.array([1.5, 0.5, 1.0]))
>>> y = ivy.log_softmax(x)
>>> print(y)
{
    a: ivy.array([-0.68, -1.68, -1.18])
}

>>> x = ivy.Container(a=ivy.array([1.0, 2.0]), b=ivy.array([0.4, -0.2]))
>>> y = ivy.log_softmax(x)
>>> print(y)
{
    a: ivy.array([-1.31, -0.313]),
    b: ivy.array([-0.437, -1.04])
}
ivy.mish(x, /, *, complex_mode='jax', out=None)[source]#

Apply the mish activation function element-wise.

Parameters:

  • x (Union[Array, NativeArray]) – input array

  • complex_mode (Literal['split', 'magnitude', 'jax'], default: 'jax') – optional specifier for how to handle complex data types. See ivy.func_wrapper.handle_complex_input for more detail.

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

ret – an array containing the mish activation of each element in x.

Examples

With ivy.Array input:

>>> x = ivy.array([-1., 0., 1.])
>>> y = ivy.mish(x)
>>> print(y)
ivy.array([-0.30340147,  0.        ,  0.86509842])

>>> x = ivy.array([1.5, 0.7, -2.4])
>>> y = ivy.zeros(3)
>>> ivy.mish(x, out = y)
>>> print(y)
ivy.array([ 1.40337825,  0.56114835, -0.20788449])

With ivy.Container input:

>>> x = ivy.Container(a=ivy.array([1.0, -1.2]), b=ivy.array([0.4, -0.2]))
>>> x = ivy.mish(x)
>>> print(x)
{
    a: ivy.array([0.86509842, -0.30883577]),
    b: ivy.array([0.28903052, -0.10714479])
}
ivy.relu(x, /, *, complex_mode='jax', out=None)[source]#

Apply the rectified linear unit function element-wise.

If the input is complex, then by default each element is set to zero if either its real part is strictly negative or if its real part is zero and its imaginary part is negative. This behaviour can be changed by specifying a different complex_mode.

Parameters:

  • x (Union[Array, NativeArray]) – input array

  • complex_mode (Literal['split', 'magnitude', 'jax'], default: 'jax') – optional specifier for how to handle complex data types. See ivy.func_wrapper.handle_complex_input for more detail.

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

ret – an array containing the rectified linear unit activation of each element in x.

Examples

With ivy.Array input:

>>> x = ivy.array([-1., 0., 1.])
>>> y = ivy.relu(x)
>>> print(y)
ivy.array([0., 0., 1.])

>>> x = ivy.array([1.5, 0.7, -2.4])
>>> y = ivy.zeros(3)
>>> ivy.relu(x, out = y)
>>> print(y)
ivy.array([1.5, 0.7, 0.])

With ivy.Container input:

>>> x = ivy.Container(a=ivy.array([1.0, -1.2]), b=ivy.array([0.4, -0.2]))
>>> x = ivy.relu(x, out=x)
>>> print(x)
{
    a: ivy.array([1., 0.]),
    b: ivy.array([0.40000001, 0.])
}
ivy.sigmoid(x, /, *, complex_mode='jax', out=None)[source]#

Apply the sigmoid function element-wise.

Parameters:

  • x (Union[Array, NativeArray]) – input array.

  • complex_mode (Literal['split', 'magnitude', 'jax'], default: 'jax') – optional specifier for how to handle complex data types. See ivy.func_wrapper.handle_complex_input for more detail.

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the input broadcast to. default: None

Return type:

Array

Returns:

ret – an array containing the sigmoid activation of each element in x. sigmoid activation of x is defined as 1/(1+exp(-x)).

Examples

With ivy.Array input:

>>> x = ivy.array([-1.0, 1.0, 2.0])
>>> y = ivy.sigmoid(x)
>>> print(y)
ivy.array([0.2689414 , 0.7310586 , 0.88079703])

>>> x = ivy.array([-1.0, 1.0, 2.0])
>>> y = ivy.zeros(3)
>>> ivy.sigmoid(x, out=y)
>>> print(y)
ivy.array([0.2689414 , 0.7310586 , 0.88079703])

With ivy.Container input:

>>> x = ivy.Container(a=ivy.array([0.]),
...                   b=ivy.Container(c=ivy.array([1.]),
...                                   d=ivy.array([2.])))
>>> y = ivy.sigmoid(x)
>>> print(y)
{
    a: ivy.array([0.5]),
    b: {
        c: ivy.array([0.7310586]),
        d: ivy.array([0.88079703])
    }
}

>>> x = ivy.Container(a=ivy.array([0.]),
...                   b=ivy.Container(c=ivy.array([1.]),
...                                   d=ivy.array([2.])))
>>> y = ivy.Container(a=ivy.array([0.]),
...                   b=ivy.Container(c=ivy.array([0.]),
...                                   d=ivy.array([0.])))
>>> ivy.sigmoid(x, out=y)
>>> print(y)
{
    a: ivy.array([0.5]),
    b: {
        c: ivy.array([0.7310586]),
        d: ivy.array([0.88079703])
    }
}
ivy.softmax(x, /, *, axis=None, complex_mode='jax', out=None)[source]#

Apply the softmax function element-wise.

Parameters:

  • x (Union[Array, NativeArray]) – Input array.

  • axis (Optional[int], default: None) – The dimension softmax would be performed on. The default is None.

  • complex_mode (Literal['split', 'magnitude', 'jax'], default: 'jax') – optional specifier for how to handle complex data types. See ivy.func_wrapper.handle_complex_input for more detail.

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

ret – The input array with softmax applied element-wise.

Examples

With ivy.Array input:

>>> x = ivy.array([1.0, 0, 1.0])
>>> y = ivy.softmax(x)
>>> print(y)
ivy.array([0.422, 0.155, 0.422])

>>> x = ivy.array([[1.1, 2.2, 3.3],
...                [4.4, 5.5, 6.6]])
>>> y = ivy.softmax(x, axis = 1)
>>> print(y)
ivy.array([[0.0768, 0.231 , 0.693 ],
           [0.0768, 0.231 , 0.693 ]])
ivy.softplus(x, /, *, beta=None, threshold=None, complex_mode='jax', out=None)[source]#

Apply the softplus function element-wise.

If the input is complex, then by default we apply the softplus operation log(1+ exp(x)) to each element If threshold is set we check if either its real part is strictly negative or if its real part is zero and its imaginary part is negative then we apply input×β > threshold.

Parameters:

  • x (Union[Array, NativeArray]) – input array.

  • beta (Optional[Union[int, float]], default: None) – The beta value for the softplus formation. Default: None.

  • threshold (Optional[Union[int, float]], default: None) – values above this revert to a linear function If the input is complex, only its real part is considered. Default: None

  • complex_mode (Literal['split', 'magnitude', 'jax'], default: 'jax') – optional specifier for how to handle complex data types. See ivy.func_wrapper.handle_complex_input for more detail.

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

ret – an array containing the softplus activation of each element in x.

Examples

With ivy.Array input:

>>> x = ivy.array([-0.3461, -0.6491])
>>> y = ivy.softplus(x)
>>> print(y)
ivy.array([0.535,0.42])

>>> x = ivy.array([-0.3461, -0.6491])
>>> y = ivy.softplus(x, beta=0.5)
>>> print(y)
ivy.array([1.22, 1.09])

>>> x = ivy.array([1., 2., 3.])
>>> y = ivy.softplus(x, threshold=2)
>>> print(y)
ivy.array([1.31, 2.13, 3.  ])
ivy.softsign(x, /, out=None)[source]#

Apply the softsign function element-wise.

Parameters:

  • x (Union[Array, NativeArray]) – Input array.

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

ret – The input array with softsign applied element-wise.

Examples

With ivy.Array input: >>> x = ivy.array([1.0, 2.0, 3.0]) >>> y = ivy.softsign(x) >>> print(y) ivy.array([0.5, 0.66666667, 0.75])

This should have hopefully given you an overview of the activations submodule, if you have any questions, please feel free to reach out on our discord in the activations channel!