Navigating the Code

Categorization

Ivy uses the following categories taken from the Array API Standard:

• constants

• creation

• data_type

• elementwise

• linear_algebra

• manipulation

• searching

• set

• sorting

• statistical

• utility

In addition to these, we also add the following categories, used for additional functions in Ivy that are not in the Array API Standard:

• activations

• compilation

• device

• general

• gradients

• layers

• losses

• meta

• nest

• norms

• random

Some functions that you’re considering adding might overlap several of these categorizations, and in such cases you should look at the other functions included in each file, and use your best judgement for which categorization is most suitable.

We can always suggest a more suitable location when reviewing your pull request if needed 🙂

Submodule Design

Ivy is designed so that all methods are called directly from the ivy namespace, such as ivy.matmul(), and not ivy.some_namespace.matmul(). Therefore, inside any of the folders ivy.functional.ivy, ivy.functional.backends.some_backend, ivy.functional.backends.another_backend the functions can be moved to different files or folders without breaking anything at all. This makes it very simple to refactor and re-organize parts of the code structure in an ongoing manner.

The __init__.py inside each of the subfolders are very similar, importing each function via from .file_name import * and also importing each file as a submodule via from . import file_name. For example, an extract from ivy/ivy/functional/ivy/__init__.py is given below:

from . import elementwise
from .elementwise import *
from . import general
from .general import *
# etc.

Ivy API

All function signatures for the Ivy API are defined in the ivy.functional.ivy submodule. Functions written here look something like the following, (explained in much more detail in the following sections):

def my_func(
    x: Union[ivy.Array, ivy.NativeArray],
    /,
    axes: Union[int, Sequence[int]],
    *,
    dtype: Optional[Union[ivy.Dtype, ivy.NativeDtype]] = None,
    device: Optional[Union[ivy.Device, ivy.NativeDevice]] = None,
    out: Optional[ivy.Array] = None
) -> ivy.Array:
    """
    Explanation of the function.

    .. note::
        This is an important note.

    **Special Cases**

    For this particular case,

    - If ``x`` is ``NaN``, do something
    - If ``y`` is ``-0``, do something else
    - etc.

    Parameters
    ----------
    x
        input array. Should have a numeric data type.
    axes
        the axes along which to perform the op.
    dtype
        array data type.
    device
        the device on which to place the new array.
    out
        optional output array, for writing the result to. It must have a shape that the
        inputs broadcast to.

    Returns
    -------
    ret
        an array.

    Examples
    --------

    Some examples go here
    """
    return ivy.current_backend(x).my_func(x, axes, dtype=dtype, device=device, out=out)

We follow the Array API Standard convention about positional and keyword arguments.

• Positional parameters must be positional-only parameters. Positional-only parameters have no externally-usable name. When a method accepting positional-only parameters is called, positional arguments are mapped to these parameters based solely on their order.

• Optional parameters must be keyword-only arguments.

This convention makes it easier for us to modify functions in the future. Keyword-only parameters will mandate the use of argument names when calling functions, and this will increase our flexibility for extending function behaviour in future releases without breaking forward compatibility. Similar arguments can be kept together in the argument list, rather than us needing to add these at the very end to ensure positional argument behaviour remains the same.

The dtype, device and out arguments are always keyword-only. Arrays always have a type hint Union[ivy.Array, ivy.NativeArray] in the input and ivy.Array in the output. All functions which produce a single array include the out argument. The reasons for each of these features are explained in the following sections.

Backend API

Code in the backend submodules such as ivy.functional.backends.torch should then look something like:

def my_func(
    x: torch.Tensor,
    /,
    axes: Union[int, Sequence[int]],
    *,
    dtype: torch.dtype,
    device: torch.device,
    out: Optional[torch.Tensor] = None
) -> torch.Tensor:
    return torch.function_name(x, axes, dtype, device, out)

The dtype, device and out arguments are again all keyword-only, but dtype and device are now required arguments, rather than optional as they were in the Ivy API. All arrays also now have the same type hint torch.Tensor, rather than Union[ivy.Array, ivy.NativeArray] in the input and ivy.Array in the output. The backend methods also should not add a docstring. Again, the reasons for these features are explained in the following sections.

Submodule Helper Functions

At times, helper functions specific to the submodule are required to:

• keep the code clean and readable

• be imported in their respective backend implementations

To have a better idea on this, let’s look at an example!

Helper in Ivy

# in ivy/utils/assertions.py
def check_fill_value_and_dtype_are_compatible(fill_value, dtype):
    if (
        not (
            (ivy.is_int_dtype(dtype) or ivy.is_uint_dtype(dtype))
            and isinstance(fill_value, int)
        )
        and not (
            ivy.is_complex_dtype(dtype) and isinstance(fill_value, (float, complex))
        )
        and not (
            ivy.is_float_dtype(dtype)
            and isinstance(fill_value, (float, np.float32))
            or isinstance(fill_value, bool)
        )
    ):
        raise ivy.utils.exceptions.IvyException(
            f"the fill_value: {fill_value} and data type: {dtype} are not compatible"
        )

In the full_like() function in creation.py, the types of fill_value and dtype has to be verified to avoid errors. This check has to be applied to all backends, which means the related code is common and identical. In this case, we can extract the code to be a helper function on its own, placed in its related submodule (creation.py here). In this example, the helper function is named as _assert_fill_value_and_dtype_are_compatible().

Then, we import this submodule-specific helper function to the respective backends, where examples for each backend is shown below.

Jax

# in ivy/functional/backends/jax/creation.py

def full_like(
    x: JaxArray,
    /,
    fill_value: Number,
    *,
    dtype: jnp.dtype,
    device: jaxlib.xla_extension.Device,
    out: Optional[JaxArray] = None,
) -> JaxArray:
    ivy.utils.assertions.check_fill_value_and_dtype_are_compatible(fill_value, dtype)
    return _to_device(
        jnp.full_like(x, fill_value, dtype=dtype),
        device=device,
    )

NumPy

# in ivy/functional/backends/numpy/creation.py

def full_like(
    x: np.ndarray,
    /,
    fill_value: Number,
    *,
    dtype: np.dtype,
    device: str,
    out: Optional[np.ndarray] = None,
) -> np.ndarray:
    ivy.utils.assertions.check_fill_value_and_dtype_are_compatible(fill_value, dtype)
    return _to_device(np.full_like(x, fill_value, dtype=dtype), device=device)

TensorFlow

# in ivy/functional/backends/tensorflow/creation.py

def full_like(
    x: Union[tf.Tensor, tf.Variable],
    /,
    fill_value: Number,
    *,
    dtype: tf.DType,
    device: str,
    out: Optional[Union[tf.Tensor, tf.Variable]] = None,
) -> Union[tf.Tensor, tf.Variable]:
    ivy.utils.assertions.check_fill_value_and_dtype_are_compatible(fill_value, dtype)
    with tf.device(device):
        return tf.experimental.numpy.full_like(x, fill_value, dtype=dtype)

Note

We shouldn’t be enabling numpy behaviour in tensorflow as it leads to issues with the bfloat16 datatype in tensorflow implementations

Torch

# in ivy/functional/backends/torch/creation.py

def full_like(
    x: torch.Tensor,
    /,
    fill_value: Number,
    *,
    dtype: torch.dtype,
    device: torch.device,
    out: Optional[torch.Tensor] = None,
) -> torch.Tensor:
    ivy.utils.assertions.check_fill_value_and_dtype_are_compatible(fill_value, dtype)
    return torch.full_like(x, fill_value, dtype=dtype, device=device)

Version Unpinning

At any point in time, Ivy’s development will be predominantly focused around the latest pypi version (and all prior versions) for each of the backend frameworks.

Earlier we had our versions pinned for each framework to provide stability but later concluded that by unpinnning the versions we would be able to account for the latest breaking changes if any and support the latest version of the framework. Any prior version’s compatibility would be tested by our multiversion testing pipeline, thus keeping us ahead and in light of the latest changes.

This helps to prevent our work from culminating over a fixed version while strides are being made in the said frameworks. Multiversion testing ensures the backward compatibility of the code while this approach ensures we support the latest changes too.

Round Up

This should have hopefully given you a good feel for how to navigate the Ivy codebase.

If you have any questions, please feel free to reach out on discord in the navigating the code channel!

Video