MLC LLM, which stands for Machine Learning Compilation for Large Language Models, is a framework aimed at high-performance and universal deployment of large language models. It leverages compiler acceleration to enable the native deployment of these models with native APIs across various devices. The framework is designed to facilitate the development, optimization, and deployment of AI models using machine learning compilation techniques. It offers capabilities to build and deploy applications across different platforms, including web, iOS, and Android, with support for multiple programming languages and APIs. Additionally, MLC-LLM provides tools for model compilation, distribution of compiled models, and model architecture definition.

GGML is a minimal, C-based tensor library focussed on providing efficient training and inference for popular models on commodity hardware, specifically Apple silicon. It has a number of features that make it ideal for edge deployment such as integer quantization ranging from 8-bit quantization all the way to 4-bit, no third-party dependencies, zero memory allocations at runtime, and support for WebAssembly and WASM SIMD for running in the browser.

NVIDIA's Apex is a set of tools designed to simplify and optimize mixed precision and distributed training in PyTorch. It was created to provide utilities like Automatic Mixed Precision (AMP) and Distributed Training functionalities to users more rapidly than the official PyTorch releases. Although parts of Apex, such as apex.amp for AMP and apex.parallel.DistributedDataParallel for optimized multi-process distributed training, have been deprecated in favor of PyTorch's native features, the library offered functionalities like synchronized Batch Normalization and a custom checkpointing system for training states. Apex has been instrumental in improving performance and memory usage for deep learning models on NVIDIA GPUs

PlaidML is a tensor compiler, developed by Intel, specializing in running deep learning models on devices which are not typically well supported for such workloads, such as laptops and embedded devices. It is even able to generate Nvidia GPU code matching the performance of native libraries without having to go through CUDA. Recently it has adopted MLIR for the purpose of better extensibility and integration into the ever-growing MLIR ecosystem. PlaidML is also part of Intel's wider suite of deep learning tools, nGraph.

Intel® Extension for PyTorch (IPEX) is a powerful tool designed to optimize PyTorch performance specifically for Intel architecture, taking advantage of specialized instruction sets and acceleration units on Intel CPUs and GPUs. It enhances both eager and graph execution modes in PyTorch with custom modules, optimizers, and quantization APIs, while providing graph mode optimizations through extended fusion passes.

XLA, which stands for Accelerated Linear Algebra, is a domain-specific compiler, developed by Google, designed to optimize and accelerate the execution of the core linear algebra operations commonly used in machine learning and other numerical computing tasks. XLA is often associated with the TensorFlow, where it is used to improve the performance of TensorFlow models, and more recently, combined with an autograd engine, forms the backbone of JAX. The primary goal of XLA is to take high-level mathematical operations defined in the frontend framework and compile them into optimized low-level code that can be executed on various hardware accelerators, such as GPUs and TPU, which XLA was specifically designed to support. Its pipeline consists of decomposing the input program into a fixed set of high level operations (HLO), then performing a series of target-independent optimization passes, such as weight update sharding or kernel fusion, on this HLO IR, before offloading to device specific libraries like CuDNN for Nvidia GPUs or Eigen for CPUs, for handling hardware-dependent optimizations. Though originally built with a unique IR syntax, over the years XLA has gradually integrated MLIR into its compilation pipeline, providing the ability to both import and export MLIR modules from its corresponding HLO dialects (MHLO, LHLO) as well as adding optimization passes written fully in MLIR. In 2022, Google fully open sourced XLA, moving it to a new repository independent of Tensorflow, and created the OpenXLA foundation to continue its support and broaden the ecosystem with the development of new tools such as StableHLO, a backwards-compatible MLIR dialect for representing HLO ops, and IREE.

Tiramisu is a polyhedral compiler designed to handle both dense and sparse deep learning and other data parallel algorithms. Its defining features include its flexible representation based on the polyhedral model, a rich scheduling language for precise control of optimizations, and a four-level intermediate representation that separates algorithms, loop transformations, data layouts, and communication. Tiramisu’s polyhedral model allows it to stand out particularly in its optimization of RNNs and sparse nets, which are areas not typically as thoroughly addressed by other deep learning compilers.

Hidet is an open-source deep learning compiler, primarily written in Python, designed for end-to-end compilation of deep neural network (DNN) models from frameworks like PyTorch and ONNX into efficient CUDA kernels, specifically optimized for NVIDIA GPUs. It focuses on providing high performance through graph-level optimizations and operator-level kernel tuning. Hidet is characterized by its ease of use, allowing for easy integration of DNN models for inference, and its extensibility, enabling users to add new operators and fusion patterns to suit their specific needs.

tinygrad is a complete, mini deep learning framework, with syntax very similar to that of PyTorch, however, unlike Torch, it is by default lazily compiled and is built on top of a minimalist op set, claiming: “Due to its extreme simplicity, it aims to be the easiest framework to add new accelerators to, with support for both inference and training. If XLA is CISC, tinygrad is RISC.” Though supporting a wide range of hardware, its parent company, TinyCorp, is especially focusing on providing top tier support for AMD GPUs, which it plans on using as the base of its local deep learning compute cluster, tinybox.

MLGO is a framework designed to optimize compiler processes within LLVM using machine learning techniques, specifically Policy Gradient and Evolution Strategies. It is distinctive for embedding machine learning models directly into the compiler to make key decisions during code compilation, educing the overhead typically associated with such runtime decisions without relying on heuristics. Its design is not only targeted but also expandable, allowing for future enhancements and broader applications in compiler optimization. MLGO exemplifies the innovative intersection of machine learning and compiler technology.

SHARK is a high performance ML compiler and runtime, developed by Nod.ai, a performance focused AI startup recently acquired by AMD, specifically for accelerating PyTorch code. It is built using components from IREE and Torch-MLIR, an MLIR dialect for representing PyTorch operations, as well as an in-house, ML-based autotuner. SHARK has shown a number of promising features and results, including TorchDynamo integration, a hardware-agnostic implementation of the seminal Flash-Attention algorithm, and even exceeding the performance of OpenAI Triton while running Flash-Attention 2 on an A100 GPU.

FlexGen is a recently developed inference engine for LLMs designed to maximize throughput on a single commodity GPU. It uses a linear programming optimizer to navigate through a search space of offloading strategies in search of the most efficient ways to store and load tensors, as well as fine-grained, group-wise quantization to compress the weights and attention cache to 4 bits with negligible accuracy loss. One of FlexGen’s key contributions is the use of a “zig-zag” as opposed to row-by-row block schedule to more efficiently overlap I/O with computation. Through a combination of such strategies, FlexGen achieves significantly higher throughput than other state-of-the-art offloading methods.

TensorRT-LLM is an advanced library from NVIDIA designed to accelerate the inference process for large language models NVIDIA GPUs. It offers a streamlined, high-performance solution for developers, encapsulating state-of-the-art optimizations within an accessible Python API. This tool simplifies the complexity traditionally associated with deploying LLMs by providing pre-optimized computational kernels and a suite of tools for performance enhancement, including a backend for Triton server integration. It comes with several popular models prebuilt and includes a wide range of configurations for both single-GPU and multi-GPU settings.

Triton is an open-source programming language and compiler designed for the purpose of expressing and compiling tiled computations within neural networks into highly efficient machine code. Triton demonstrates how just a few control-flow and data handling extensions to LLVM-IR could enable various optimization routes for a neural network. Designed to provide the same kernel-level control as CUDA, Triton introduces a new, block-based rather than thread-based programming model for GPUs, offering a higher, more user-friendly abstraction. The compilation pipeline consists of writing a GPU kernel in a simple, Python or C-like format, which is then lowered into the originally LLVM-based, now MLIR-based, Triton-IR, Triton-JIT then performs a series of platform-independent, followed by platform-specific, tile-level optimization passes before generating the final LLVM/PTX code. Triton's auto-generated kernels have been shown to match or even outperform the performance of handwritten cuDNN and cuBLAS ones, however currently Triton only supports Nvidia GPUs. It has recently become the main building block behind the most efficient Torch backend for GPUs, called TorchInductor, showcasing Triton’s utility.

MLIR, which stands for Multi-Level Intermediate Representation, is a compiler infrastructure, originally developed at Google but now maintained by the LLVM foundation, which is designed to be a flexible and extensible framework for representing and optimizing programs and data structures across various stages of compilation and execution, particularly in the context of high-performance computing, machine learning, and domain-specific languages. One of the core contributions of its model is a unified IR syntax which can support this wide range of representations, from high level tensor ops with value semantics all the way down to assembly instructions, it does this through the usage of dialects, which, as the name suggests, are essentially sub-languages for these different levels of abstraction. Some examples of dialects include the affine dialect for polyhedral transformations, the arith dialect for elementary arithmetic operations, and the scf dialect for structured control flow. On the hardware-specific side, there are unique dialects for different architectures like x86 and Arm, as well as for standardized device representations such as SPIR-V for GPUs. Though initially born out of the XLA project with deep learning applications in mind, MLIR is in fact a general purpose compiler infrastructure which can be used for a wide range of domain specific tasks such as quantum computing and even circuit design.

Though yet to have been fully released, PolyBlocks is a promising MLIR-based compiler engine which utilizes a modular, block-like structure of compiler passes, making it highly extensible and reusable for a variety of hardware architectures. As the name suggests, it specializes in the use of polyhedral optimization techniques, utilizing MLIR’s affine dialect, as well as leveraging high-dimensional data spaces for compilation, supporting both Just-in-Time (JIT) and Ahead-of-Time (AOT) compilation and performing full codegen, making it fully independent of any third-party vendor libraries. Its benchmarks have shown performance which exceeds that of hand-written Nvidia libraries such as CuBLAS, CuDNN and TensorRT on A100 GPUs.

ExecuTorch: ExecuTorch is an end-to-end solution that enables on-device inference capabilities across mobile and edge devices. At a high level, it is a more robust version of PyTorch Mobile, which used TorchScript for deployment on edge devices. However, with PyTorch 2.0, it is no longer preferred to use TorchScript, and ExecuTorch replaces it with torch.compile and torch.export. One of the major benefits of ExecuTorch is that it defines a list of standardized operators, which reduces the surface that needs to be covered by third-party operator libraries and accelerator backends. ExecuTorch has also received backing from industry leaders like Meta, Arm, Apple, and Qualcomm.

IREE, which stands for Intermediate Representation Execution Environment, is an MLIR based compiler and runtime stack specifically designed to be able to scale down to meet the tight constraints of mobile and edge deployment, while still retaining the ability to scale up to more traditional, large scale settings like datacenters and compute clusters. IREE provides an end-to-end compilation pipeline, ranging from high level optimizations similar to XLA, all the way down to a Hardware Abstraction Layer (HAL) which generates device specific code. IREE has taken much inspiration from, and integrates seamlessly with Vulkan, a popular cross-platform GPU API which uses SPIR-V, a standardized binary intermediate language for GPUs. One of IREE’s greatest strengths is the high degree of modularity that it provides due to its utilization of a different MLIR dialect for each layer of abstraction, from tensor-based input dialects such as StableHLO, TOSA and Torch-MLIR, to internal dialects handling low-level tasks such as dataflow analysis, partitioning and scheduling. IREE stands as being the most robust end-to-end compiler/runtime stack built fully around MLIR, providing a concrete example of its utility.

KeOps (Kernel Operations) is a library designed to boost tensor computations by leveraging mathematical structure common in machine learning and applied mathematics, like distance matrices, kernel matrices, and point cloud convolutions. It utilizes symbolic tensors, which are efficient mathematical formulations, supporting operations with CPU and GPU integration. KeOps is memory-efficient, offering significant speed-ups to GPU programs, supports automatic differentiation, and has a linear memory footprint. It integrates well with standard packages and supports multi-GPU configurations.

TACO, which stands for Tensor Algebra Compiler, is a library and compiler, specializing in its support for a wide range of tensor computations and storage formats. It automates the generation of efficient kernels which can compete with hand-written ones, and, uniquely at the time of its inception, its operands may be either fully dense, fully sparse, or a mix of the two.

TVM, which stands for Tensor Virtual Machine, is an end-to-end compiler stack for deep learning, aiming to provide a comprehensive set of optimizations for a wide variety of hardware backends, including custom accelerators. It uses the same scheduling language model as that which was first introduced by Halide, allowing an algorithm to be defined separately from its scheduling logic. Unlike most DL compilers, which tend to be graph-based, TVM’s Relay IR is a functional language with full support for common programming constructs such as conditionals and loops, giving it better support for more complex, dynamic models. TVM generates its own low-level, hardware specific code for a diverse set of backends and utilizes an ML-based cost model called AutoTVM which adapts and improves codegen based on continuous data collection. Recently it has been added as a backend of torch.compile.

PI is a lightweight Python frontend for MLIR, specializing in converting PyTorch modules into their corresponding Torch-MLIR format, without any having any dependencies on PyTorch itself.

Torch-MLIR is a project that seeks to bridge PyTorch with the MLIR (Multi-Level Intermediate Representation) ecosystem. As part of the LLVM Incubator process, Torch-MLIR is not yet an official part of LLVM releases but serves as a compiler support path from PyTorch to MLIR. Torch-MLIR enables hardware vendors to efficiently map PyTorch and other frameworks into MLIR, allowing for easier targeting of hardware-specific optimizations. The project supports several paths for lowering models to the Torch MLIR Dialect and then to existing MLIR dialects. On the PyTorch frontend, this includes TorchScript and LazyTensorCore, with the addition of support for TorchDynamo/PyTorch 2.0 in progress. On the backend, a variety of different lowering paths are supported for different use cases, the core ones being StableHLO, TOSA and Linalg (+ arith, tensor, etc).

vLLM is a recently released LLM inference engine which utilizes a new algorithm called PagedAttention. Based on virtual memory and paging techniques long used in operating systems, this algorithm seeks to minimize the unnecessary growth of the key-value cache in LLMs, a common problem faced in production. By storing contiguous keys and values in a non-contiguous, block-structured memory space, PagedAttention reduces the wasted memory from the usual 60-80% to less than 4%, achieving near optimal utilization. While also attempting to achieve high throughput, vLLM differs from FlexGen in that it is a serving engine more focused on distributed settings rather than optimizing for a single device.

Kernl is an open source, PyTorch-based inference engine built using OpenAI Triton, CUDA graphs and TorchDynamo. Kernl aims to minimize the GPU memory bandwidth bottleneck with kernel fusion while also making sure that such kernels are easy to understand and customize, being each under 200 lines of code.

OneAPI is Intel's comprehensive toolkit for developing and optimizing applications across diverse computing architectures such as CPUs, GPUs, and FPGAs. It's built around a unified programming model that aims to future-proof software development by allowing a single codebase to span multiple types of hardware. The toolkit includes advanced compilers, powerful libraries, and migration tools that facilitate the transition from proprietary code (like CUDA) to an open, standards-based workflow. OneAPI stands out for its emphasis on ease of use, performance optimization, and its broad support for heterogeneous computing, making it a versatile choice for developers targeting Intel's hardware ecosystems.

Mojo is not just a compiler, but rather an entirely new programming language recently introduced by Chris Latner’s (of LLVM fame) company, Modular. Offering a syntax nearly identical to that of Python, Mojo seeks to combine Python’s ease of use with the low-level control and runtime efficiency which languages like C and C++ provide. Being a superset of Python, Mojo provides full interoperability with existing packages in the Python ecosystem such as pandas and matplotlib. Mojo also introduces a slew of new features which Python lacks, including type checking, ownership, SIMD (Single Instruction Multiple Data) programming and even autotuning features to finetune parameters for optimal performance on one’s target hardware. Built on top of MLIR, Mojo also allows full extensibility and interoperability with any existing tools in the MLIR ecosystem, and can even be thought of as a programming language for MLIR, as it provides the ability to write custom MLIR code in a more user-friendly, Pythonic syntax. Along with this new language, Modular also provides an AI inference engine (written using Mojo kernels) which existing models written in PyTorch or Tensorflow can be easily ported to for efficient inference. Currently Mojo supports all CPU architectures and is planning on supporting GPUs and custom accelerators in the near future.

Glow, which is an abbreviation for Graph-Lowering, consists of a strongly typed, two-phase Intermediate Representation, splitting optimizations up between the high level and low level respectively. Glow’s compilation pipeline seeks to solve the issue of targeting a large number of opcodes for different hardware backends by decomposing all higher level operations into a finite set of simpler, more universally supported ones, hence eliminating the need to implement any new layer or activation as a custom op for every hardware backend. Like TVM and IREE, Glow generates its own machine-specific code rather than offloading to third-party compute primitive libraries. The low level side of its IR uses an instruction-based, address-only format, allowing it to excel at memory-related optimizations such as memory-allocation and instruction scheduling, however, due to the strong typing, Glow does not support dynamic shapes such as some of the other compilers (XLA, TVM, IREE) do. Despite being developed by Meta directly under the PyTorch umbrella, direct support for Glow has yet to have been added to the PyTorch API, as any PyTorch code must first be converted to ONNX before being passed in.

Candle is a Rust-based, minimalist machine learning framework maintained by HuggingFace and designed to offer efficient serverless inference. Though recently developed, Candle already offers excellent support for efficient performance on both GPU and CPU, leveraging compute primitive libraries such as cuDNN for Nvidia GPUs and Intel MKL for CPUs, as well as providing the ability to easily add custom kernels for state-of-the-art algorithms like Flash-Attention. By removing all Python dependencies while retaining an easy to use, torch-like syntax, Candle can greatly improve the efficiency of the production workload and allow for the deployment of lightweight binaries. In addition to this, it also supports WASM, allowing models to be run fully in the browser.

TensorRT is a high-performance deep learning inference library developed by NVIDIA. It is designed to optimize and accelerate the inference of deep neural networks for production deployment on NVIDIA GPUs. TensorRT is commonly used in applications that require real-time, low-latency, and high-throughput processing of deep learning models. Given the DAG of the model as an input, TensorRT creates its own DAG. Then it applies optimizations like operation fusion in kernel level and memory rearrangements to speed up the inference. It also eliminates redundant operations that either don’t contribute to the output or when removed they don’t change the final result. For example, if there are two successive reshape operations that cancel each other, both will be dropped during the optimization process. It supports dynamic shapes for inputs when models are simple like CNNs or MLPs. Internally it uses pre-compiled CUDA kernels from linear algebra and deep learning libraries like cuBLAS and cuDNN.

BladeDISC, developed by Alibaba, is an open-sourced dynamic shape compiler designed to overcome the challenges associated with the static shape limitations of previous deep learning compilers. Leveraging MLIR, BladeDISC allows for dynamic shape semantics, providing a flexible and robust solution for AI computing tasks that require diverse input shapes and sizes.

SparseML is an open source library developed and maintained by [Neural Magic](https://neuralmagic.com/) for applying compression recipes to neural networks. Currently, it supports pruning, quantization and knowledge distillation for compressing Vision, NLP, and now, large language models as well. SparseML also provides pre-compressed and pre-quantized models in their SparseZoo.

Pruna AI makes, in one line of code, any AI model faster, cheaper, smaller, greener on any hardware. It covers CV, NLP, audio, graphs for predictive and generative AI

The bitsandbytes is a lightweight wrapper around CUDA custom functions, in particular 8-bit optimizers, matrix multiplication (LLM.int8()), and quantization functions.

Efficient and accurate low-bit weight quantization (INT3/4) for LLMs, supporting instruction-tuned models and multi-modal LMs.

TensorLy-Torch is a Python library for deep tensor networks that builds on top of TensorLy and PyTorch. It allows to easily leverage tensor methods in a deep learning setting and comes with all batteries included.

SONY’s Model Compression Toolkit is a model compression tool that focuses on quantization and comes with a suite of features that make it easier to optimise neural networks for efficient deployment, including synthetic image data generation and visualisation tools.

NNI automates feature engineering, neural architecture search, hyperparameter tuning, and model compression for deep learning.

Torch-Pruning(TP) is a library for sturtured pruning that works with a wide range of deep neural networks including LLMs, Diffusion Models, YOLOv7, Yolov8, Vision Transformers, Swin Transformers, BERT, FasterRCNN, SSD, ResNe(X)t, ConvNext, DenseNet, ConvNext, RegNet, DeepLab, etc. Torch-Pruning deploys an algorithm called DepGraph to remove parameters physically

Pytorch has built in support for quantization as well as pruning under it's model optmization module

TensorLy is a Python library that aims at making tensor learning simple and accessible. It allows simple performing of tensor decomposition, tensor learning and tensor algebra. Its backend system allows to seamlessly perform computation with NumPy, PyTorch, JAX, MXNet, TensorFlow or CuPy, and run methods at scale on CPU or GPU.

AIMET is a library that provides advanced model quantization and compression techniques for trained neural network models. It provides features that have been proven to improve run-time performance of deep learning neural network models with lower compute and memory requirements and minimal impact to task accuracy.

SqueezeLLM is a post-training quantization framework that incorporates a new method called Dense-and-Sparse Quantization to enable efficient LLM serving."

8 bit Weight and Activation Quantization for LLMs

🤗 Optimum is an extension of 🤗 Transformers and Diffusers, providing a set of optimization tools enabling maximum efficiency to train and run models on targeted hardware, while keeping things easy to use.

GPTQ is a technique for post-training quantization, which is used to quantize large language models (LLMs) such as GPT. This method minimizes the storage requirements for GPT models by decreasing the bit count necessary to represent each weight within the model, reducing it from 32 bits to a mere 3-4 bits or even 2 bits.

Intel® Neural Compressor aims to provide popular model compression techniques such as quantization, pruning (sparsity), distillation, and neural architecture search on mainstream frameworks such as TensorFlow, PyTorch, ONNX Runtime, and MXNet, as well as Intel extensions such as Intel Extension for TensorFlow and Intel Extension for PyTorch.

BigDL is a set of toolkits developed by Intel to seamlessly scales data analytics & AI applications to be run easily on both laptops to cloud infrastructures. The project introduces the following set of libraries: - [LLM](https://github.com/intel-analytics/BigDL/tree/main/python/llm): Low-bit (INT3/INT4/INT5/INT8) large language model library for Intel CPU/GPU - [Orca](https://bigdl.readthedocs.io/en/latest/doc/Orca/index.html): Distributed Big Data & AI (TF & PyTorch) Pipeline on Spark and Ray - [Nano](https://bigdl.readthedocs.io/en/latest/doc/Nano/index.html): Transparent Acceleration of Tensorflow & PyTorch Programs on Intel CPU/GPU - [DLlib](https://bigdl.readthedocs.io/en/latest/doc/DLlib/index.html): “Equivalent of Spark MLlib” for Deep Learning - [Chronos](https://bigdl.readthedocs.io/en/latest/doc/Chronos/index.html): Scalable Time Series Analysis using AutoML - [Friesian](https://bigdl.readthedocs.io/en/latest/doc/Friesian/index.html): End-to-End Recommendation Systems - [PPML](https://bigdl.readthedocs.io/en/latest/doc/PPML/index.html): Secure Big Data and AI (with SGX Hardware Security)"

Xilinx which was acquired by AMD and is a company that specializes in the design and development of field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), and adaptive SoC architecture. One of the most popular hardware they’ve manufactured is the VCK5000 which is built on the AMD 7nm Versal adaptive system-on-chip architecture. The Versal adaptive SOC is a disruptive 7 nm architecture that combines heterogeneous compute engines with a breadth of hardened memory and interfacing technologies for superior performance/watt over competing 10 nm FPGAs . In this context VCK5000 is ideal for CNN, RNN, and NLP acceleration for your cloud and edge applications.

Ampere is a pioneering semiconductor design firm poised for the next generation, shaping the future of computing through a groundbreaking strategy in CPU design that prioritizes high performance, energy efficiency, and sustainable cloud computing.

Intel Movidius VPUs empower resource-intensive computer vision and AI tasks with remarkable efficiency. Through the integration of highly parallel programmable computing and dedicated AI hardware acceleration in a distinctive architecture that minimizes data movement, Movidius VPUs strike a harmonious balance between power efficiency and computational performance.

Sima is building an ultra low-power software and chip solution for machine learning at the edge.

NorthPole signifies a pioneering advancement in chip architecture, delivering substantial enhancements in energy, space, and time efficiency. Employing the ResNet-50 model as a benchmark, NorthPole showcases significantly superior efficiency compared to typical 12-nm GPUs and 14-nm CPUs. Notably, NorthPole itself is constructed using 12 nm node processing technology. In both cases, NorthPole exhibits a remarkable 25-fold improvement in energy efficiency, measured in terms of frames processed per joule of power consumed. Furthermore, NorthPole excels in terms of latency and computational footprint, specifically in frames processed per second per billion transistors utilized. It is important to highlight that, on the ResNet-50 model, NorthPole outperforms all major contemporary architectures, even those utilizing more advanced technology processes, such as a GPU built on a 4 nm process.

Hailo is a hardware vendor that offers SOTA AI accelerators and vision processors uniquely designed to accelerate embedded deep learning applications on edge devices. Hailo employs a revolutionary architecture, representing a clean-slate approach to the design of a specialized technology stack. The company has created a domain-specific processor that significantly outperforms the Von Neumann architecture for deep learning tasks.

Tenstorrent is an innovative hardware manufacturer focused on developing processors featuring a unique grid architecture called Tensix cores. These Tensix cores consist of a sophisticated combination of components, including five RISC processors, an array math unit dedicated to tensor operations, a SIMD (Single Input Multiple Data) unit for efficient vector operations, specialized hardware for accelerating network packet processing and compression/decompression tasks, and a generous cache of 1-2 megabytes of SRAM memory. The unique thing about Tenstorrent boards is that they can be used similar to a GPU and can be inserted into any Gen4 x16 PCIe slot on the motherboard.

Corsair hardware involves a groundbreaking technique that shifts computational tasks into Static Random Access Memory (SRAM), fundamentally transforming the interaction between memory and computing. Corsair utilizes a distinctive DIMC engine, consisting of tailored digital circuits, to seamlessly integrate computing functionalities within programmable memory. Corsair achieves this by introducing a multiplying component through the incorporation of four additional transistors into a 6T SRAM bit cell. The most important part of the chip design is the DIMC engine which is a digital version of in-memory compute.

Rebellions is a pioneering AI semiconductor startup that aims to accelerate the development of a next-generation AI chip, capitalizing on the increasing interest in the hardware required for running artificial intelligence services. They have announced a strategic partnership with Samsung Electronics to co-develop their next-generation AI chip, Rebel. The chip will be manufactured using Samsung’s cutting-edge 4-nanometer fabrication process and will incorporate Samsung Electronics’ HBM3E memory technology.

Tesla Dojo is a supercomputer designed and built by Tesla for computer vision video processing and recognition. It will be utilized to train Tesla's machine learning models, enhancing its Full Self-Driving (FSD) advanced driver-assistance system. Dojo operates using its own arithmetic formats and methods. This standard defines Tesla's arithmetic formats and methods for the new 8-bit and 16-bit binary floating-point arithmetic in computer programming environments for deep learning neural network training.

Google's Tensor Processing Units (TPUs) are custom-designed application-specific integrated circuits (ASICs) developed by Google specifically for accelerating machine learning workloads, particularly those that involve training and running deep neural networks. TPUs are optimized for matrix multiplication / linear algebra operations, which are fundamental to many deep learning algorithms. TPUs employ a systolic array architecture which is essentially just a large physical matrix. Software designed for TPUs necessitates compilation through the Accelerator Linear Algebra (XLA) compiler. XLA operates as a just-in-time compiler, processing the graph generated by an ML framework application and converting the linear algebra, loss, and gradient elements of the graph into TPU machine code.

AWS Inferentia, developed by AWS, is a specialized machine learning chip crafted for delivering top-tier inference with remarkable performance. Essentially in architecture the Infertia devices are just a composition of multiple different accelerator engines like ScalarEngine for scalar-computations , VectorEngine for Vector computations , TensorEngine for tensor operations in design it’s very complex but very similar to the systolic array architecture in Google TPU’s. AWS Inferentia 2 is very similar to Inferentia 1 in architecture the notable difference being it has 4 times more memory capacity, 16.4 times higher memory bandwidth along with higher generation PCIE for faster data exchange.

SambaNova is a company that is constructing an advanced, integrated, and secure AI system for multiple applications. The company addresses the challenges posed by the demands of large models, lagging performance, scarcity, and the high cost of legacy compute systems, which have limited access to scaled AI. SambaNova provides dedicated hardware that is always available, reliable, and predictable. They offer services ranging from purpose-built chips to the most powerful models, all of which are available at a lower cost to build and run.

Anari excels in developing innovative solutions that leverage advanced cloud and hardware technologies to achieve superior vector search performance in data-intensive applications. They have an platform which provides an unparalleled foundation for fast and scalable vector search, revolutionizing business data analysis.

The CS-2 system represents a holistic solution encompassing innovations in three fundamental domains. First, it features the second-generation Cerebras Wafer Scale Engine (WSE-2), setting a new standard as the industry's largest and only processor with multi-trillion transistors. Secondly, The CS-2 incorporates the Cerebras software platform, Working well with all of the tools Cerebras SDK has to offer. Compared to clusters of GPUs — which can take weeks or months to set up, require extensive parameter tuning to get functioning correctly, occupy dozens of datacenter racks and require proprietary InfiniBand to cluster — the CS-2 takes minutes to set up. Simply plug in the standardsbased 100 Gigabit Ethernet links to a switch, and you you can start training your models.

Groq specializes in developing hardware for artificial intelligence (AI) applications. Groq has designed a tensor processing unit (TPU) called the Groq Tensor Streaming Processor (TSP), which is aimed at accelerating machine learning workloads.

The Grace Hopper Superchip by NVIDIA is a cutting-edge architectural innovation that merges the remarkable computing power of the NVIDIA Hopper GPU with the adaptability and performance of the NVIDIA Grace CPU. This integration is made possible through the utilization of a high-speed NVIDIA NVLink Chip-2-Chip (C2C) interconnect, facilitating rapid data exchange and shared memory between the GPU and CPU components. This powerful combination along with a large amount of high-bandwidth-memory (HBM) enables seamless cooperation between these processing units and enhances overall system performance. The NVLink Switch System optimizes data transfers and coordination among various components, thereby further improving the efficiency and speed of data processing within the system. The NVIDIA Grace CPU provides up to 512 GB of LPDDR5X memory, which delivers the optimal balance between memory capacity, energy efficiency, and performance. It supplies up to 546 GB/s of LPDDR5X memory bandwidth, which NVLink-C2C makes accessible to the GPU at 900 GB/s total bandwidth.

Flex Logix Technologies is a semiconductor startup designing reconfigurable AI accelerator chips. They offer EFLX eFPGA, providing a customized array with precisely the resources you need, eliminating wastage. Additionally, certain portions of an FPGA design are fixed, allowing them to be hardwired into your SoC.

Kneron produces AI SoC processors that offer the ideal balance of performance, power saving, and cost for hardware makers that are looking to benefit from on-device edge AI. Powered by Kneron’s proprietary NPU that accelerates neural network models, Kneron SoC make possible endless AI applications for smart devices.

Kalray has multiple different hardware offerings, including their unique and patented DPU (Data Processing Unit) processors, acceleration cards, as well as market-leading data management and storage hardware, either separated or in combination.

Lightmatter is delivering a new paradigm in semiconductor chip architecture and the next transition for large-scale computing. The company has developed photonic processors that are faster, more efficient and cooler than any conventional processors in existence today and is answering the call for increased compute speed, low energy density, and reduced chip heating. Lightmatter is set to enable the continued rapid growth of artificial intelligence computing while minimizing its well-known and growing impact on the environment.

SiFive provides processor core IP based on the RISC-V architecture, allowing others to design and customize their own processors according to their specific needs. The company aims to promote open collaboration and innovation in the field of processor design by providing accessible and customizable solutions through the RISC-V ISA. This approach contrasts with traditional closed-source processor architectures, offering more flexibility and transparency to developers and manufacturers.

Graphcore's IPU (Intelligence Processing Unit) is a type of specialized hardware accelerator designed for machine learning workloads. The IPU is optimized for highly parallelized and computationally intensive deep learning tasks, making it well-suited for training and inference in neural networks. Graphcore's IPU has a unique architecture, which is specifically designed to handle the irregular and sparse data patterns often encountered in deep learning models where traditional GPUs fall short. The IPU comes in different generations, such as the IPU-M2000 and IPU-POD, each offering various levels of performance and scalability for AI applications. The IPU is built around a tile architecture, which consists of multiple processing nodes connected in a two-dimensional grid. Each tile includes a number of processor cores, a local memory, and a router for communication between tiles.

BrainChip is a vendor focusing on making AI ubiquitous through innovation that accelerates personalized artificial intelligence everywhere. They've developed Akida technology which is inspired by the brain, the most efficient cognitive processor that we know of. It is the result of more than 15 years of AI architecture research.

Mythic goes beyond conventional digital architectures, memory, and calculation elements and features the Mythic Analog Matrix Processors (Mythic AMP™), which offers significant advantages in power, performance, and cost. Mythic’s unique analog compute architecture performs trillions of operations required for AI inference inside dense flash-memory arrays.

The Gaudi acceleration platform was conceived and designed to cater to the training and inference requirements of large-scale artificial intelligence applications, providing enterprises and organizations with efficient and high-performance deep learning computing capabilities. The Gaudi chip is equipped with a powerful GEMM engine, serving as a robust computational unit. This engine collaborates with an array of tiles, each featuring dedicated local scratchpad memory and a Tensor Processing Core (TPC). These TPCs are equipped with vector computation units capable of handling operations with 8-bit, 16-bit, or 32-bit precision. Communication between TPCs and GEMM Engines is facilitated through DMA and shared memory, while they are connected to the host processor via PCIe.

The Nvidia Blackwell architecture succeeds the previous-generation Nvidia Ada Lovelace architecture , And wiil also host an entirely new SM (streaming multiprocessor) structure which is part of the new design; with dedicated denoising accelerators to assist in ray tracing performance. CUDA cores will get a minor bump in counts to 18,432

WebGPU is a groundbreaking browser standard set to transform web applications, with the shared goal of significantly boosting their efficiency and performance to rival native applications, all while ensuring compatibility across diverse hardware platforms. WebGPU, building on lower-level graphics APIs, introduces more extensive GPU-accelerated computing capabilities, surpassing its predecessor, WebGL, and enabling web applications to leverage modern graphics hardware for tasks like rendering 3D graphics and data processing. WebAssembly, or WASM, complements this effort by unlocking CPU-based computing capabilities, running near-native code in web applications, improving responsiveness, and is already integrated into most modern web browsers. Visit (https://caniuse.com/?search=webgpu) to see if your

The AWS Trainium is very similar to Inferentia in architecture, being a composition of multiple different accelerator engines such as ScalarEngine for scalar computations, VectorEngine for vector computations, and TensorEngine for tensor operations. The notable and major difference is that Trainium hardware features NeuronLink-v2 instead of v1.

Genesis Cloud is a platform that provides specialized GPUs and other compute infrastructure. They provide a fast launch time of less than 2 minutes and autoscale up and down for inference.

Skypilot abstracts away cloud infrastructure burdens across providers like AWS, GCP and Azure. It is a framework to control and run ML models in any cloud. Skypilot maximizes GPU availibility by provisioning GPUs across different regions and zones that a user can access. It also is easy to use and supports most ML workloads with minimal changes in code

Google Vertex AI is a managed machine learning service and is an end-to-end machine learning platform for building, training and deploying ML models. For deployment, Vertex AI allows users to upload their model to their model registry and easily deploy the container image to a production environment.

FluidStack offers a large-scale GPU cloud for AI and LLM training, featuring access to over 50,000 GPUs and thousands of data centers across more than 50 countries. It provides instant access to interconnected NVIDIA A100s and H100s, emphasizing rapid deployment and cost-effective solutions for training and inference.

Paperspace, a product by Digital Ocean, plays a pivotal role in the complete machine learning (ML) lifecycle, covering everything from ML development and training to deployment. Specifically, when it comes to deployment, Paperspace empowers users to easily transform their machine learning models into API endpoints. Users simply need to provide their model file and configure parameters, including features like autoscaling, and there is minimal to no extra effort required from the user.

Banana is a serverless GPU service tailored for AI applications. It allows users to quickly deploy AI models using custom Python framework, connect to data stores, and run inference efficiently. Banana offers built-in CI/CD support, facilitating Docker image creation and deployment on their serverless GPU infrastructure. The service excels in autoscaling applications, minimizing cold boot times to ensure rapid response to changing traffic patterns.

Lepton AI is a serverless GPU cloud platform for model deployment. They allow model deployment with any direct use of Kubernetes, reduce the infrastructure burden and allow the user to deploy with a single command. Deployments also autoscale on demand.

Modal specializes in running generative AI models and large-scale batch jobs. It offers a custom-built container system and infrastructure management, allowing scaling to hundreds of GPUs. Modal emphasizes rapid deployment without YAML, integrating container images and hardware specifications directly in code.

Inferless offers a serverless GPU solution for scaling machine learning inference without the complexities of server management. It enables rapid model deployment and customization, focusing on reducing infrastructure costs and enhancing user flexibility. Inferless streamlines the deployment process, handling scalability and operational challenges.

Qwak simplifies the ML development lifecycle, offering a single platform that streamlines the process from prototype to production. It's designed to reduce the complexities of MLOps, allowing teams to focus on building ML projects without infrastructure worries. Qwak is recognized by leading ML teams worldwide for its comprehensive and efficient ML solutions.

Baseten is a machine learning infrastructure service designed for efficient deployment and serving of ML models. It offers quick setup, open-source model packaging using Truss, and horizontally scalable infrastructure to handle growing traffic without complex configurations. The service optimizes each step of the ML pipeline for faster model scaling and provides comprehensive logs and health metrics for monitoring. Users can customize the infrastructure and choose between deploying on their infrastructure or using AWS and GCP credits (available soon for startups) for deployment on Baseten.

Mystic is a low-latency serverless API platform that enables the quick and cost-efficient deployment of machine learning models. With a RESTful API and Python SDK, it offers a seamless way to run AI models and get API endpoints within seconds. Users only pay for inference time, with serverless pricing on a shared cluster.

Coreweave is a specialized cloud service for GPU acceleration. It allows users to run jobs on bare metal nodes while fully managing Kubernetes, eliminating the need for users to manage any infrastructure. This combines the ease of use of serverless architectures with the fast performance of Kubernetes. It supports autoscaling and spins up new instances in seconds. All that users need to provide is a Docker container and Coreweave automates the rest.

AnyScale provides an AI Application Platform for developing, running, and scaling AI workloads, leveraging Ray, a popular open-source framework. It is designed for high-performance computing, offering significant throughput and cost advantages.

Cerebrium simplifies the process of training, deploying, and monitoring machine learning models using a minimal amount of code. It offers seamless deployment with support for major ML frameworks, including PyTorch, ONNX, and HuggingFace models, as well as prebuilt models optimized for sub-second latency. Additionally, it provides effortless model fine-tuning and monitoring capabilities with integration into top ML observability platforms, enabling alerts about feature or prediction drift, model version comparisons, and in-depth insights into model performance.

LaunchFlow is a versatile platform designed for building and deploying real-time applications directly from your code editor. With its seamless integration into popular code editors like VSCode, it allows users to create and deploy applications in less than 60 seconds. This platform is 100% Python-powered and offers a drag-and-drop editor, preloaded templates, connectors for various cloud providers, and VSCode extensions for both local and remote development. LaunchFlow automates cloud infrastructure provisioning from application code, ensuring serverless operation without the need for server management. It offers ready-to-use templates for real-time IoT, machine learning, and security applications, enabling effortless scaling and efficient data analysis.

Outerbounds offers a unified platform for data science, ML, and AI projects, featuring the open-source Metaflow for application design and development. It provides modern cloud workstations, easy APIs, and secure, scalable deployment options. Outerbounds is designed for efficient compute and flexible integration, ensuring high availability and SOC2-compliant security for all data and ML applications.

Valohai is an MLOps platform designed for ML pioneers, providing the tools needed for a wide range of machine learning applications. It simplifies the training and deployment of machine learning models, whether it's large language models (LLMs), computer vision, or other ML tasks. Users can get started quickly using templates, fine-tune LLMs with their own data, or work on image classification tasks. Valohai supports a flexible approach, allowing users to start from scratch or existing projects without the need to learn a new SDK.

Beam Cloud is a serverless GPU platform for deploying ML models without managing the infrastructure burden. Along with autoscaling, it comes with a coldstart time of 1.21s and a web-based dashboard apart from a simple API

Microsoft Azure Machine Learning is a platform that provides a suite of tools for building, training and deploying ML models. It covers the whole lifecycle - from data preprocessing to model deployment and monitoring. Azure ML provides different deployment options such as Azure Kubernetes Service (AKS), Azure App Service and Azure Functions

OctoAI offers an efficient and user-friendly platform for GenAI app development, featuring an Image Generation solution and a Compute Service. Users can leverage pre-optimized endpoints for open source models or fine-tune models for their specific needs. The platform provides autoscaling and ample compute capacity, ensuring reliability and scalability.

Crusoe Cloud is a cloud computing platform that provides high performance, while remaining Carbon negative, at a good price in comparison to other platforms. It offers NVIDIA's GPUs and GPU enabled VMs for deploying your models. GPU VMs can be created and setup on demand and are charged on that basis.

Jarvis Labs is a platform to rent GPU servers. With high availability across reserved and spot instances, and preconfigured environments, they make it easy for the user to both train and deploy models

Runpod is a provider of serverless GPU compute for AI training and inference. Apart support for autoscaling and a coldstart time of less than half a second, they also charge by the second.

Hyperstack provides access to NVIDIA GPUs, including H100s and A100s, for accelerated AI, rendering, and data analytics. It offers managed Kubernetes services, eco-friendly infrastructure, and efficient networking and storage solutions. Hyperstack's cloud services are designed for high-demand workloads, offering cost-effective and scalable solutions for diverse needs.

Replicate offers a simplified approach to running machine learning models at scale, making it accessible even for those without deep machine learning knowledge. Users can run models with just a few lines of code using Replicate's Python library or query the API directly with their preferred tool. The platform hosts a vast library of machine learning models, including language models, video creation and editing models, super resolution models, image restoration models, and more, contributed by the community. Replicate also introduces Cog, an open-source tool for packaging machine learning models in production-ready containers. It streamlines the deployment process, automatically generating scalable API servers for defined models and offering automatic scaling to handle varying traffic loads.

Amazon SageMaker is a machine learning service by Amazon Web Services (AWS) that offers a complete suite of tools for building, training, and deploying machine learning models. SageMaker simplifies the machine learning process by automating various tasks, such as data preprocessing, model training, and hyperparameter tuning. Users can choose from built-in algorithms or bring their own custom code to train models. Once models are trained, SageMaker allows for easy deployment, scaling, and monitoring of the models in production environments.

ZenML is an open-source MLOps framework designed for creating simplified machine learning workflows. It facilitates easy pipeline creation and execution using standard Python code without requiring special infrastructure or software.