Learning ONNX for trading - page 15

[MetaQuotes]

INT8 Inference of Quantization-Aware trained models using ONNX-TensorRT

INT8 Inference of Quantization-Aware trained models using ONNX-TensorRT

Dheeraj Peri, a deep learning software engineer at NVIDIA, explains the basics of quantization and how TensorRT supports quantized networks through various fusions. They focus on models trained using TensorFlow 2.0 framework and how to perform post-training quantization (PTQ) and quantization-aware training (QAT). The process of deploying a model trained using Nvidia TF2 quantization toolkit with ONNX-TensorRT is explained, and the accuracy and latency results are presented for various ResNet models. Overall, the end-to-end QAT workflow from TensorFlow to TensorRT deployment via ONNX-TensorRT is demonstrated.

• 00:00:00 In this section, Dheeraj, a deep learning software engineer at NVIDIA, discusses the basics of quantization and how TensorRT supports quantized networks through various fusions. He explains that quantization is the process of converting continuous values into a discrete set of values using linear or non-linear scaling techniques. They focus on models that are trained using TensorFlow 2.0 framework and how to perform post-training quantization (PTQ) and quantization-aware training (QAT). Dhiraj also highlights the differences between NVIDIA's quantization toolkit and TF mod toolkit, where the nodes are placed differently in the convolution layers.
  
  
• 00:05:00 In this section, the process of deploying a model trained using Nvidia TF2 quantization toolkit with ONNX-TensorRT is explained. The process involves quantizing the pre-trained TensorFlow 2.0 model with the Nvidia toolkit, fine-tuning it for a small number of epochs to simulate the quantization process, and then converting the model into an ONNX format. The ONNX graph is then used to generate TensorRT engine using TensorRT API. The accuracy and latency results for various ResNet models are presented and it is observed that quantization-aware-trained models (QAT) show better accuracy than post-training quantization (PTQ) models during inference. The QAT models show a similar latency compared to PTQ models but it depends on the placement of QDQ nodes and their fusions. Overall, the end-to-end QAT workflow from TensorFlow to TensorRT deployment via ONNX-TensorRT is demonstrated.

[MetaQuotes]

Practical Post Training Quantization of an ONNX Model

Practical Post Training Quantization of an ONNX Model

The video discusses how to implement quantization to reduce the size of a TensorFlow model to an ONNX quantized model. The ONNX model is significantly smaller in size and can be executed faster on a CPU. The author provides code snippets and instructions on how to implement dynamic quantization and how to check the CPU speed.

The video shows the process of quantizing a machine learning model to make it faster and lighter, while acknowledging that it may lead to a drop in accuracy. The ONNX and TensorFlow models are compared to a quantized model, with the latter being found to be faster and lighter. However, the quantized model does not benefit as much from the use of GPUs as the other models do. The accuracy of the quantized model is then evaluated and found to have only a slight drop. The process of visualizing ONNX models is also discussed, with the use of Loot Rodas Neutron app demonstrated. The overall process results in a reduction in model size from one gigabyte to 83 megabytes with minimal loss in accuracy.

[MetaQuotes]

QONNX: A proposal for representing arbitrary-precision quantized NNs in ONNX

QONNX: A proposal for representing arbitrary-precision quantized NNs in ONNX

The speaker discusses low precision quantization, with an example of its application in wireless communication. They propose QONNX, a dialect for representing arbitrary-precision quantized neural networks in ONNX. QONNX simplifies the quantization representation, extends it to a wider set of scenarios, and offers options for different types of roundings and binary quantization. It is being used for deployment on FPGAs and is integrated into the Brevitas Python quantization library, with NQCDQ set to be integrated into the next release.

• 00:00:00 In this section, the speaker talks about the concept of low precision quantization, which means quantization below 8 bits. The speaker provides an example of how low precision quantization was used in a modulation classification task for wireless communication, achieving high throughput with reduced latency thanks to quantization aware training. The speaker explains the fundamentals of uniform quantization and proposes extending the representational power of ONNX for low precision neural networks using clipping as an extra function over integer boundaries between quantized and dequantized nodes. However, the speaker acknowledges that this approach has limitations, including being limited to quantized linear operators with an 8-bit output and the inability to adopt different types of roundings.
  
  
• 00:05:00 In this section, the speaker introduces QONNX, which is a dialect for representing arbitrary-precision quantized neural networks in ONNX. QONNX simplifies the quantization representation by merging a sequence of operations for fake quantization into just one node, while also extending it to a wider set of scenarios. It offers options for different types of roundings, broadcasting bit inputs, and binary quantization. The format is being leveraged for deployment on FPGAs as part of the fast machine learning effort, with various tools available for dealing with QONNX that integrate with ONNX runtime and pre-trained low precision models. QONNX is already integrated into the Brevitas Python quantization library, and NQCDQ is set to be integrated into the next release.

[MetaQuotes]

GRCon20 - Deep learning inference in GNU Radio with ONNX

GRCon20 - Deep learning inference in GNU Radio with ONNX

The video discusses using ONNX as an open format for integrating deep learning as a flexible, open-source solution in the radiofrequency domain. The speaker presents their new module GR DNN DN4, which uses Python interfaces for both GNU Radio and ONNX, and demonstrates its capabilities with an example of automatic modulation classification using a deep convolutional neural network model trained on simulated data generated by GNU Radio. They also discuss the requirements and challenges of using deep learning for classification on SDR data with the BGG16 model and suggest using hardware acceleration, such as a GPU, to improve inference and achieve real-time results. The project is open source and collaboration is encouraged.

• 00:00:00 In this section of the video, Oscar Rodriguez discusses his work with deep learning inference in GNU Radio with ONNX. The main goal of their project was to integrate deep learning as a flexible and open-source solution to the radiofrequency domain. They chose ONNX as it is an open format that allows machine learning interoperability among different frameworks, solving the problem of incompatible deep learning frameworks. However, there is a cost of adapting models to ONNX and there may be operational availability issues with certain operations, although this is mitigated by the fact that ONNX is actively developed and supported by Microsoft. Ultimately, ONNX provides an abstraction layer between the user's model and different deep learning frameworks.
  
  
• 00:05:00 section discusses the use of ONNX, which allows for the design and training of machine learning models in various frameworks such as TensorFlow and PyTorch before converting them into a common format for use in the ONNX block. ONNX defines a set of basic operations commonly used in deep learning models, and its runtime provides interfaces and support for various software and hardware accelerations. The runtime also creates a graph representation of the model which assigns operations to different execution providers based on available accelerators.
  
  
• 00:10:00 In this section, the speaker discusses the extensibility of execution providers in ONNX, which allows for the support of new hardware platforms as long as all ONNX operations have been implemented on that platform. They then introduce their new module, GR DNN DN4, which uses Python interfaces for both GNU Radio and ONNX. The sync module adapts inputs to the expected format of the model, fits the model with the transformed data, and then transforms the output back to a single-dimensional format. The module also allows for the selection of different execution providers supported in ONNX. The speaker goes on to demonstrate the capabilities of GR DNN DN4 with an example of automatic modulation classification using a deep convolutional neural network model trained on simulated data generated by GNU Radio.
  
  
• 00:15:00 In this section, the speaker discusses using deep learning for classification on SDR data with the BGG16 model. They explain that the input of the model requires a vector of 128 IQ values, which must be adapted to the output of the SDR device. They also note that deep learning inference is computationally intensive and that performance depends on the model's complexity. The speaker concludes by suggesting that using hardware acceleration, such as a GPU, can improve inference and achieve real-time results.
  
  
• 00:20:00 In this section, the speaker discusses a new radio implementation that can integrate deep learning inference and software-defined radio (SDR) using a standard format for deep learning model representation and supporting various acceleration methods. The speaker demonstrates how the module can be used for automatic modulation classification and achieve real-time inference with hardware acceleration. The speaker also discusses future improvements to the module, including making it more flexible for different types of deep learning models and including pre-processing functionality within the block. The project is open source and collaboration is encouraged.