RKNN Inference Test

1. Introduction to RKNPU

NPU (Neural Processing Unit) is a processor specifically designed to accelerate neural network computations. To meet the demands of artificial intelligence, Rockchip has gradually integrated NPUs into its processors. The NPU built into Rockchip processors is called RKNPU. The LuckFox Core3576 series core board is equipped with the Rockchip RK3576 chip, which has a self-developed NPU. This NPU offers high computation precision and supports INT4, INT8, INT16/FP16/BF16, and TF32 mixed quantization. To use RKNPU2, you need to use the SDK and toolset.

Data Type	INT8	INT16/FP16/BF16	INT4	TF32
Single-Core Performance (TOPS)	2	1	4	0.5
Dual-Core Performance (TOPS)	4	3	8	1
Sparse Performance (TOPS)	6	3	\	\

2. Introduction to RKNN-Toolkit2

RKNN-Toolkit2 provides C or Python interfaces on PC platforms, simplifying the deployment and execution of models. With this tool, users can easily perform model conversion, quantization, inference, performance and memory evaluation, quantization accuracy analysis, and model encryption. The RKNN software stack helps users quickly deploy AI models onto Rockchip chips. The overall framework is shown below:

To use RKNPU, users first need to run the RKNN-Toolkit2 tool on their computer to convert the trained model into RKNN format. Then, the model can be deployed on the development board using the RKNN C API or Python API. This section introduces how users can quickly use RKNPU on Core3576 series boards.

3 RKNN-Toolkit2 Installation (Ubuntu 22.04 X86_64 Platform)

3.1 Local Installation

1. Environment Requirements.

   Operating System Version	Ubuntu18.04(x64)	Ubuntu20.04(x64)	Ubuntu22.04(x64)
   Python Version	3.6/3.7	3.8/3.9	3.10/3.11

2. Download RKNN-Toolkit2.

   git clone https://github.com/airockchip/rknn-toolkit2.git

3. Install Python Environment.

   sudo apt-get update
   sudo apt-get install python3 python3-dev python3-pip
   sudo apt-get install libxslt1-dev zlib1g zlib1g-dev libglib2.0-0 libsm6 libgl1-mesa-glx libprotobuf-dev gcc

4. Install RKNN-Toolkit2 Dependencies.

   pip3 install -r rknn-toolkit2/packages/requirements_cpxx-2.2.0.txt
   
   # such as: 
   pip3 install -r rknn-toolkit2/packages/requirements_cp310-2.2.0.txt

   Select the corresponding dependencies based on your Python version:

   Python Version	RKNN-Toolkit2 Dependencies
   3.6	requirements_cp36-2.2.0.txt
   3.7	requirements_cp37-2.2.0.txt
   3.8	requirements_cp38-2.2.0.txt
   3.9	requirements_cp39-2.2.0.txt
   3.10	requirements_cp310-2.2.0.txt
   3.11	requirements_cp311-2.2.0.txt
   3.12	requirements_cp312-2.2.0.txt

5. Install RKNN-Toolkit2.

   pip3 install rknn-toolkit2/packages/rknn_toolkit2-x.x.x+xxxxxxxx-cpxx-cpxx-linux_x86_64.whl
   
   # such as: 
   pip3 install rknn-toolkit2/packages/rknn_toolkit2-2.2.0+81f21f4d-cp310-cp310-linux_x86_64.whl

   The package name format is: rknn_toolkit2-{version}+{commit}-cp{python_version}-cp{python_version}-linux_x86_64.whl. Choose the appropriate package based on the Python version:

   Python Version	RKNN-Toolkit2 Package
   3.6	rknn_toolkit2-2.2.0-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
   3.7	rknn_toolkit2-2.2.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
   3.8	rknn_toolkit2-2.2.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
   3.9	rknn_toolkit2-2.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
   3.10	rknn_toolkit2-2.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
   3.11	rknn_toolkit2-2.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
   3.12	rknn_toolkit2-2.2.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

   If no errors occur after running the following command, the installation is successful:

   python3
   from rknn.api import RKNN

3.2 Conda Installation

It is recommended to use Conda to create a Python virtual environment, allowing flexible switching between different applications and avoiding version mismatches. For example, different Python virtual environments are required for AI model training and conversion.

3.2.1 Install Miniconda

1. Check if Miniconda or another conda tool is installed by running:

   conda --version

2. Download the installer:

   wget https://mirrors.tuna.tsinghua.edu.cn/anaconda/miniconda/Miniconda3-4.6.14-Linux-x86_64.sh

3. Install Miniconda:

   chmod 777 Miniconda3-4.6.14-Linux-x86_64.sh
   bash Miniconda3-4.6.14-Linux-x86_64.sh

   • Note: The miniconda installation package must be given execute permission using chmod 777.

   • Follow the installation steps, read and accept the license terms, and proceed with the installation. The miniconda folder will be created in the home directory.

4. Enter the Conda base environment:

   source ~/miniconda3/bin/activate # Miniconda3 installation directory (customize based on your setup)
   # After activation, the command prompt should change to:
   # (base) xxx@xxx:~$

5. To automatically activate the Miniconda environment each time you open a terminal, add the activation command to your shell configuration file:

   vim nano ~/.bashrc
   
   # Add the following line at the end:
   source ~/miniconda3/bin/activate
   
   # exit conda environment
   conda deactivate

3.2.2 Creating RKNN-Toolkit2 Conda Environment

1. Create the RKNN-Toolkit2 development Conda environment, specifying Python version 3.8 (recommended version):

   conda create -n RKNN-Toolkit2 python=3.9
   • Type y to confirm installation of the default packages.

2. Enter the RKNN-Toolkit2 Conda Environment.

   conda activate RKNN-Toolkit2

3. Verify the Correct Python Version.

   python --version
   • Note: In some development environments, the Python version may not switch properly after creating the Conda environment. Restarting the terminal may resolve this issue.

4. Clone the RKNN-Toolkit2 repository:

   git clone https://github.com/airockchip/rknn-toolkit2.git

5. Enter the directory:

   cd rknn-toolkit2

6. Install RKNN-Toolkit2 related dependency libraries. cp38 is the corresponding Conda environment python version. The version used in the experiment is 3.8, so use the dependency with the suffix cp38

   pip install tf-estimator-nightly==2.8.0.dev2021122109 
   pip install -r rknn-toolkit2/packages/requirements_cp39-2.2.0.txt

7. Install RKNN-Toolkit2:

   pip install rknn-toolkit2/packages/rknn_toolkit2-2.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
   • Select the installation package from the packages folder based on your Python version. For Python 3.8, use the package with the cp38 suffix. The 81f21f4d in the filename is a commit ID; choose the appropriate version for your setup.

8. Verify Installation,If no errors occur, the installation is successful.

   python
   >>> from rknn.api import RKNN

4. rknn_model_zoo Application Example

rknn_model_zoo is a collection of deployment examples for mainstream algorithms supported by Rockchip NPU. The latest examples include mobilenet and yolo model deployments. This chapter demonstrates the usage of the rknn_model_zoo example using yolov10.

4.1 Export the RKNN Model

1. Download rknn_model_zoo.

   git clone https://github.com/airockchip/rknn_model_zoo.git

2. Obtain the YOLOv10 ONNX Model.

   cd <rknn_model_zoo Path>/rknn_model_zoo/examples/yolov10/model
   chmod a+x download_model.sh
   ./download_model.sh

3. Execute the model conversion program convert.py in the rknn_model_zoo/examples/yolov10/python directory. Usage:

   conda activate RKNN-Toolkit2
   cd <rknn_model_zoo Path>/rknn_model_zoo/examples/yolov10/python
    python3 convert.py ../model/yolov10n.onnx rk3576
   # output model will be saved as ../model/yolov10.rknn

   python3 convert.py <onnx_model> <TARGET_PLATFORM> <dtype(optional)> <output_rknn_path(optional)>

   Parameters:

   • <onnx_model>: Path to the ONNX model.
   • <TARGET_PLATFORM>: NPU platform name (e.g., rk3576).
   • <quant_dtype> (optional): Quantization type (i8 for INT8 quantization, fp for no quantization; default: i8).
   • <output_rknn_path> (optional): Output path for the RKNN model (default: same directory as the ONNX model).

4.2 Compile and Build

1. After successfully converting the ONNX model to the RKNN model, cross-compile the examples in the rknn_model_zoo/examples/yolov10 directory. Before compiling the examples, you need to set the following environment variables:

   export GCC_COMPILER=/home/xt/Luckfox/omni3576/sdk-1026/prebuilts/gcc/linux-x86/aarch64/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu

2. Execute the build-linux.sh script in the rknn_model_zoo directory. This script will compile the example:

   chmod +x ./build-linux.sh
   ./build-linux.sh -t rk3576 -a aarch64 -d yolov10

   • Compilation process:

     (RKNN-Toolkit2) luckfox@luckfox:~/rknn_model_zoo$ ./build-linux.sh -t rk3576 -a aarch64 -d yolov10
     ./build-linux.sh -t rk3576 -a aarch64 -d yolov10
     /home/xt/Luckfox/omni3576/sdk-1026/prebuilts/gcc/linux-x86/aarch64/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu
     ===================================
     BUILD_DEMO_NAME=yolov10
     BUILD_DEMO_PATH=examples/yolov10/cpp
     TARGET_SOC=rk3576
     TARGET_ARCH=aarch64
     BUILD_TYPE=Release
     ENABLE_ASAN=OFF
     DISABLE_RGA=OFF
     INSTALL_DIR=/home/xt/conda/rknn_model_zoo/install/rk3576_linux_aarch64/rknn_yolov10_demo
     BUILD_DIR=/home/xt/conda/rknn_model_zoo/build/build_rknn_yolov10_demo_rk3576_linux_aarch64_Release
     CC=/home/xt/Luckfox/omni3576/sdk-1026/prebuilts/gcc/linux-x86/aarch64/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-gcc
     CXX=/home/xt/Luckfox/omni3576/sdk-1026/prebuilts/gcc/linux-x86/aarch64/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-g++
     ===================================
     -- The C compiler identification is GNU 10.3.1
     -- The CXX compiler identification is GNU 10.3.1
     -- Detecting C compiler ABI info
     -- Detecting C compiler ABI info - done
     -- Check for working C compiler: /home/xt/Luckfox/omni3576/sdk-1026/prebuilts/gcc/linux-x86/aarch64/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-gcc - skipped
     -- Detecting C compile features
     -- Detecting C compile features - done
     -- Detecting CXX compiler ABI info
     -- Detecting CXX compiler ABI info - done
     -- Check for working CXX compiler: /home/xt/Luckfox/omni3576/sdk-1026/prebuilts/gcc/linux-x86/aarch64/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-g++ - skipped
     -- Detecting CXX compile features
     -- Detecting CXX compile features - done
     -- !!!!!!!!!!!CMAKE_SYSTEM_NAME: Linux
     -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
     -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
     -- Check if compiler accepts -pthread
     -- Check if compiler accepts -pthread - yes
     -- Found Threads: TRUE  
     -- Configuring done (0.4s)
     -- Generating done (0.0s)
     -- Build files have been written to: /home/xt/conda/rknn_model_zoo/build/build_rknn_yolov10_demo_rk3576_linux_aarch64_Release
     [ 33%] Building C object utils.out/CMakeFiles/imageutils.dir/image_utils.c.o
     [ 33%] Building C object utils.out/CMakeFiles/fileutils.dir/file_utils.c.o
     [ 33%] Building C object utils.out/CMakeFiles/audioutils.dir/audio_utils.c.o
     [ 33%] Building C object utils.out/CMakeFiles/imagedrawing.dir/image_drawing.c.o
     [ 41%] Linking C static library libaudioutils.a
     ...
     ...
     ...
     [100%] Linking CXX executable rknn_yolov10_demo
     [100%] Built target rknn_yolov10_demo
     [ 16%] Built target imageutils
     [ 33%] Built target fileutils
     [ 50%] Built target imagedrawing
     [ 83%] Built target rknn_yolov10_demo
     [100%] Built target audioutils
     Install the project...
     -- Install configuration: "Release"
     -- Installing: /home/xt/conda/rknn_model_zoo/install/rk3576_linux_aarch64/rknn_yolov10_demo/./rknn_yolov10_demo
     -- Set non-toolchain portion of runtime path of "/home/xt/conda/rknn_model_zoo/install/rk3576_linux_aarch64/rknn_yolov10_demo/./rknn_yolov10_demo" to "$ORIGIN/../lib"
     -- Installing: /home/xt/conda/rknn_model_zoo/install/rk3576_linux_aarch64/rknn_yolov10_demo/model/bus.jpg
     -- Installing: /home/xt/conda/rknn_model_zoo/install/rk3576_linux_aarch64/rknn_yolov10_demo/model/coco_80_labels_list.txt
     -- Installing: /home/xt/conda/rknn_model_zoo/install/rk3576_linux_aarch64/rknn_yolov10_demo/model/yolov10.rknn
     -- Installing: /home/xt/conda/rknn_model_zoo/install/rk3576_linux_aarch64/rknn_yolov10_demo/lib/librknnrt.so
     -- Installing: /home/xt/conda/rknn_model_zoo/install/rk3576_linux_aarch64/rknn_yolov10_demo/lib/librga.so

3. After cross-compilation is completed, an install directory will be generated in the rknn_model_zoo directory, which contains the compiled program and library files.

   (RKNN-Toolkit2) ubuntu@ubuntu:~/conda/rknn_model_zoo/install/rk3576_linux_aarch64/rknn_yolov10_demo$ ls
   lib  model  rknn_yolov10_demo

4.3 Run the Program

1. Transfer the rknn_yolov10_demo directory to the development board and execute:

   scp -r rknn_yolov10_demo/ luckfox@192.168.253.105:/home/luckfox

2. After inference, the output image out.png will be generated.

   luckfox@luckfox:~/rknn_yolov10_demo$ ./rknn_yolov10_demo ./model/yolov10.rknn ./model/bus.jpg 
   load lable ./model/coco_80_labels_list.txt
   model input num: 1, output num: 6
   input tensors:
     index=0, name=images, n_dims=4, dims=[1, 640, 640, 3], n_elems=1228800, size=1228800, fmt=NHWC, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
   output tensors:
     index=0, name=487, n_dims=4, dims=[1, 64, 80, 80], n_elems=409600, size=409600, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-38, scale=0.114574
     index=1, name=501, n_dims=4, dims=[1, 80, 80, 80], n_elems=512000, size=512000, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.002001
     index=2, name=508, n_dims=4, dims=[1, 64, 40, 40], n_elems=102400, size=102400, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-57, scale=0.095044
     index=3, name=522, n_dims=4, dims=[1, 80, 40, 40], n_elems=128000, size=128000, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003505
     index=4, name=529, n_dims=4, dims=[1, 64, 20, 20], n_elems=25600, size=25600, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-58, scale=0.061253
     index=5, name=543, n_dims=4, dims=[1, 80, 20, 20], n_elems=32000, size=32000, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003792
   model is NHWC input fmt
   model input height=640, width=640, channel=3
   origin size=640x640 crop size=640x640
   input image: 640 x 640, subsampling: 4:2:0, colorspace: YCbCr, orientation: 1
   scale=1.000000 dst_box=(0 0 639 639) allow_slight_change=1 _left_offset=0 _top_offset=0 padding_w=0 padding_h=0
   rga_api version 1.10.1_[0]
   rknn_run
   bus @ (88 137 556 438) 0.925
   person @ (109 234 225 536) 0.910
   person @ (210 240 284 511) 0.906
   person @ (478 233 560 519) 0.796
   person @ (80 330 114 518) 0.428
   write_image path: out.png width=640 height=640 channel=3 data=0x7fbdd5c010
   
   
   luckfox@luckfox:~/rknn_yolov10_demo$ ls
   lib  model  out.png  rknn_yolov10_demo

3. Download and check the image. Label definitions are in rknn_yolov10_demo/model/coco_80_labels_list.txt. For example, the detector can locate office supplies in the image.

   ./rknn_yolov10_demo ./model/yolov10.rknn ./model/laptop.jpg 
   • Inference Results: