RKMPI Instance User Guide

1 Introduction

Camera image capture, preprocessing and hardware coding are realized by using the RKMPI library, image processing is carried out by combining with opencv-mobile, RKNN reasoning results can be labeled on the image and then used as a streaming media server to carry out RTSP pushing, The Local area network PC below can pull and view images using the VLC software.

Examples include the basic use of RKMPI library, two ways of opencv-mobile processing data frames, and two ways of combining RKNN inference. Main effects to be achieved:

AI-IPC base on retinaface face detection
AI-IPC base on yolov5 object detection

2 Support platform

DEMO	CPU	system	Camera
luckfox_pico_rtsp_opencv	RV1103(需修改分辨率)、RV1106	buildroot	sc3336
luckfox_pico_rtsp_opencv_capture	RV1103、RV1106	buildroot	sc3336
luckfox_pico_rtsp_retinaface	RV1103、RV1106	buildroot	sc3336
luckfox_pico_rtsp_retinaface_osd	RV1103、RV1106	buildroot	sc3336
luckfox_pico_rtsp_yolov5	RV1106	buildroot	sc3336

RV1103：Luckfox Pico Luckfox Pico Mini A Luckfpx Pico Miini B Luckfox Pico Plus
RV1106：Luckfox Pico Pro Luckfox Pico Max Luckfox Pico Ultra Luckfox Pico Ultra W

3. Environment construction

Verify that Luckfox-pico Sdk is installed correctly and that the environment variables for the cross-compilation chain are set correctly
```
arm-rockchip830-linux-uclibcgnueabihf-gdb --version
```
- Displays a description of the version information for the cross-compilation chain, indicating that the environment variables are set correctly.

Use the compiled buildroot image or get the image from the network disk. Make sure the camera driver, rknpu driver, rockit driver and rkisp driver are loaded successfully. Perform lsmod the verification on the Luckfox-pico board end.

# lsmod
Module                  Size  Used by    Tainted: G
rockit                154528  0
rknpu                  24552  0
mpp_vcodec            375759  1 rockit
rga3                   86680  1 rockit
sc3336                  9940  1
os04a10                15198  0
phy_rockchip_csi2_dphy     7674  0
phy_rockchip_csi2_dphy_hw     8161  0
video_rkisp           148435  1 rockit
video_rkcif           130130  0
rk_dvbm                 5813  2 mpp_vcodec,video_rkisp

Notice: To ensure that the root file system in the image has enough free space, it is recommended to use a system running on an SD card.

4 RKM PI library usage

4.1 System overview

The Rockchip Media Process Interface ( RKMPI ) provided by Rockchip can support the rapid development of multimedia application software. The platform integrates the hardware resources of the Rockchip chip and shields the complex underlying processing related to the chip from the application software, so that users can complete the development of media processing by calling the corresponding interfaces at the software level, such as input video capture, H264 / H265 / JPEG encoding, video image processing, etc.

System framework

Application layer: application software system developed by users based on RKMPI and other drivers.
RKMPI layer: complete the media processing function based on the adaptation layer (the existing Rockchip chip adaptation packaging interface).
Operating system adaptation layer: It is based on the driver interface provided by the existing hardware module of the Rockchip chip.
Operating system layer: Linux-based system.
Hardware layer: It is composed of Rockchip chip and necessary peripheral devices.

IPC Application Framework

The VI module captures video images, cuts and zooms them, and outputs multi-channel image data with different resolutions.
The VPSS module receives the image transmitted by the VI module, and can process the image by cutting, scaling, rotating, pixel format conversion and the like.
The VENC module can directly receive the image captured by the VI module or the image data output after VPSS processing, superimpose the OSD image set by the user through the RGN module, and then encode and output the corresponding code stream according to different protocols.
Each module channel is bound, and the processed data stream will be directly transmitted to the next bound multimedia module.

4.2 Memory management

Each component of the RKMPI library mainly uses the memory buffer pool to manage the memory. The memory buffer pool allocates and recycles the memory through the memory block, so that the memory resources can be reasonably used in each media processing module. The memory buffer pool is isolated from other application memories to ensure the stable operation of the RKMPI library, but the application data cannot directly operate the memory in the memory buffer pool.

Memory Pool Operation Mode

Internal distribution In the private mode, the memory buffer pool is applied by the channel of the media processing module, and the applied size and number are applied by the channel according to the set parameters.
External allocation In the shared mode, the memory buffer pool can be created by the user to set the size, attributes, and data of the memory buffer block and pass them to the specified module for use.

4.3 Video Input VI

Video Input ( VI ) Component realizes the function of receiving video data from MIPI Rx (including MIPI interface and LVDS interface) and other interfaces. The VI component stores the received data into the specified memory area to realize the acquisition of video data.

VI Internal Data Processing

VI is divided into three levels in terms of software: dev, pipe and channel.

VI device: supports several timing inputs and is responsible for parsing the timing.
VI pipe: It is bound to the back end of the device and is responsible for reprocessing the data parsed by the device. It is mainly used to load the ISP algorithm.
VI channel: the last acquisition channel of video input.

VI function usage Guide

Before initializing VI module, please run ISP algorithm to realize automatic exposure control, automatic gain control, automatic white balance, color correction and other operations to ensure the quality of captured images.
```
SAMPLE_COMM_ISP_Init(CamId, hdr_mode, multi_sensor, iq_dir); 
```
Run the ISP algorithm
```
SAMPLE_COMM_ISP_Run(CamId);
```
Start the VI device
```
RK_MPI_VI_EnableDev(devID);
```

Bind VI devices and VI pipes

RK_MPI_VI_SetDevBindPipe(devID, &stBindPipe);

Set the VI channel properties

RK_MPI_VI_SetChnAttr(PipeID, ChnID, &Chn_attr);

Notice: The VI channel's output resolution reference camera module supports resolution ratios to avoid image distortion.

Start the VI channel
```
RK_MPI_VI_EnableChn(PipeID, ChnID);
```

Bind other multimedia module

MPP_CHN_S viChn;
viChn.enModId  = RK_ID_VI;
viChn.s32DevID = DevID;
viChn.s32ChnID = ChnID;
RK_MPI_SYS_Bind(&viChn, &otherChn);

Unbind from other multimedia
```
RK_MPI_SYS_UnBind(&viChn,&otherChn);
```

Shut down VI channels and device

RK_MPI_VI_DisableChn(PipeID,ChnID);
RK_MPI_VI_DisableDev(DevID);

4.4 Video Processing Subsystem VPSS

Video Processing Subsystem ( VPSS ) supports image scaling, fixed angle rotation, pixel format conversion, cropping and other processing of the input image.

VPSS context

VPSS can be bound with modules such as VI and VENC, where the former is the input source of VPSS and the latter is the receiver of VPSS. Groups can be managed through the VPSS interface, and each group can be bound to only one input source.

VPSS group: control input. Each component reuses the hardware device, and the hardware processes the tasks submitted by each group in turn.
VPSS channel: control the output. Each channel has the functions of scaling and cropping. The image processed by the channel is sent to the receiver of VPSS.

VPSS function usage Guide

Create a VPSS group

RK_MPI_VPSS_CreateGrp(GrpID, &VpssGrpAttr);

Setting VPSS Channel Properties
```
RK_MPI_VPSS_SetChnAttr(GrpID, ChnID, &VpssChnAttr);
```
- Notice: The VPSS group is responsible for managing the input. When setting the attributes, pay attention to setting enPixelFormat the attributes to be consistent u32MaxW with the bound input source and u32MaxH larger than the captured image size of the input source.
Start the VPSS channel
```
RK_MPI_VPSS_EnableChn(GrpID, ChnID);
```
Start the VPSS group
```
RK_MPI_VPSS_StartGrp(GrpID);
```

Output VPSS data

Bind other multimedia module

MPP_CHN_S vpssChn;
vpssChn.enModId  = RK_ID_VI;
vpssChn.s32DevID = DevID;
vpssChn.s32ChnID = ChnID;
RK_MPI_SYS_Bind(&vpssChn, &otherChn);

Acquire frame data directly

RK_MPI_VPSS_GetChnFrame(GrpID, ChnID, &VpssFrame,-1);
void *data = RK_MPI_MB_Handle2VirAddr(VpssFrame.stVFrame.pMbBlk);

Stop and shut down the VPSS group

RK_MPI_VPSS_StopGrp(GrpID);
RK_MPI_VPSS_DestroyGrp(GrpID);

4.5 Video Encoding VENC

The video encoding ( VENC ) component supports multi-channel real-time encoding and each channel of encoding is independent, and supports the scheduling of the Region module to superimpose and shield the encoded image content while encoding. The main supported encoding formats are H264, H265, JPEG and MJPEG.

VENC internal data processing

The VENC obtains the image from the data frame, performs image processing and RGN module processing, and then transmits the image to the encoding channel, and encodes the image in combination with the time stamp and other information of the data frame.

VENC channel: It manages the input and encoding attributes of the image, and uses the RK_MPI_VENC_ StartRecvFrame interface to start video encoding and receive the input image data.

VENC function usage Guide

Create and set up a VENC channel

RK_MPI_VENC_CreateChn(chnId, &stAttr);

The VENC channel requires settings enType and u32Profile parameters to set the encoding type

// H264
stAttr.stVencAttr.enType = RK_VIDEO_ID_AVC;
stAttr.stVencAttr.u32Profile = H264E_PROFILE_MAIN;

VENC needs to set the corresponding structure parameters according to the set encoding type

// Set H264 Attribute
stAttr.stRcAttr.enRcMode = VENC_RC_MODE_H264CBR;
stAttr.stRcAttr.stH264Cbr.u32BitRate = BitRate;
stAttr.stRcAttr.stH264Cbr.u32Gop = Gop;

Start receiving frame data

RK_MPI_VENC_StartRecvFrame(ChnId, &stRecvParam);

Get frame data from VENC output

RK_MPI_VENC_GetStream(ChnId, &stFrame, -1);

Obtain the virtual address of the VENC frame memory block

void *Data = RK_MPI_MB_Handle2VirAddr(stFrame.pstPack->pMbBlk);

Stop receiving frame data
```
RK_MPI_VENC_StopRecvFrame(ChnId);
```
Close the VENC channel
```
RK_MPI_VENC_DestroyChn(ChnId);
```

5 The RKRTSP library usage Guide

RKRTSP is a set of RTSP server software interfaces provided by RockChip for testing only. It is only used for functional verification of the RKMPI library. The RTSP streaming function can be quickly realized through the software interface.

5.1 Basic use of the RKRTSP library

Create an demo of RTSP
```
g_rtsplive = create_rtsp_demo(port);
```

Create the RTSP interface

g_rtsp_session = rtsp_new_session(g_rtsplive, "/live/0");

Setting the RTSP transport properties
```
rtsp_set_video(g_rtsp_session, RTSP_CODEC_ID_VIDEO_H264, NULL, 0);
```
- Notice: Currently, the RKRtsp library software interface does not support MJPEG streaming.

Ynchronization RTSP timestamp

rtsp_sync_video_ts(g_rtsp_session, rtsp_get_reltime(), rtsp_get_ntptime());

Update video stream settings and drive event

rtsp_tx_video(g_rtsp_session, (uint8_t *)pData, PackLen, PTS);
rtsp_do_event(g_rtsplive);

Delete the RTSP instance
```
rtsp_del_demo(g_rtsplive);
```

6 Example of frame number of opencv-mobile annotation pushing

6.1 Image frame acquisition

The internal memory of the RKMPI library is managed by the memory buffer. There are two ways to process the captured image and then push the RTSP stream in the application layer.

Capture using VI component
The captured image is converted into pixel format by VPSS to obtain frame data, the frame data is processed by opencv-mobile after being converted into cv::Mat type, and finally the frame data is copied back to the virtual address of the memory buffer block, Use RK_MPI_VENC_SendFrame to transmit VPSS frame data to the VENC component for encoding.
Capture with opencv-mobile
A memory buff pool is established, image data store in a memory buffer block is obtained from that memory buffer pool, an opencv-mobile is use to capture an image for processing, a pixel format is converted and copied into the memory buffer block, Finally RK_MPI_VENC_SendFrame, the structure data storing the address of the VIDEO_FRAME_INFO_S memory buffer block, timestamp, image size and other information is transferred to the VENC component for encoding.

6.2. Example of RTSP streaming based on VI image capture

Running effect

The VI component of Luckfox-pico pro is used to capture 2304 × 1296 images, and opencv-mobile is used to mark the frame rate and encode it as H264 for RTSP streaming, which can reach about 23 frames.

Notice: Testing with Luckfox-pico mini/plus may not work properly due to memory limitations. Try lowering the capture resolution.

Implementation process

RKRTSP, RKMPI, RKAIQ initialization

// RKAIQ Init
SAMPLE_COMM_ISP_Init(CamID, hdr_mode, multi_sensor, iq_dir);
SAMPLE_COMM_ISP_Run(CamID);

// RTSP Init
g_rtsplive = create_rtsp_demo(554);
g_rtsp_session = rtsp_new_session(g_rtsplive, "/live/0");
rtsp_set_video(g_rtsp_session, RTSP_CODEC_ID_VIDEO_H264, NULL, 0);
rtsp_sync_video_ts(g_rtsp_session, rtsp_get_reltime(), rtsp_get_ntptime());

// RKMPI Init
RK_MPI_SYS_Init();

Initialization of VI, VPSS and VENC components

// VI Init
vi_dev_init();
vi_chn_init(0, width, height);

// VPSS Init
vpss_init(0, width, height);

// VENC Init
RK_CODEC_ID_E enCodecType = RK_VIDEO_ID_AVC;
venc_init(0, width, height, enCodecType);

The pixel format of VI channel shall be consistent with the pixel format of VPSS group

vi_chn_attr.enPixelFormat = RK_FMT_YUV420SP; // VI Chn

stGrpVpssAttr.enPixelFormat = RK_FMT_YUV420SP; // VPSS Group

To facilitate the inferencing of data into the RKNN model, the pixel format is converted to RGB888 using the VPSS component
```
stVpssChnAttr.enPixelFormat = RK_FMT_RGB888;
```
VENC sets the relevant properties for H264 encoding
```
stAttr.stRcAttr.enRcMode = VENC_RC_MODE_H264CBR;
stAttr.stRcAttr.stH264Cbr.u32BitRate = BitRate;
stAttr.stRcAttr.stH264Cbr.u32Gop = GOP;
```
- Notice: The GOP used in the routine is 1. Set all data frames as I frames. Use VLC software on the PC side to set the cache time not to be less than 500 ms. You can try to modify it for optimization.

Bind VI channel to VPSS

stSrcChn.enModId = RK_ID_VI;
stSrcChn.s32DevId = 0;
stSrcChn.s32ChnId = 0;

stvpssChn.enModId = RK_ID_VPSS;
stvpssChn.s32DevId = 0;
stvpssChn.s32ChnId = 0;
s32Ret = RK_MPI_SYS_Bind(&stSrcChn, &stvpssChn);

Acquire a VPSS data frame and a virtual address of a corresponding memory buffer block

RK_MPI_VPSS_GetChnFrame(DevId, ChnId, &stVpssFrame, -1);
void *data = RK_MPI_MB_Handle2VirAddr(stVpssFrame.stVFrame.pMbBlk);

Use opencv-mobile to annotate framerate

cv::Mat frame(height,width,CV_8UC3,data);   
cv::putText(frame,fps_text,
            cv::Point(40, 40),
            cv::FONT_HERSHEY_SIMPLEX,1,
            cv::Scalar(0,255,0),2); 

Send the data to VENC and encode it in H264 format
```
RK_MPI_VENC_SendFrame(vpssChn, &stVpssFrame,-1);
```

The RTSP push flow is performed

RK_MPI_VENC_GetStream(vencChn, &stFrame, -1);
RK_MPI_MB_Handle2VirAddr(stFrame.pstPack->pMbBlk);
rtsp_tx_video(g_rtsp_session, (uint8_t *)pData, stFrame.pstPack->u32Len,
        stFrame.pstPack->u64PTS);
rtsp_do_event(g_rtsplive);

Resource release

RK_MPI_SYS_UnBind(&stSrcChn, &stvpssChn);

RK_MPI_VI_DisableChn(DevId, ChnId);
RK_MPI_VI_DisableDev(DevId);

RK_MPI_VPSS_StopGrp(GrpId);
RK_MPI_VPSS_DestroyGrp(Grpid);

RK_MPI_VENC_StopRecvFrame(ChnId);
RK_MPI_VENC_DestroyChn(ChnId);
// RKRTSP
rtsp_del_demo(g_rtsplive);
// RKIRQ
SAMPLE_COMM_ISP_Stop(CamId);
// RKMPI
RK_MPI_SYS_Exit();

6.3 Example of RTSP streaming based on opencv-mobile capture

Running effect

On Luckfox-pico pro, use opencv-mobile to capture images, label the frame rate, and then encode them as H264 for RTSP streaming, which can reach about 9 frames.

Notice: Using opencv-mobile to capture camera images is relatively simple, but it loses the hardware acceleration of VI components and VPSS components, and there is a significant loss in performance. It is recommended to use VI to capture images.

Implementation process

Create a memory cache pool and allocate memory cache blocks

MB_POOL src_Pool = RK_MPI_MB_CreatePool(&PoolCfg);
MB_BLK src_Blk = RK_MPI_MB_GetMB(src_Pool, width * height * 3, RK_TRUE);

The number of memory blocks that create the settings for the memory buffer pool cannot be less than the number of captured image pixels
```
PoolCfg.u64MBSize = width * height * channel ;
PoolCfg.u32MBCnt = num;
PoolCfg.enAllocType = MB_ALLOC_TYPE_DMA;
```

RKRTSP, RKMPI initialization

// RTSP Init
g_rtsplive = create_rtsp_demo(554);
g_rtsp_session = rtsp_new_session(g_rtsplive, "/live/0");
rtsp_set_video(g_rtsp_session, RTSP_CODEC_ID_VIDEO_H264, NULL, 0);
rtsp_sync_video_ts(g_rtsp_session, rtsp_get_reltime(), rtsp_get_ntptime());

// RKMPI Init
RK_MPI_SYS_Init();

The VENC component is initialized
```
// VENC Init
RK_CODEC_ID_E enCodecType = RK_VIDEO_ID_AVC;
venc_init(0, width, height, enCodecType);
```
- VENC sets the relevant properties for H264 encoding
```
stAttr.stRcAttr.enRcMode = VENC_RC_MODE_H264CBR;
stAttr.stRcAttr.stH264Cbr.u32BitRate = BitRate;
stAttr.stRcAttr.stH264Cbr.u32Gop = GOP;
```
  - Notice: The GOP used in the routine is 1. Set all data frames as I frames. Use VLC software on the PC side to set the cache time not to be less than 500 ms. You can try to modify it for optimization.

Build a VIDEO_FRAME_INFO_S data frame structure

VIDEO_FRAME_INFO_S h264_frame;
h264_frame.stVFrame.u32Width = width;
h264_frame.stVFrame.u32Height = height;
h264_frame.stVFrame.u32VirWidth = width;
h264_frame.stVFrame.u32VirHeight = height;
h264_frame.stVFrame.enPixelFormat =  RK_FMT_RGB888; 
h264_frame.stVFrame.u32FrameFlag = 160;
h264_frame.stVFrame.pMbBlk = src_Blk;
unsigned char *data = (unsigned char *)RK_MPI_MB_Handle2VirAddr(src_Blk);

The data frame pixel format shall be consistent with the VENC channel pixel format.

Capture the image with opencv-mobile and label the frame rate, copy the image data to the memory buffer block of the data frame in RGB888 format

cap >> bgr;
cv::putText(bgr,fps_text,
        cv::Point(40, 40),
        cv::FONT_HERSHEY_SIMPLEX,1,
        cv::Scalar(0,255,0),2);

    for (int y = 0; y < height; ++y) {
      for (int x = 0; x < width; ++x) {
          cv::Vec3b pixel = bgr.at<cv::Vec3b>(y, x);
          data[(y * width + x) * 3 + 0] = pixel[2]; // Red
          data[(y * width + x) * 3 + 1] = pixel[1]; // Green
          data[(y * width + x) * 3 + 2] = pixel[0]; // Blue
        }
    }

Send the data to VENC and encode it in H264 format
```
RK_MPI_VENC_SendFrame(vpssChn, &h264_Frame,-1);
```

The RTSP push flow is performed

RK_MPI_VENC_GetStream(vencChn, &stFrame, -1);
RK_MPI_MB_Handle2VirAddr(stFrame.pstPack->pMbBlk);
rtsp_tx_video(g_rtsp_session, (uint8_t *)pData, stFrame.pstPack->u32Len,
        stFrame.pstPack->u64PTS);
rtsp_do_event(g_rtsplive);

Resource release

// Destory MB
RK_MPI_MB_ReleaseMB(src_Blk);
// Destory Pool
RK_MPI_MB_DestroyPool(src_Pool);
// VENC
RK_MPI_VENC_StopRecvFrame(ChnId);
RK_MPI_VENC_DestroyChn(ChnId);
// RKRTSP
rtsp_del_demo(g_rtsplive);
// RKMPI
RK_MPI_SYS_Exit();

7 RKNN inference combined with RTSP push flow Example

In order to maximize the frame rate of RTSP streaming, the example uses VI components to implement camera image capture. In the labeling of RKNN inference results, two ways are used to achieve:

Upload the marked image to the VENC component for encoding and transmission
After the image is uploaded to the VENC component, the RGN module is used to annotate the result in the form of OSD.

7.1 Draw RKNN inference results based on opencv-mobile

Running effect

The VI component of Luckfox-pico pro is used to capture 720 × 480 resolution images, which are converted into pixel format by VPSS component and scaled by opencv-mobile for RKNN model reasoning. The obtained result is marked on the original data frame and transmitted to the VENC component for encoding transmission.

Notice: Luckfox-pico mini/plus cannot run yolov5 object detection routines due to memory limitations.

Using the retinaface face detection model, the frame rate can reach about 11 frames.
Using the yolov5 object detection model, the frame rate can reach about 7 frames.

Implementation process

RKRTSP, RKMPI, RKAIQ, and RKNN initialization

// RKNN Init
init_rknn_model(model_path, &rknn_app_ctx);

// RKAIQ Init
SAMPLE_COMM_ISP_Init(CamID, hdr_mode, multi_sensor, iq_dir);
SAMPLE_COMM_ISP_Run(CamID);

// RTSP Init
g_rtsplive = create_rtsp_demo(554);
g_rtsp_session = rtsp_new_session(g_rtsplive, "/live/0");
rtsp_set_video(g_rtsp_session, RTSP_CODEC_ID_VIDEO_H264, NULL, 0);
rtsp_sync_video_ts(g_rtsp_session, rtsp_get_reltime(), rtsp_get_ntptime());

// RKMPI Init
RK_MPI_SYS_Init();

Initialization of VI, VPSS and VENC components

// VI Init
vi_dev_init();
vi_chn_init(0, width, height);

// VPSS Init
vpss_init(0, width, height);

// VENC Init
RK_CODEC_ID_E enCodecType = RK_VIDEO_ID_AVC;
venc_init(0, width, height, enCodecType);

The pixel format of VI channel shall be consistent with the pixel format of VPSS group

vi_chn_attr.enPixelFormat = RK_FMT_YUV420SP; // VI Chn

stGrpVpssAttr.enPixelFormat = RK_FMT_YUV420SP; // VPSS Group

To facilitate the inferencing of data into the RKNN model, the pixel format is converted to RGB888 using the VPSS component.
```
stVpssChnAttr.enPixelFormat = RK_FMT_RGB888;
```
VENC sets the relevant properties for H264 encoding
```
stAttr.stRcAttr.enRcMode = VENC_RC_MODE_H264CBR;
stAttr.stRcAttr.stH264Cbr.u32BitRate = BitRate;
stAttr.stRcAttr.stH264Cbr.u32Gop = GOP;
```
- Notice: The GOP used in the routine is 1. Set all data frames as I frames. Use VLC software on the PC side to set the cache time not to be less than 500 ms. You can try to modify it for optimization.

Bind VI channel to VPSS

stSrcChn.enModId = RK_ID_VI;
stSrcChn.s32DevId = 0;
stSrcChn.s32ChnId = 0;

stvpssChn.enModId = RK_ID_VPSS;
stvpssChn.s32DevId = 0;
stvpssChn.s32ChnId = 0;
s32Ret = RK_MPI_SYS_Bind(&stSrcChn, &stvpssChn);

Acquire a VPSS data frame and a virtual address of a corresponding memory buffer block

RK_MPI_VPSS_GetChnFrame(DevId, ChnId, &stVpssFrame, -1);
void *data = RK_MPI_MB_Handle2VirAddr(stVpssFrame.stVFrame.pMbBlk);

After reading the VPSS data frame, opencv-mobile processes the image into the required size of the model inference
```
cv::Mat frame(height,width,CV_8UC3, data);          
cv::Mat model_input;
cv::resize(frame, model_input, cv::Size(model_width,model_height), 0, 0, cv::INTER_LINEAR);
```
- Notice: If the model is obtained by using the feature value which requires a high proportion of the original image, the image can not be simply scaled, and the operation of scaling and filling the black edge is needed to process the image.

And label that obtained reason result on the VPSS data frame image.

memcpy(rknn_app_ctx.input_mems[0]->virt_addr, model_input.data, 640*640*3);
inference_rknn_model(&rknn_app_ctx, &od_results);

Send the data to VENC and encode it in H264 format
```
RK_MPI_VENC_SendFrame(vpssChn, &stVpssFrame,-1);
```

The RTSP push flow is performed

RK_MPI_VENC_GetStream(vencChn, &stFrame, -1);
RK_MPI_MB_Handle2VirAddr(stFrame.pstPack->pMbBlk);
rtsp_tx_video(g_rtsp_session, (uint8_t *)pData, stFrame.pstPack->u32Len,
        stFrame.pstPack->u64PTS);
rtsp_do_event(g_rtsplive);

Resource release

RK_MPI_SYS_UnBind(&stSrcChn, &stvpssChn);

RK_MPI_VI_DisableChn(DevId, ChnId);
RK_MPI_VI_DisableDev(DevId);

RK_MPI_VPSS_StopGrp(GrpId);
RK_MPI_VPSS_DestroyGrp(Grpid);

RK_MPI_VENC_StopRecvFrame(ChnId);
RK_MPI_VENC_DestroyChn(ChnId);
// RKRTSP
rtsp_del_demo(g_rtsplive);
// RKIRQ
SAMPLE_COMM_ISP_Stop(CamId);
// RKMPI
RK_MPI_SYS_Exit();
// RKNN
release_rknn_model(&rknn_app_ctx);  

7.2 Draw RKNN inference results based on OSD rendering

Running effect

The VI component of Luckfox-pico pro is used to capture 720 × 480 resolution images, two output channels are set, one channel is bound to the VPSS group, one channel is bound to the VENC channel, two threads are opened, and one thread is responsible for RTSP pushing. One thread runs the RKNN model inference and displays the inference result on the VENC code stream in the form of OSD.

Implementation process

RKRTSP, RKMPI, RKAIQ, and RKNN initialization

// RKNN Init
init_rknn_model(model_path, &rknn_app_ctx);

// RKAIQ Init
SAMPLE_COMM_ISP_Init(CamID, hdr_mode, multi_sensor, iq_dir);
SAMPLE_COMM_ISP_Run(CamID);

// RTSP Init
g_rtsplive = create_rtsp_demo(554);
g_rtsp_session = rtsp_new_session(g_rtsplive, "/live/0");
rtsp_set_video(g_rtsp_session, RTSP_CODEC_ID_VIDEO_H264, NULL, 0);
rtsp_sync_video_ts(g_rtsp_session, rtsp_get_reltime(), rtsp_get_ntptime());

// RKMPI Init
RK_MPI_SYS_Init();

Initialization of VI, VPSS and VENC components

// VI Init
vi_dev_init();
vi_chn_init(0, width, height);

// VPSS Init
vpss_init(0, width, height);

// VENC Init
RK_CODEC_ID_E enCodecType = RK_VIDEO_ID_AVC;
venc_init(0, width, height, enCodecType);

The pixel format of VI channel shall be consistent with the bound receiver.

vi_chn_attr.enPixelFormat = RK_FMT_YUV420SP;              // VI Chn

stGrpVpssAttr.enPixelFormat = RK_FMT_YUV420SP;            // VPSS Group

stChnVencAttr.stVencAttr.enPixelFormat = RK_FMT_YUV420SP; // VENC Chn

To facilitate the inferencing of data into the RKNN model, the pixel format is converted to RGB888 using the VPSS component
```
stVpssChnAttr.enPixelFormat = RK_FMT_RGB888;
```
VENC sets the relevant properties for H264 encoding
```
stAttr.stRcAttr.enRcMode = VENC_RC_MODE_H264CBR;
stAttr.stRcAttr.stH264Cbr.u32BitRate = BitRate;
stAttr.stRcAttr.stH264Cbr.u32Gop = GOP;
```
- Notice: The GOP used in the routine is 1. Set all data frames as I frames. Use VLC software on the PC side to set the cache time not to be less than 500 ms. You can try to modify it for optimization.

Channel binding

VI channel 0 is bound to VENC channel

stSrcChn.enModId = RK_ID_VI;
stSrcChn.s32DevId = 0;
stSrcChn.s32ChnId = 0;
  
stvencChn.enModId = RK_ID_VENC;
stvencChn.s32DevId = 0;
stvencChn.s32ChnId = 0;
s32Ret = RK_MPI_SYS_Bind(&stSrcChn, &stvencChn);

VI Channel 1 is bound to a VPSS group

stSrcChn.enModId = RK_ID_VI;
stSrcChn.s32DevId = 0;
stSrcChn.s32ChnId = 1;

stvpssChn.enModId = RK_ID_VPSS;
stvpssChn.s32DevId = 0;
stvpssChn.s32ChnId = 0;
s32Ret = RK_MPI_SYS_Bind(&stSrcChn, &stvpssChn);

The RKNN inference thread
- Acquire a VPSS data frame and a virtual address of a corresponding memory buffer block
```
RK_MPI_VPSS_GetChnFrame(DevId, ChnId, &stVpssFrame, -1);
void *data = RK_MPI_MB_Handle2VirAddr(stVpssFrame.stVFrame.pMbBlk);
```
- After reading the VPSS data frame image, opencv-mobile processes the image into the required size of the model inference.
```
cv::Mat frame(height,width,CV_8UC3, data);          
cv::Mat model_input;  
cv::resize(frame, model_input, cv::Size(model_width,model_height), 0, 0, cv::INTER_LINEAR);
```
  - Notice: If the model is obtained by using the feature value which requires a high proportion of the original image, the image can not be simply scaled, and the operation of scaling and filling the black edge is needed to process the image.
- Obtaining the reasoning result and displaying the result on the VENC code stream in the form of OSD
```
memcpy(rknn_app_ctx.input_mems[0]->virt_addr, model_input.data, 640*640*3);
inference_rknn_model(&rknn_app_ctx, &od_results);
```
  - Notice: The RGN module displays the OSD on the code stream in the form of overlay, which occupies a large amount of memory, and only one face is annotated in the routine.

The push thread performs RTSP push

RK_MPI_VENC_GetStream(vencChn, &stFrame, -1);
RK_MPI_MB_Handle2VirAddr(stFrame.pstPack->pMbBlk);
rtsp_tx_video(g_rtsp_session, (uint8_t *)pData, stFrame.pstPack->u32Len,
        stFrame.pstPack->u64PTS);
rtsp_do_event(g_rtsplive);

Resource release

RK_MPI_SYS_UnBind(&stSrcChn0, &stvencChn);
RK_MPI_SYS_UnBind(&stSrcChn1, &stvpssChn);

RK_MPI_VI_DisableChn(DevId, ChnId0);
RK_MPI_VI_DisableChn(DevId, ChnId1);
RK_MPI_VI_DisableDev(DevId);

RK_MPI_VPSS_StopGrp(GrpId);
RK_MPI_VPSS_DestroyGrp(Grpid);

RK_MPI_VENC_StopRecvFrame(ChnId);
RK_MPI_VENC_DestroyChn(ChnId);
// RKRTSP
rtsp_del_demo(g_rtsplive);
// RKIRQ
SAMPLE_COMM_ISP_Stop(CamId);
// RKMPI
RK_MPI_SYS_Exit();
// RKNN
release_rknn_model(&rknn_app_ctx);  

8 Compile and run

8.1 Compilation (Run on PC)

Clone the git repository

git clone https://github.com/LuckfoxTECH/luckfox_pico_rkmpi_example.git

Set environment variables
```
export LUCKFOX_SDK_PATH=< luckfox-pico Sdk 地址>
```
Note: Use the absolute path.

Run ./build.sh and select the example to compile

1) luckfox_pico_rtsp_opencv
2) luckfox_pico_rtsp_opencv_capture
3) luckfox_pico_rtsp_retinaface
4) luckfox_pico_rtsp_retinaface_osd
5) luckfox_pico_rtsp_yolov5
Enter your choice [1-5]:

8.2 Running (Run on the Luckfox Pico board)

After compilation, the corresponding deployment folders will be generated in the luckfox_pico_rkmpi_example/install directory

luckfox_pico_rtsp_opencv_demo  
luckfox_pico_rtsp_opencv_capture_demo  
luckfox_pico_rtsp_retinaface_demo
luckfox_pico_rtsp_retinaface_osd_demo
luckfox_pico_rtsp_yolov5_demo 

Upload the generated deployment folders to the Luckfox Pico board (using adb, ssh, etc.), navigate to the folder on the board and run the demo
```
# Run on the Luckfox Pico board. <Demo Target> is the executable in the deployment folder
chmod a+x <Demo Target>
./<Demo Target>
```
Use VLC to open the network stream rtsp://172.32.0.93/live/0 (modify the IP address according to your actual setup to retrieve the image).

Note

Before running the demo, execute RkLunch-stop.sh to stop the background program rkicp, which starts by default on Luckfox Pico and occupies the camera.
Since the RV1103 has limited system resources, reduce the video capture resolution if it cannot run normally.

RKMPI Instance User Guide

1 Introduction​

2 Support platform​

3. Environment construction​

4 RKM PI library usage​

4.1 System overview​

System framework ​

IPC Application Framework ​

4.2 Memory management​

Memory Pool Operation Mode​

4.3 Video Input VI​

VI Internal Data Processing​

VI function usage Guide ​

4.4 Video Processing Subsystem VPSS​

VPSS context​

VPSS function usage Guide ​

4.5 Video Encoding VENC​

VENC internal data processing​

VENC function usage Guide ​

5 The RKRTSP library usage Guide​

5.1 Basic use of the RKRTSP library​

6 Example of frame number of opencv-mobile annotation pushing​

6.1 Image frame acquisition​

6.2. Example of RTSP streaming based on VI image capture​

Running effect​

Implementation process ​

6.3 Example of RTSP streaming based on opencv-mobile capture​

Running effect​

Implementation process ​

7 RKNN inference combined with RTSP push flow Example​

7.1 Draw RKNN inference results based on opencv-mobile​

Running effect​

Implementation process ​

7.2 Draw RKNN inference results based on OSD rendering​

Running effect​

Implementation process ​

8 Compile and run​

8.1 Compilation (Run on PC)​

8.2 Running (Run on the Luckfox Pico board)​

Note​

1 Introduction

2 Support platform

3. Environment construction

4 RKM PI library usage

4.1 System overview

System framework

IPC Application Framework

4.2 Memory management

Memory Pool Operation Mode

4.3 Video Input VI

VI Internal Data Processing

VI function usage Guide

4.4 Video Processing Subsystem VPSS

VPSS context

VPSS function usage Guide

4.5 Video Encoding VENC

VENC internal data processing

VENC function usage Guide

5 The RKRTSP library usage Guide

5.1 Basic use of the RKRTSP library

6 Example of frame number of opencv-mobile annotation pushing

6.1 Image frame acquisition

6.2. Example of RTSP streaming based on VI image capture

Running effect

Implementation process

6.3 Example of RTSP streaming based on opencv-mobile capture

Running effect

Implementation process

7 RKNN inference combined with RTSP push flow Example

7.1 Draw RKNN inference results based on opencv-mobile

Running effect

Implementation process

7.2 Draw RKNN inference results based on OSD rendering

Running effect

Implementation process

8 Compile and run

8.1 Compilation (Run on PC)

8.2 Running (Run on the Luckfox Pico board)

Note