opencv-mobile

This article introduces "opencv-mobile," a compact version of the OpenCV library with only 1/10 of the official size, and its application on the LuckFox Pico platform. By utilizing the luckfox-pico MIPI CSI camera and rkaiq/rga hardware acceleration, opencv-mobile achieves access to the camera stream, ISP image adjustment, and hardware acceleration.

Original source:opencv-mobile now supports luckfox-pico MIPI CSI camera and rkaiq/rga hardware acceleration

GitHub repository: https://github.com/nihui/opencv-mobile

Quick Steps

Create a project folder:

mkdir opencv-mobile-test
cd opencv-mobile-test

Download the opencv-mobile luckfox-pico precompiled package and unzip it in the virtual machine:
opencv-mobile-4.8.1-luckfox-pico.zip
```
unzip opencv-mobile-4.8.1-luckfox-pico.zip
```

CMake configuration:

Create a file:

vi CMakeLists.txt

Add the following content and change <SDK Directory> to your own SDK path, such as /home/luckfox/luckfox-pico/

project(opencv-mobile-test)
cmake_minimum_required(VERSION 3.5)
set(CMAKE_CXX_STANDARD 11)

SET(CMAKE_C_COMPILER "<SDK Directory>/tools/linux/toolchain/arm-rockchip830-linux-uclibcgnueabihf/bin/arm-rockchip830-linux-uclibcgnueabihf-gcc")
SET(CMAKE_CXX_COMPILER "<SDK Directory>/tools/linux/toolchain/arm-rockchip830-linux-uclibcgnueabihf/bin/arm-rockchip830-linux-uclibcgnueabihf-g++")

set(OpenCV_DIR "${CMAKE_CURRENT_SOURCE_DIR}/opencv-mobile-4.8.1-luckfox-pico/lib/cmake/opencv4")
find_package(OpenCV REQUIRED)
include_directories(${OpenCV_INCLUDE_DIRS})

add_executable(opencv-mobile-test main.cpp)

target_link_libraries(opencv-mobile-test ${OpenCV_LIBS})

Source code

Create a file:

vi main.cpp

Add the following content:

#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>

#include <unistd.h>   // sleep()

int main()
{
    cv::VideoCapture cap;
    cap.set(cv::CAP_PROP_FRAME_WIDTH, 320);
    cap.set(cv::CAP_PROP_FRAME_HEIGHT, 240);
    cap.open(0);

    const int w = cap.get(cv::CAP_PROP_FRAME_WIDTH);
    const int h = cap.get(cv::CAP_PROP_FRAME_HEIGHT);
    fprintf(stderr, "%d x %d\n", w, h);

    cv::Mat bgr[9];
    for (int i = 0; i < 9; i++)
    {
        cap >> bgr[i];

        sleep(1);
    }

    cap.release();

    // combine into big image
    {
        cv::Mat out(h * 3, w * 3, CV_8UC3);
        bgr[0].copyTo(out(cv::Rect(0, 0, w, h)));
        bgr[1].copyTo(out(cv::Rect(w, 0, w, h)));
        bgr[2].copyTo(out(cv::Rect(w * 2, 0, w, h)));
        bgr[3].copyTo(out(cv::Rect(0, h, w, h)));
        bgr[4].copyTo(out(cv::Rect(w, h, w, h)));
        bgr[5].copyTo(out(cv::Rect(w * 2, h, w, h)));
        bgr[6].copyTo(out(cv::Rect(0, h * 2, w, h)));
        bgr[7].copyTo(out(cv::Rect(w, h * 2, w, h)));
        bgr[8].copyTo(out(cv::Rect(w * 2, h * 2, w, h)));

        cv::imwrite("out.jpg", out);
    }

    return 0;
}

Compilation:

Execute the following commands:

mkdir build
cd build
cmake ..
make

After successful compilation, you will get the executable file opencv-mobile-test

luckfox@luckfox:~/opencv-mobile-test/build$ ls
CMakeCache.txt  CMakeFiles  cmake_install.cmake  Makefile  opencv-mobile-test

Project directory structure:

opencv-mobile-test/             # Project root directory
├── build                       # Build output directory
├── CMakeLists.txt              # Project CMake configuration file
├── main.cpp                    # Project source code file
└── opencv-mobile-4.8.1-luckfox-pico/  # Directory for opencv-mobile library

Transfer files:

scp opencv-mobile-test root@development board ip address:/root

Run the program:

After the development board is connected to the camera, power it on and release the camera resources after startup:

killall rkipc

Run the program

# ./opencv-mobile-test
[19083.672366] stream_cif_mipi_id0: s_power 1, entity use_count 1
   devpath = /dev/video11
   driver = rkisp_v7
   card = rkisp_mainpath
   bus_info = platform:rkisp-vir0
   version = 20000
   capabilities = 84201000
   device_caps = 4201000
   fmt = UYVY 4:2:2  59565955
   fmt = Y/CbCr 4:2:2  3631564e
   fmt = Y/CrCb 4:2:2  3136564e
   fmt = Y/CrCb 4:2:0  3132564e
       size = 32 x 16  ~  2304 x 1296  (+8 +8)
   fmt = Y/CbCr 4:2:0  3231564e
   fmt = Y/CrCb 4:2:0 (N-C)  31324d4e
   fmt = Y/CbCr 4:2:0 (N-C)  32314d4e
rkaiq log level ff0
[19083.854380] stream_cif_mipi_id0: open video, entity use_countt 2
[19083.854602] stream_cif_mipi_id1: open video, entity use_countt 1
/dev/video11 does not support changing fps
rga_api version 1.9.1_[0]
[19083.869267] rkisp rkisp-vir0: first params buf queue
[19083.871403] rkisp_hw ffa00000.rkisp: set isp clk = 198000000Hz
[19083.871447] rkcif-mipi-lvds: sditf_reinit_mode, mode->rdbk_mode 0, mode->name rkisp-vir0, link_mode 1
[19083.873291] rkcif-mipi-lvds: strea320 x 240
m[0] start streaming
[19083.873461] rockchip-mipi-csi2 ffa20000.mipi-csi2: stream on, src_sd: 796325b9, sd_name:rockchip-csi2-dphy0
[19083.873476] rockchip-mipi-csi2 ffa20000.mipi-csi2: stream ON
[19083.873524] rockchip-csi2-dphy0: dphy0, data_rate_mbps 506
[19083.873554] rockchip-csi2-dphy csi2-dphy0: csi2_dphy_s_stream stream on:1, dphy0
[19083.873566] rockchip-csi2-dphy csi2-dphy0: csi2_dphy_s_stream stream on:1, dphy0
[19092.985051] rkcif-mipi-lvds: stream[0] start stopping, total mode 0x2, cur 0x2
[19093.008280] rockchip-mipi-csi2 ffa20000.mipi-csi2: stream off, src_sd: 796325b9, sd_name:rockchip-csi2-dphy0
[19093.008329] rMessageParser process loop exit!o
ckchip-mipi-csi2 ffa20000.mipi-csi2: stream OFF
[19093.010662] rockchip-csi2-dphy csi2-dphy0: csi2_dphy_s_stream_stop stream stop, dphy0
[19093.010721] rockchip-csmpp[3524]: mpp_buffer: mpi_buf_id = 136, dma_buf_fd = 5

mpp[3524]: mpp: mClinetFd 7 open ok attr.chan_id 0

i2-dphy csi2-dphy0: csi2_dphy_s_stream stream on:0, dphy0
[19093.010771] rockchip-csi2-dphy csi2-dphy0: csi2_dphy_s_stream stream on:0, dph# y0
[19093.010986] rkcif-mipi-lvds: stream[0] stopping finished, dma_en 0x0
[19093.024110] stream_cif_mipi_id1: close video, entity use_count 0
[19093.024207] stream_cif_mipi_id0: close video, entity use_count 1
[19093.024859] stream_cif_mipi_id0: s_power 0, entity use_count 0
[19093.043230] mpp_vcodec: 44: num_chan = 0
[19093.043341] mpp_vcodec: 103: chan_entry->handle f6d67a53, enc f6d67a53
[19093.044028] 755: MPP_ENC_SET_CFG in
[19093.044076] 524: MPP_ENC_SET_RC_CFG bps 0 [0 : 0] fps [1:1] gop 1

After successful execution, you will obtain a composite image named out.jpg

# ls
opencv-mobile-test out.jpg

The above are the detailed steps to help newcomers quickly use opencv-mobile on the LuckFox Pico platform. The following is the main content of the original article:

1. TL;DR

The opencv-mobile highgui module implements camera stream access based on v4l2.
Automatically loads the rkaiq library at runtime to achieve ISP image adjustment.
Dynamically loads the rga library at runtime to achieve YUV2BGR hardware acceleration.
No need to modify the code; the cv::VideoCapture is automatically supported, with the ability to set the resolution.
Tested and validated specifically for luckfox-pico; other platforms are not guaranteed.

Download the latest version of opencv-mobile luckfox-pico precompiled package:

opencv-mobile-4.8.1-luckfox-pico.zip

Extract it into the project directory, set the OpenCV_DIR path in CMake to find the package:

project(opencv-mobile-test)
cmake_minimum_required(VERSION 3.5)
set(CMAKE_CXX_STANDARD 11)

set(OpenCV_DIR "${CMAKE_CURRENT_SOURCE_DIR}/opencv-mobile-4.8.1-luckfox-pico/lib/cmake/opencv4")
find_package(OpenCV REQUIRED)

add_executable(opencv-mobile-test main.cpp)

target_link_libraries(opencv-mobile-test ${OpenCV_LIBS})

1.1 opencv-mobile

https://github.com/nihui/opencv-mobilegithub.com/nihui/opencv-mobile

opencv-mobile minimizes the compiled OpenCV library by adjusting compilation parameters and removing parts of the OpenCV source code.

It provides common OpenCV functionality such as image reading and writing, image processing, matrix operations, etc. It is version-synced with the upstream and has no third-party dependencies.

In most cases, it seamlessly replaces the official OpenCV library with only 1/10 of the size, making it particularly suitable for mobile and embedded environments with special size requirements.

1.2 luckfox-pico

LuckFox Pico series is a low-cost micro Linux development board based on the Rockchip RV1103/RV1106 chip. RV1103/RV1106 is an AI application processor SoC for vision processing, featuring a single-core ARM A7 and built-in NPU with computing power up to 0.5TOPs.

1.3 v4l2

V4L2, short for Video for Linux2, is a kernel driver designed specifically for video devices. In video development, manipulating V4L2 device nodes allows direct interaction with the camera.

1.4 rkaiq

RkAiq (Rockchip Automatical Image Quality) continuously obtains image statistics from ISP, combines IQ Tuning parameters, and uses a series of algorithms to calculate new hardware parameters for ISP, Sensor, etc. This process iterates continuously, ultimately achieving optimal image quality.

1.5 rga

RGA (Raster Graphic Acceleration Unit) is an independent 2D hardware accelerator used for accelerating common 2D graphics operations such as point/line drawing, image scaling, rotation, bitBlt, alpha blending, etc.

2. Some Implementation Details and Limitations

2.1 Loading specific versions of librkaiq/librga

https://github.com/LuckfoxTECH/luckfox-picogithub.com/LuckfoxTECH/luckfox-pico

From the luckfox-pico SDK, you can see that media/rga has a version number. Find the corresponding version source code on GitLab:

https://gitlab.com/rk3588_linux/linux/linux-rga/-/tree

Similar to the mpp library, rkaiq/rga often adds new features and modifies interface usage. It is necessary to use the specific version from the SDK to ensure compatibility.

To reduce compilation coupling, opencv-mobile uses runtime dlopen/dlsym to load librkaiq/librga. Even if the library is missing during compilation, it remains compatible and usable.

2.2 Whitelist

Optimized code has been validated on luckfox-pico max, but due to the incompatibility between rkaiq/rga versions, it cannot guarantee availability on other Rockchip platforms.

When loading aiq/rga libraries, an additional check is performed on /proc/device-tree/model to ensure that the device is a luckfox-pico device; otherwise, camera opening will fail.

2.3 Using the camera stream processed by ISP

There are many /dev/videoN devices in the system, and the device with the name rkisp_mainpath in /sys/class/video4linux/videoN/name is the one that has been optimized after ISP processing.

Directly reading from /dev/video0 provides Bayered format, which is raw data and cannot be used directly. Reading from /dev/video11 provides an image with correct color and brightness after ISP processing.

2.4 Resolution and Frame Rate

ISP processing has alignment requirements for resolution:

Width (w) should be a multiple of 16.
Height (h) should be a multiple of 2.

If the user specifies the output resolution, it will be aligned according to these rules and will try to maintain the aspect ratio by padding and then crop to the user's desired resolution when returning the image.

Resolution has a significant impact on frame rate, especially under 1080p, where the memcpy time of returning cv::Mat is apparent, and v4l2 cannot truly set the fps.

2.5 v4l2 Video Buffer

Many v4l2 tutorials online mention using multiple video buffers in a rotating strategy. This strategy is suitable for scenarios where the camera actively pushes images.

opencv's interface typically involves users actively pulling images. Using a single buffer ensures that users always pull the latest frame.

2.6 Behavior of rkaiq

The rkaiq interface is relatively opaque, and without being enabled, you get images with incorrect colors. However, as long as sysctl_start is called, the data obtained from v4l2 is in the processed state.

At the beginning, when the camera is opened, rkaiq needs a few seconds for image statistics, so the initial frames are quite dark as ISP automatic processing has not yet occurred.

2.7 rga Only Works on DMA Buffer

rga's input can come from v4l2 expbuf importbuffer_fd. However, testing rga output directly to cv::Mat data importbuffer_virtualaddr will hang.

Unfortunately, a new DMA buffer must be opened separately to store the rga conversion results. After the conversion is complete, it is then memcpy'd to cv::Mat. This DMA buffer can be reused.

3. Camera Testing and Saving Frame Images

Use cv::VideoCapture to open the camera and set the resolution to 320x240.
Capture one frame every 1 second.
Close the camera.
Concatenate the nine frames and save them as a single image (cv::imwrite benefits from rkmpp acceleration!).
The first two frames at the beginning are very dark because rkaiq is still gathering image information and hasn't had time to automatically process ISP.

#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>

#include <unistd.h>   // sleep()

int main()
{
    cv::VideoCapture cap;
    cap.set(cv::CAP_PROP_FRAME_WIDTH, 320);
    cap.set(cv::CAP_PROP_FRAME_HEIGHT, 240);
    cap.open(0);

    const int w = cap.get(cv::CAP_PROP_FRAME_WIDTH);
    const int h = cap.get(cv::CAP_PROP_FRAME_HEIGHT);
    fprintf(stderr, "%d x %d\n", w, h);

    cv::Mat bgr[9];
    for (int i = 0; i < 9; i++)
    {
        cap >> bgr[i];

        sleep(1);
    }

    cap.release();

    // combine into big image
    {
        cv::Mat out(h * 3, w * 3, CV_8UC3);
        bgr[0].copyTo(out(cv::Rect(0, 0, w, h)));
        bgr[1].copyTo(out(cv::Rect(w, 0, w, h)));
        bgr[2].copyTo(out(cv::Rect(w * 2, 0, w, h)));
        bgr[3].copyTo(out(cv::Rect(0, h, w, h)));
        bgr[4].copyTo(out(cv::Rect(w, h, w, h)));
        bgr[5].copyTo(out(cv::Rect(w * 2, h, w, h)));
        bgr[6].copyTo(out(cv::Rect(0, h * 2, w, h)));
        bgr[7].copyTo(out(cv::Rect(w, h * 2, w, h)));
        bgr[8].copyTo(out(cv::Rect(w * 2, h * 2, w, h)));

        cv::imwrite("out.jpg", out);
    }

    return 0;
}

After running, you get out.jpg:

Quick Steps​

1. TL;DR​

2. Some Implementation Details and Limitations​

3. Camera Testing and Saving Frame Images​

Quick Steps

1. TL;DR

2. Some Implementation Details and Limitations

3. Camera Testing and Saving Frame Images