Skip to main content

CSI Camera

Luckfox Pico Zero is compatible with both the IMX415-98 IR-CUT Camera and the ISG1321 Camera, supporting 8MP and 1.3MP resolutions respectively, suitable for various application scenarios.

  • ISG1321 Camera: Features the ISG1321 image sensor, a 1.38MP monochrome global shutter camera module with high sensitivity, high signal-to-noise ratio, and excellent low-light performance. It comes with a 95.5° wide-angle distortion-free lens.
  • IMX415-98 IR-CUT Camera: Equipped with the IMX415 image sensor, this 8MP camera module has an F2.0 aperture and a 98° wide-angle lens, offering high signal-to-noise ratio and low-light capability.

1. Hardware Connection​

Luckfox Pico Zero: The metal side of the ribbon cable should face the back side of the board.
xxxxxx

2. Checking the Camera and IP Address​

Note: The orientation of the text on the blue ribbon cable from the factory may vary. Regardless of the text direction, just ensure that the metal side of the camera ribbon cable faces the RV1106. Please verify that the connection is proper before powering on.

  1. Inspect the camera. If the camera is successfully recognized, an "rkipc.ini" file will be generated.

    # ls /userdata/
    ethaddr.txt lost+found image.bmp video0 video2 rkipc . ini video1

  2. Obtain the device's IP address. The default USB static IP address for Luckfox Pico is 172.32.0.93.

    # ifconfig
    lo        Link encap:Local Loopback
              inet addr:127.0.0.1  Mask:255.0.0.0
              UP LOOPBACK RUNNING  MTU:65536  Metric:1
              RX packets:0 errors:0 dropped:0 overruns:0 frame:0
              TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

    usb0      Link encap:Ethernet  HWaddr 9E:49:D7:C8:F4:A3
              inet addr:172.32.0.93  Bcast:172.32.255.255  Mask:255.255.0.0
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:763 errors:0 dropped:116 overruns:0 frame:0
              TX packets:4 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:149160 (145.6 KiB)  TX bytes:1544 (1.5 KiB)

  3. For Luckfox Pico Pro/Max/Ultra/Pi/Zero, due to the presence of an Ethernet port, you can also view the wired IP address.

    # ifconfig
    eth0      Link encap:Ethernet  HWaddr 82:D8:B7:D8:94:5A
              inet addr:192.168.10.86  Bcast:192.168.11.255  Mask:255.255.252.0
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:144 errors:0 dropped:0 overruns:0 frame:0
              TX packets:2 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:11210 (10.9 KiB)  TX bytes:684 (684.0 B)
              Interrupt:51

    lo        Link encap:Local Loopback
              inet addr:127.0.0.1  Mask:255.0.0.0
              UP LOOPBACK RUNNING  MTU:65536  Metric:1
              RX packets:0 errors:0 dropped:0 overruns:0 frame:0
              TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

    usb0      Link encap:Ethernet  HWaddr F2:84:8E:BA:A0:E8
              inet addr:172.32.0.93  Bcast:172.32.255.255  Mask:255.255.0.0
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:85 errors:0 dropped:39 overruns:0 frame:0
              TX packets:4 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:13126 (12.8 KiB)  TX bytes:1544 (1.5 KiB)

3. Configure the RNDIS Virtual Network Interface​

If you need to use a static IP for streaming, you must configure RNDIS to ensure that the virtual network adapter places the computer and the development board on the same local network.

  1. Windows Security Center -> Firewall and Network Protection -> Turn off the firewall.
    xxx

  2. Configure a static IP for the RNDIS network card. Open Settings -> Advanced Network Settings -> Change Adapter Options.

  3. The network card's name is usually similar to "Remote NDIS based Internet Sharing Device." Right-click and choose Properties.
    xxx

  4. Double-click on "Internet Protocol Version 4 (TCP/IPv4)." Set the IPv4 address to 172.32.0.100 to avoid conflicts with other devices.
    xxx
    xxx

4. Stream Using VLC Media Player​

  1. Download and install VLC media player.

  2. Open the VLC media player software. Navigate to Media -> Open Network Stream.
    xxx

  3. Enter the corresponding IP address: rtsp://172.32.0.93/live/0.
    xxx

  4. The Luckfox Pico Plus/Pro/Max version can also use the eth0 IP address for streaming.
    xxx

  5. Click Play to view the camera feed. You can rotate the lens for focusing adjustments.
    xxx

5. Optimize VLC Network Streaming​

By default, VLC software buffers 1 second (1000ms=1s) of video, and you can moderately reduce the buffer time to improve real-time performance. However, setting the latency too low may lead to packet loss or stuttering, so it is recommended to keep it not lower than 300ms.

  • The effect is as follows:
    xxx

6. Simulated UVC Device​

6.1 Fastboot Mode​

rndis.sh and usb_config.sh cannot be used simultaneously. After executing either script, a system reboot is required to switch between modes. Only one mode (UVC or RNDIS) can be activated per boot cycle.

RTSP Streaming​

  1. Flash the system image that supports fastboot mode. Log in to the board via the serial port.

  2. Configure network on the board:

    # Set static IP over USB
    rndis.sh

    # Use dynamic IP via Ethernet
    udhcpc -i eth0

  3. Activate the RTSP service:

    echo rtsp_start > /tmp/smart_door.ini

  4. Follow Section 4: "Using VLC Media Player for Streaming" to stream video.

uvc​

  1. Flash the system image that supports fastboot mode. Log in to the board via the serial port.

  2. Start UVC service:

    usb_config.sh 
    echo uvc_start > /tmp/smart_door.ini

  3. Open Device Manager on the host PC. A new UVC Camera device will appear under the "Cameras" category.
    xxx

  4. Use PotPlayer, press Ctrl + J to view the camera stream.

  5. To configure resolution, right-click in PotPlayer, then go to "Open" → "Device Settings".

6.2 Standard Boot Mode​

  1. In the luckfox-pico/project/cfg/BoardConfig_IPC/BoardConfig-*-.mk file, add the RK_APP_TYPE option and set it to UVC_TINY:

    export RK_APP_TYPE="RKIPC_RV1106 UVC_TINY"

  2. Stop the rkipc program and disable ADB service (ADB reconnection can interrupt UVC connection):

    RkLunch-stop.sh
    killall S99usb0config

  3. Start the UVC service:

    usb_config.sh 
    rk_mpi_uvc &

  4. Open PotPlayer, press Ctrl + J to view the camera feed.

  5. To set detailed resolution, right-click in PotPlayer and navigate to "Open" → "Device Settings".

7. Capturing Raw Images with V4L2​

[Note]

Since the image is not processed by the ISP (Image Signal Processor), the frames captured through the V4L2 interface may appear dark or greenish. If ISP processing is required, you can use the default RKIPC program or refer to the example programs in the RKMPI section of this manual.

7.1 V4L2 Introduction​

V4L2, short for Video for Linux 2, is a kernel framework on the Linux operating system designed to support video devices. It provides a set of APIs and driver interfaces that allow user-space applications to interact with various video devices such as cameras and TV cards. V4L2 is a part of the Linux kernel, dedicated to handling video input and output

7.2 V4L2 Framework​

Video devices corresponding to V4L2 have /dev/videoX as their device nodes. Video devices use high-frequency cameras or cameras as input sources. The Linux kernel drives such devices, receiving and processing the corresponding video informatio.
xxx

  1. User Space
    • Applications typically operate cameras using the libv4l library or can choose to write applications based on the character device /dev/videoX.
    • guvcview is commonly used for USB cameras, providing a user-friendly graphical interface suitable for desktop environments (currently not supported on Luckfox Pico).
    • yavta is typically used for CSI cameras, which are commonly employed in embedded systems. yavta is a lightweight command-line tool suitable for testing and debugging in a terminal environment (not enabled by default).
    • v4l2 utilities: The v4l-utils toolset mainly includes two commonly used tools, namely media-ctl and v4l2-ctl.
  2. Kernel Space
    • RKVICAP and RKISP drivers are mainly based on the v4l2/media framework. They implement hardware configuration, interrupt handling, control buffer rotation, and control subdevices (such as MIPI D-PHY and sensor) power-up and power-down functions. (For more detailed information, refer to the document Rockchip_Driver_Guide_VI_CN_v1.1.3.pdf)
    • MIPI D-PHY is a physical layer protocol defined by the MIPI Alliance, commonly used to connect image sensors and image processors.
    • Video Buffer 2 is a subsystem in the Linux kernel responsible for memory management and device I/O operations for video buffers in the Linux system. It is typically used for handling cameras and other video input devices.
    • Cameras transfer video data to the processing unit through D-PHY. During this process, vb2 is responsible for allocating and managing memory buffers used to store video frames.
    • CCI (Camera Control Interface): The camera control interface is used to send control commands from the image processor to the camera. It is mainly implemented through GPIO (power supply, chip select) and I2C (sending configuration commands to the sensor).
  3. Hardware
    • CSIC Controller: Its main function is to achieve high-speed data transfer between the image sensor and the image processor.
    • I2C Controller: Used for communication with the I2C block of the image sensor. This is typically used for configuring and controlling sensor parameters such as exposure time and gain.
    • GPIO Controller: Used for communication with the image sensor. It is usually used to provide power or select the sensor, possibly by controlling the sensor's power or chip select pins.
    • EHCI/OHCI: OHCI is mainly used for USB 1.1 devices, while EHCI is used to support USB 2.0 high-speed devices. The combination of these two provides the ability for the computer to simultaneously support different speed USB devices, providing a consistent software and hardware interface. Here, it refers to USB type cameras.
  4. External Devices
    • Camera Interfaces: CSI (2 x 2Lanes or 1 x 4Lanes 2.5Gbps/Lane), DVP (BT601 / BT656 / BT1120 pclk: 150MHz), and USB 2.0.

7.3 V4L2-utils Toolkit​

V4L2-utils is a set of utilities for testing and configuring V4L2 devices, providing command-line access to the V4L2 interface. The Luckfox SDK's Buildroot package has V4L2-utils enabled by default.

7.4 v4l2-ctl​

  1. List available video devices:

    v4l2-ctl --list-devices
    # v4l2-ctl --list-devices
    rkisp-statistics (platform: rkisp):
            /dev/video23
            /dev/video24

    rkcif-mipi-lvds (platform:rkcif):
            /dev/media2

    rkcif (platform:rkcif-mipi-lvds):
            /dev/video4
            /dev/video5
            /dev/video6
            /dev/video7
            /dev/video8
            /dev/video9
            /dev/video10
            /dev/video11
            /dev/video12
            /dev/video13
            /dev/video14

    rkisp_mainpath (platform:rkisp-vir0):
            /dev/video15
            /dev/video16
            /dev/video17
            /dev/video18
            /dev/video19
            /dev/video20
            /dev/video21
            /dev/video22
            /dev/media3

    USB Camera: USB Camera (usb-xhci-hcd.0.auto-1.1):
            /dev/video0
            /dev/video1
            /dev/media0

    USB 2.0 Camera: USB Camera (usb-xhci-hcd.0.auto-1.2):
            /dev/video2
            /dev/video3
            /dev/media1
    • rkisp_mainpath: CSI Camera
    • USB Camera: USB Camera

  2. List supported formats for a specific device:

    v4l2-ctl --device=/dev/video15 --list-formats-ext

    v4l2-ctl --device=/dev/video0 --list-formats-ext

  3. Display camera parameters:

    v4l2-ctl --device=/dev/video15 --list-ctrls

  4. Record video:

    # For CSI Camera (requires turning off the built-in RKIPC first)
    killall rkipc
    v4l2-ctl --device=/dev/video15 --set-fmt-video=width=640,height=480,pixelformat=NV12 --stream-mmap --stream-to=video50.yuv --stream-count=30

    # For USB Camera
    v4l2-ctl --device=/dev/video0 --set-fmt-video=width=640,height=480,pixelformat=YUYV --stream-mmap --stream-to=video100.yuv --stream-count=30
    • --device: Specifies the camera's corresponding device file.
    • --set-fmt-video: Specifies the width, height, and pixel format (identifying the pixel format).
    • --stream-mmap: Specifies the buffer type as mmap, i.e., buffers allocated by the kernel that are physically contiguous or mapped through iommu.
    • --stream-to: Specifies the file path for saving frame data.
    • --stream-skip: Specifies to discard the first 3 frames (not saved to the file).
    • --stream-count: Specifies the number of frames to capture, excluding those discarded with --stream-skip.

  5. If the error "The pixelformat 'UYVY' is invalid" appears, it indicates that the camera does not support this format. Use the second step's method to check the formats supported by the camera.

  6. If you see <<<<<<<<< 5.79 fpsit indicates successful video recording.
    xxx

7.5 media-ctl​

  1. Check how many media nodes are available:

    # ls /dev/media*
    /dev/media0  /dev/media1  /dev/media2

  2. View the topology structure:

    media-ctl -d /dev/media3 -p

  3. If the above command does not read the Entity of the CSI camera, use the following command to view all information:

    # v4l2-ctl --all -d /dev/video15
    Driver Info:
            Driver name      : rkisp_v7
            Card type        : rkisp_mainpath
            Bus info         : platform:rkisp-vir0
            Driver version   : 2.0.0
            Capabilities     : 0x84201000
                    Video Capture Multiplanar
                    Streaming
                    Extended Pix Format
                    Device Capabilities
            Device Caps      : 0x04201000
                    Video Capture Multiplanar
                    Streaming
                    Extended Pix Format
    Media Driver Info:
            Driver name      : rkisp-vir0
            Model            : rkisp0
            Serial           :
            Bus info         :
            Media version    : 5.10.110
            Hardware revision: 0x00000000 (0)
            Driver version   : 5.10.110
    Interface Info:
            ID               : 0x03000007
            Type             : V4L Video
    Entity Info:
            ID               : 0x00000006 (6)
            Name             : rkisp_mainpath
            Function         : V4L2 I/O
            Pad 0x01000009   : 0: Sink
              Link 0x0200000a: from remote pad 0x1000004 of entity 'rkisp-isp-subdev' (Unknown V4L2 Sub-Device): Data, Enabled
    Priority: 2
    Format Video Capture Multiplanar:
            Width/Height      : 2304/1296
            Pixel Format      : 'NV12' (Y/CbCr 4:2:0)
            Field             : None
            Number of planes  : 1
            Flags             :
            Colorspace        : Default
            Transfer Function : Default
            YCbCr/HSV Encoding: Default
            Quantization      : Full Range
            Plane 0           :
               Bytes per Line : 2304
               Size Image     : 4478976
    Selection Video Capture: crop, Left 0, Top 0, Width 2304, Height 1296, Flags:
    Selection Video Capture: crop_bounds, Left 0, Top 0, Width 2304, Height 1296, Flags:
    Selection Video Output: crop, Left 0, Top 0, Width 2304, Height 1296, Flags:
    Selection Video Output: crop_bounds, Left 0, Top 0, Width 2304, Height 1296, Flags:

    Image Processing Controls

  4. Check which device can output images:

    media-ctl -d /dev/media3 -e "rkisp_mainpath"

7.6 Displaying Videos​

The captured images or videos can be copied to PC Ubuntu and viewed using the ffplay tool. ffplay is a command-line tool in the FFmpeg toolkit used to play audio and video files. It is based on the FFmpeg library, supports a wide range of audio/video formats, and can play or preview multimedia content in real-time in the command line.

  1. Play a video:

    ffplay -video_size 640x480 -pixel_format nv12 -framerate 10 -i video50.yuv
    ffplay -video_size 640x480 -pixel_format yuyv422 -framerate 10 -i video100.yuv
    • -video_size size : Set the frame size.
    • -pixel_format format: Set the pixel format.
    • -framerate 10: Set the video frame rate to 10 frames per second.
      xxx
      xxx

  2. Convert video formats

    ffmpeg -f rawvideo -pixel_format yuyv422 -video_size 640x480 -framerate 30 -i video.yuv -c:v libx264 output.mp4