CSI Camera

Luckfox Pico 、Luckfox Pico Mini A/B and Luckfox Pico Plus/Pro/Max are compatible with the SC3336 3MP Camera (A) module. This camera module utilizes the SC3336 photosensitive chip from SITWEI, supporting a resolution of 3 million pixels. It possesses high sensitivity, a high signal-to-noise ratio, and performs well in low-light conditions. This enables it to provide more delicate and true-to-life night vision full-color images, better adapting to changes in ambient lighting conditions.The current streaming is only available for the buildroot system.

Camera Inspection

First, connect the camera, ensuring that the connection is normal before powering on. The system will automatically recognize the camera; please be careful not to reverse the wiring. When connecting the camera to Luckfox Pico, Luckfox Pico Mini A/B, and LuckFox Pico Plus development boards, make sure that the blue side of the camera ribbon cable faces the chip on the development board.
Flash the latest image from the cloud drive, then log in to the development board and use the luckfox-config tool to enable the CSI interface.
```
luckfox-config
```
Inspect the camera. If the camera is successfully recognized, an "rkipc.ini" file will be generated.
```
# ls /userdata/
ethaddr.txt lost+found image.bmp video0 video2 rkipc . ini video1
```

Obtain the device's IP address. The default IP address for Luckfox Pico 、Luckfox Pico Mini A/B and Luckfox Pico Plus/Pro/Max is 172.32.0.93.

# ifconfig
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

usb0      Link encap:Ethernet  HWaddr 9E:49:D7:C8:F4:A3
          inet addr:172.32.0.93  Bcast:172.32.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:763 errors:0 dropped:116 overruns:0 frame:0
          TX packets:4 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:149160 (145.6 KiB)  TX bytes:1544 (1.5 KiB)

For Luckfox Pico Plus/Pro/Max, due to the presence of an Ethernet port, you can also view the wired IP address.

# ifconfig
eth0      Link encap:Ethernet  HWaddr 82:D8:B7:D8:94:5A
          inet addr:192.168.10.86  Bcast:192.168.11.255  Mask:255.255.252.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:144 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:11210 (10.9 KiB)  TX bytes:684 (684.0 B)
          Interrupt:51

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

usb0      Link encap:Ethernet  HWaddr F2:84:8E:BA:A0:E8
          inet addr:172.32.0.93  Bcast:172.32.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:85 errors:0 dropped:39 overruns:0 frame:0
          TX packets:4 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:13126 (12.8 KiB)  TX bytes:1544 (1.5 KiB)

Configure the RNDIS Virtual Network Interface

Windows Security Center -> Firewall and Network Protection -> Turn off the firewall.
Configure a static IP for the RNDIS network card. Open Settings -> Advanced Network Settings -> Change Adapter Options.
The network card's name is usually similar to "Remote NDIS based Internet Sharing Device." Right-click and choose Properties.
Double-click on "Internet Protocol Version 4 (TCP/IPv4)." Set the IPv4 address to 172.32.0.100 to avoid conflicts with other devices.

Stream Using VLC Media Player

Download and install VLC media player.
Open the VLC media player software. Navigate to Media -> Open Network Stream.
Enter the corresponding IP address: rtsp://172.32.0.93/live/0.
The Luckfox Pico Plus/Pro/Max version can also use the eth0 IP address for streaming.
Click Play to view the camera feed. You can rotate the lens for focusing adjustments.

Optimize VLC Network Streaming

By default, VLC software buffers 1 second (1000ms=1s) of video, and you can moderately reduce the buffer time to improve real-time performance. However, setting the latency too low may lead to packet loss or stuttering, so it is recommended to keep it not lower than 300ms.

The effect is as follows:

V4L2 Introduction

V4L2, short for Video for Linux 2, is a kernel framework on the Linux operating system designed to support video devices. It provides a set of APIs and driver interfaces that allow user-space applications to interact with various video devices such as cameras and TV cards. V4L2 is a part of the Linux kernel, dedicated to handling video input and output

V4L2 Framework

Video devices corresponding to V4L2 have /dev/videoX as their device nodes. Video devices use high-frequency cameras or cameras as input sources. The Linux kernel drives such devices, receiving and processing the corresponding video informatio

User Space
- Applications typically operate cameras using the libv4l library or can choose to write applications based on the character device /dev/videoX.
- guvcview is commonly used for USB cameras, providing a user-friendly graphical interface suitable for desktop environments (currently not supported on Luckfox Pico).
- yavta is typically used for CSI cameras, which are commonly employed in embedded systems. yavta is a lightweight command-line tool suitable for testing and debugging in a terminal environment (not enabled by default).
- v4l2 utilities: The v4l-utils toolset mainly includes two commonly used tools, namely media-ctl and v4l2-ctl.
Kernel Space
- RKVICAP and RKISP drivers are mainly based on the v4l2/media framework. They implement hardware configuration, interrupt handling, control buffer rotation, and control subdevices (such as MIPI D-PHY and sensor) power-up and power-down functions. (For more detailed information, refer to the document Rockchip_Driver_Guide_VI_CN_v1.1.3.pdf)
- MIPI D-PHY is a physical layer protocol defined by the MIPI Alliance, commonly used to connect image sensors and image processors.
- Video Buffer 2 is a subsystem in the Linux kernel responsible for memory management and device I/O operations for video buffers in the Linux system. It is typically used for handling cameras and other video input devices.
- Cameras transfer video data to the processing unit through D-PHY. During this process, vb2 is responsible for allocating and managing memory buffers used to store video frames.
- CCI (Camera Control Interface): The camera control interface is used to send control commands from the image processor to the camera. It is mainly implemented through GPIO (power supply, chip select) and I2C (sending configuration commands to the sensor).
Hardware
- CSIC Controller: Its main function is to achieve high-speed data transfer between the image sensor and the image processor.
- I2C Controller: Used for communication with the I2C block of the image sensor. This is typically used for configuring and controlling sensor parameters such as exposure time and gain.
- GPIO Controller: Used for communication with the image sensor. It is usually used to provide power or select the sensor, possibly by controlling the sensor's power or chip select pins.
- EHCI/OHCI: OHCI is mainly used for USB 1.1 devices, while EHCI is used to support USB 2.0 high-speed devices. The combination of these two provides the ability for the computer to simultaneously support different speed USB devices, providing a consistent software and hardware interface. Here, it refers to USB type cameras.
External Devices
- Camera Interfaces: CSI (2 x 2Lanes or 1 x 4Lanes 2.5Gbps/Lane), DVP (BT601 / BT656 / BT1120 pclk: 150MHz), and USB 2.0.

V4L2-utils Toolkit

V4L2-utils is a set of practical tools for testing and configuring V4L2 devices, providing command-line access to the V4L2 interface. The v4l2-utils package in the Luckfox SDK's buildroot package already includes tools, but it needs to be manually enabled. To enable it, add V4L2 to the buildroot directory and search with the keyword "libv4l2".

v4l2-ctl

List available video devices:

v4l2-ctl --list-devices

# v4l2-ctl --list-devices
rkisp-statistics (platform: rkisp):
        /dev/video23
        /dev/video24

rkcif-mipi-lvds (platform:rkcif):
        /dev/media2

rkcif (platform:rkcif-mipi-lvds):
        /dev/video4
        /dev/video5
        /dev/video6
        /dev/video7
        /dev/video8
        /dev/video9
        /dev/video10
        /dev/video11
        /dev/video12
        /dev/video13
        /dev/video14

rkisp_mainpath (platform:rkisp-vir0):
        /dev/video15
        /dev/video16
        /dev/video17
        /dev/video18
        /dev/video19
        /dev/video20
        /dev/video21
        /dev/video22
        /dev/media3

USB Camera: USB Camera (usb-xhci-hcd.0.auto-1.1):
        /dev/video0
        /dev/video1
        /dev/media0

USB 2.0 Camera: USB Camera (usb-xhci-hcd.0.auto-1.2):
        /dev/video2
        /dev/video3
        /dev/media1

rkisp_mainpath: CSI Camera
USB Camera: USB Camera

List supported formats for a specific device:

v4l2-ctl --device=/dev/video15 --list-formats-ext

v4l2-ctl --device=/dev/video0 --list-formats-ext

Display camera parameters:

v4l2-ctl --device=/dev/video15 --list-ctrls

Record video:
```
# For CSI Camera (requires turning off the built-in RKIPC first)
killall rkipc
v4l2-ctl --device=/dev/video15 --set-fmt-video=width=640,height=480,pixelformat=NV12 --stream-mmap --stream-to=video50.yuv --stream-count=60

# For USB Camera
v4l2-ctl --device=/dev/video0 --set-fmt-video=width=640,height=480,pixelformat=YUYV --stream-mmap --stream-to=video100.yuv --stream-count=100
```
- --device: Specifies the camera's corresponding device file.
- --set-fmt-video: Specifies the width, height, and pixel format (identifying the pixel format).
- --stream-mmap: Specifies the buffer type as mmap, i.e., buffers allocated by the kernel that are physically contiguous or mapped through iommu.
- --stream-to: Specifies the file path for saving frame data.
- --stream-skip: Specifies to discard the first 3 frames (not saved to the file).
- --stream-count: Specifies the number of frames to capture, excluding those discarded with --stream-skip.
If the error "The pixelformat 'UYVY' is invalid" appears, it indicates that the camera does not support this format. Use the second step's method to check the formats supported by the camera.
If you see "<<<<<<<<< 5.79 fps," it indicates successful video recording.

media-ctl

Check how many media nodes are available:

# ls /dev/media*
/dev/media0  /dev/media1  /dev/media2

View the topology structure:
```
media-ctl -d /dev/media3 -p
```

If the above command does not read the Entity of the CSI camera, use the following command to view all information:

# v4l2-ctl --all -d /dev/video15
Driver Info:
        Driver name      : rkisp_v7
        Card type        : rkisp_mainpath
        Bus info         : platform:rkisp-vir0
        Driver version   : 2.0.0
        Capabilities     : 0x84201000
                Video Capture Multiplanar
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps      : 0x04201000
                Video Capture Multiplanar
                Streaming
                Extended Pix Format
Media Driver Info:
        Driver name      : rkisp-vir0
        Model            : rkisp0
        Serial           :
        Bus info         :
        Media version    : 5.10.110
        Hardware revision: 0x00000000 (0)
        Driver version   : 5.10.110
Interface Info:
        ID               : 0x03000007
        Type             : V4L Video
Entity Info:
        ID               : 0x00000006 (6)
        Name             : rkisp_mainpath
        Function         : V4L2 I/O
        Pad 0x01000009   : 0: Sink
          Link 0x0200000a: from remote pad 0x1000004 of entity 'rkisp-isp-subdev' (Unknown V4L2 Sub-Device): Data, Enabled
Priority: 2
Format Video Capture Multiplanar:
        Width/Height      : 2304/1296
        Pixel Format      : 'NV12' (Y/CbCr 4:2:0)
        Field             : None
        Number of planes  : 1
        Flags             :
        Colorspace        : Default
        Transfer Function : Default
        YCbCr/HSV Encoding: Default
        Quantization      : Full Range
        Plane 0           :
           Bytes per Line : 2304
           Size Image     : 4478976
Selection Video Capture: crop, Left 0, Top 0, Width 2304, Height 1296, Flags:
Selection Video Capture: crop_bounds, Left 0, Top 0, Width 2304, Height 1296, Flags:
Selection Video Output: crop, Left 0, Top 0, Width 2304, Height 1296, Flags:
Selection Video Output: crop_bounds, Left 0, Top 0, Width 2304, Height 1296, Flags:

Image Processing Controls

Check which device can output images:

media-ctl -d /dev/media3 -e "rkisp_mainpath"

Displaying Videos

The captured images or videos can be copied to Ubuntu and viewed using the ffplay tool. ffplay is a command-line tool in the FFmpeg toolkit used to play audio and video files. It is based on the FFmpeg library, supports a wide range of audio/video formats, and can play or preview multimedia content in real-time in the command line.

Play a video:

ffplay -video_size 640x480 -pixel_format nv12 -framerate 10 -i video50.yuv
ffplay -video_size 640x480 -pixel_format yuyv422 -framerate 10 -i video100.yuv  

-video_size size : Set the frame size.
-pixel_format format: Set the pixel format.
-framerate 10: Set the video frame rate to 10 frames per second.

Convert video formats

ffmpeg -f rawvideo -pixel_format yuyv422 -video_size 640x480 -framerate 30 -i video.yuv -c:v libx264 output.mp4

CSI Camera

Camera Inspection​

Configure the RNDIS Virtual Network Interface​

Stream Using VLC Media Player​

Optimize VLC Network Streaming​

V4L2 Introduction​

V4L2 Framework​

V4L2-utils Toolkit​

v4l2-ctl​

media-ctl​

Displaying Videos​