Video capture

Browse source code on GitHub

Description

This sample application uses the Video to capture frames from a video capture device then uses the Display to display them onto an LCD screen (if any).

Requirements

This sample needs a video capture device (e.g. a camera) but it is not mandatory. Supported boards and camera modules include:

Also Arduino Nicla Vision can be used in this sample as capture device, in that case The user can transfer the captured frames through on board USB.

Wiring

On MIMXRT1064-EVK, the MT9M114 camera module should be plugged in the J35 camera connector. A USB cable should be connected from a host to the micro USB debug connector (J41) in order to get console output via the freelink interface.

On MIMXRT1170-EVK, the OV5640 camera module should be plugged into the J2 camera connector. A USB cable should be connected from a host to the micro USB debug connector (J11) in order to get console output via the daplink interface.

On STM32H7B3I Discovery kit, connect the ST B-CAMS-OMV-MB1683 shield to the board on CN7 connector. A USB cable should be connected from a host to the micro USB connector in order to get console output.

On STM32N6570-DK, connect the ST B-CAMS-IMX-MB1854 camera module to the J4 CSI connector. A USB cable should be connected from a host to both USB-C ports for power, flashing and console output.

On the XIAO ESP32S3, connect the “sense” camera module and the Seeed Studio XIAO Round Display to their respective connectors.

For Arduino Nicla Vision there is no extra wiring required.

Building and Running

For MIMXRT1064-EVK, build this sample application with the following commands:

west build -b mimxrt1064_evk --shield dvp_fpc24_mt9m114 --shield rk043fn66hs_ctg samples/subsys/video/capture

For MIMXRT1170-EVK, build this sample application with the following commands:

west build -b mimxrt1170_evk/mimxrt1176/cm7 --shield nxp_btb44_ov5640 --shield rk055hdmipi4ma0 samples/subsys/video/capture

For Arduino Nicla Vision, build this sample application with the following commands:

west build -b arduino_nicla_vision/stm32h747xx/m7 samples/subsys/video/capture

For FRDM-MCXN947, build this sample application with the following commands, using the DVP 20-pin OV7670 Camera Module and NXP LCD_PAR_S035 TFT LCD Module connected to the board:

west build -b frdm_mcxn947/mcxn947/cpu0 --shield dvp_20pin_ov7670 --shield lcd_par_s035_8080 samples/subsys/video/capture

For STM32H7B3I Discovery kit, build this sample application with the following commands, using the ST B-CAMS-OMV-MB1683 shield with a compatible camera module:

west build -b stm32h7b3i_dk --shield st_b_cams_omv_mb1683 samples/subsys/video/capture

For STM32N6570-DK, build this sample application with the following commands, using the ST B-CAMS-IMX-MB1854 camera module:

west build -b stm32n6570_dk --shield st_b_cams_imx_mb1854 samples/subsys/video/capture

For XIAO ESP32S3, build this sample application with the following commands, using the Seeed Studio XIAO Round Display shield:

west build -b xiao_esp32s3/esp32s3/procpu/sense --shield seeed_xiao_round_display samples/subsys/video/capture

For testing purpose and without the need of any real video capture and/or display hardwares, a video software pattern generator is supported by the above build commands without specifying the shields, and using Video Software Generator Snippet (video-sw-generator):

west build -b native_sim/native/64 -S video-sw-generator samples/subsys/video/capture

Still without any hardware, Video Native FIFO Snippet (video-native-fifo) feeds the sample with frames produced on the host instead of generated on the device, which allows using a real webcam, a video file, or one of the ffmpeg test sources:

west build -b native_sim/native/64 -S video-native-fifo samples/subsys/video/capture

The driver defaults to 320x240 RGB565, which is also the geometry and pixel format of the native_sim SDL display, so the frames are shown in the SDL window without any further configuration. Create the FIFO and start the application:

$ mkfifo /tmp/zephyr-cam.fifo
$ build/zephyr/zephyr.exe --video-fifo=/tmp/zephyr-cam.fifo

Then, from another terminal, feed it a test pattern:

$ ffmpeg -re -f lavfi -i testsrc2=size=320x240:rate=10 \
    -pix_fmt rgb565le -f rawvideo -y /tmp/zephyr-cam.fifo

Replace the ffmpeg input with -f v4l2 -i /dev/video0 to capture from a real webcam, or with -i <file> to replay a video file, and scale it to 320x240: see the driver documentation for a webcam example.

For controlling the camera device using shell commands instead of continuously capturing the data, append -DCONFIG_VIDEO_SHELL=y to the build command:

west build -b mimxrt1064_evk --shield dvp_fpc24_mt9m114 --shield rk043fn66hs_ctg samples/subsys/video/capture -- -DCONFIG_VIDEO_SHELL=y

For STM32H7B3I Discovery kit with shell commands:

west build -b stm32h7b3i_dk --shield st_b_cams_omv_mb1683 samples/subsys/video/capture -- -DCONFIG_VIDEO_SHELL=y

Sample Output

 Video device: csi@402bc000
 - Capabilities:
   RGBP width [480; 480; 0] height [272; 272; 0]
   YUYV width [480; 480; 0] height [272; 272; 0]
   RGBP width [640; 640; 0] height [480; 480; 0]
   YUYV width [640; 640; 0] height [480; 480; 0]
   RGBP width [1280; 1280; 0] height [720; 720; 0]
   YUYV width [1280; 1280; 0] height [720; 720; 0]
 - Default format: RGBP 480x272

 Display device: display-controller@402b8000
 - Capabilities:
   x_resolution = 480, y_resolution = 272, supported_pixel_formats = 40
   current_pixel_format = 32, current_orientation = 0

 Capture started
 Got frame 0! size: 261120; timestamp 249 ms (delta 249 ms)
 Got frame 1! size: 261120; timestamp 282 ms (delta 33 ms)
 Got frame 2! size: 261120; timestamp 316 ms (delta 34 ms)
 Got frame 3! size: 261120; timestamp 350 ms (delta 34 ms)
 Got frame 4! size: 261120; timestamp 384 ms (delta 34 ms)
 Got frame 5! size: 261120; timestamp 418 ms (delta 34 ms)
 Got frame 6! size: 261120; timestamp 451 ms (delta 33 ms)

<repeats endlessly>

If using the shell, the capture will not start unless the Kconfig option VIDEO_SHELL_AND_CAPTURE is set. In both cases, the shell can be used to change parameters on the fly:

uart:~$ video --help
video - Video driver commands
Subcommands:
  start    : Start a video device and its sources
             Usage: start <device>
  stop     : Stop a video device and its sources
             Usage: stop <device>
  capture  : Capture a given number of buffers from a device
             Usage: capture <device> <num-buffers>
  format   : Query or set the video format of a device
             Usage: format <device> <dir> [<fourcc> <width>x<height>]
  frmival  : Query or set the video frame rate/interval of a device
             Usage: frmival <device> [<n>fps|<n>ms|<n>us]
  ctrl     : Query or set video controls of a device
             Usage: ctrl <device> [<ctrl> <value>]
uart:~$

References

See also

Video Drivers