Video data processing arrangement, process for managing video data, computer program and computer program product

State of the art

The invention relates to a video data processing arrangement according to claim 1. Furthermore, the invention relates to a process for managing video data, a computer program and a computer program product.

In normal computer systems, a computer program (user-space program) needs to share the bandwidth with the operating system and other programs, this will affect latency and bandwidth of the memory that will be available for such programs. Especially, in real-time video applications, the data rates are quite high, so that the reduced latency and bandwidth leads to a delay in the real-time video application.

Disclosure of the invention

The invention concerns a video data processing arrangement with the features of claim 1, a process for managing video data with the media data processing arrangement with the features of claim 12, a computer program with the features of claim 13 and a computer program product with the features of claim 14. Preferred or advantageous embodiments of the invention are disclosed by the dependent claims, the description and the figures as attached.

Subject-matter of the invention is a video data processing arrangement, which is adapted to process video data. The video data is especially realized as a video stream and/or a plurality of video-frames, which represent a video stream. The video data may be provided by a storage, especially storage medium, a cloud storage et cetera. Preferably, the video data is provided by a camera, especially her surveillance camera. It is especially preferred, that the video data is realized as real-time video data, whereby the video data processing arrangement is adapted to process the video data in real-time.

The video data processing arrangement is realized on a computer hardware, whereby the computer hardware may be embodied as a video camera, a computer, a cloud instance et cetera. Optionally, the video data processing arrangement comprises the video camera as an input device. Alternatively or additionally, the video data processing arrangement comprises an output for providing the processed video data and/or meta data based on the processed video data. The output can be connected to a storage for storing the processed video data and/or meta data, to a display for displaying the processed video data and/or meta data and/or a signaling device, for example a traffic light, for outputting signals on basis of the processed video data and/or meta data. The storage, display and/or signaling device may be a part of the video data processing arrangement.

The video data processing arrangement comprises a host operating system, whereby the host operating system is adapted for running on the computer hardware. Furthermore, the host operating system is adapted for managing hardware and/or software resources of the computer hardware. Especially, the host operating system may be adapted to provide common services for computer programs.

The video data processing arrangement further comprises a virtual machine, which is adapted for running on the host operating system as a virtual platform. Especially, the virtual machine uses the common services of the host operating system. The virtual machine may be a system virtual machine, preferably using a full virtualization, which is a substitute for a real machine. The system virtual machine optionally manages hardware components of the computer hardware, such as a CPU or a GPU. The virtual machine may be a process virtual machine, which optionally manages only common services of itself. Optionally, the virtual machine may manage virtual hardware of the computer hardware, like a virtual CPU or a virtual GPU. The video data processing arrangement comprises a (virtual) platform operating system for managing virtual and/or physical hardware resources of computer hardware and/or software resources of the virtual machine. In case of hardware resources, the platform operating system is adapted for managing for example storage means, data busses et cetera supporting the system virtual machine. In case of virtual hardware resources, the platform operating system is adapted for managing virtualized hardware, especially supporting the virtual machine.

Additionally, the platform operating system is adapted for providing common services for apps. The term "app" is a common abbreviation of “application”. Especially, an app is a computer program designed to carry out a specific task.

The video data processing arrangement comprises an app adapted to the platform operating system. The app may be realized as a so-called third-party app. The app uses at least a part of the common services for apps of the platform operating system.

The video data processing arrangement comprises a host input interface for receiving the video data. The host input interface may be realized as a connection to a camera, especially a surveillance camera, alternatively it may be realized as a connection to a network, a storage et cetera. Preferably, the host input interface may be part of the host operating system and/or of the computer hardware.

Furthermore, the video data processing arrangement comprises a host output interface for providing the received or preprocessed video data from the host input interface to the platform operating system and/or to the virtual machine.

The video data processing arrangement provides a machine input interface for receiving the video data from the host output interface.

According to invention, the Video data processing arrangement provides a transfer shared memory, whereby the host output interface is adapted for providing the video data in the transfer shared memory and the machine input interface is adapted for receiving the video data from the transfer shared memory. The transfer shared memory is realized as a virtual memory, which can preferably be handled by the platform operating system. Alternatively the transfer shared memory is realized as physical memory, forming for example a part of the computer hardware.

It is an idea of the invention, that storing and retrieving video data from memory means and especially copying the video data from one memory means to another memory means is time-consuming and burdensome for the bus system(s). Instead of providing an output memory means for the host output interface and an input memory means for the machine input interface, whereby the video data must be transferred from the output memory means to the input memory means, it is proposed to use the transfer shared memory as the host output memory means and the machine input memory means. As a consequence, one copying process of the video data as a time-consuming and resource consuming process is omitted, so that latency is reduced and the available bandwidth of the video data processing arrangement is more efficiently used. The invention is about how to use software techniques to efficiently transfer large quantities of video data on the video data processing arrangement and deliver it to the app, especially to 3rd party applications.

Processing of live and/or pre-recorded encoded and/or decoded especially high resolution, high frame-rate video data as the video data consumes a lot of hardware resources in terms of bandwidth and latency. By reducing the number of copying processes, the latency can be enhanced and the bandwidth can be saved.

It is especially preferred, that the system output interface and the app input interface are adapted to use the shared memory as own memory areas. For example, both interfaces may be adapted to have read and write rights.

Preferably, the app is adapted to run on the platform operating system. The app is especially embodied as a native app for the platform operating system.

In a preferred realization, the app is adapted to perform video analysis tasks and/or security video analysis tasks on the video data. For example the app may be adapted for object and/or person detection and/or tracking, detection of events like burglary, entering prohibited zones, detection of fire, detection of fleeing or persons, face detection etcetera. Other video analysis tasks concern retail store management, Crowd management, People counting, CO VID-19 detection, Queue management, Indoor detection, Motion detection, Object recognition, Anomaly detection, Crossline detection, Health & safety analysis, Heat mapping, Intrusion detection, Perimeta protection, Parking management, Traffic analysis, Demographic estimation, Dynamic masking of persons or objects, Loitering detection, Fall detection of persons, Face recognition, License plate recognition, Sabotage detection, Emotion estimation, Zone masking, Filtered search results, Filtered by Retail store management Retail, Out-of-shelf analytics provide near real-time information on product stock levels and can alert employees when restocking is required, Crowd Gathering Safety, Real Time Crowd Gathering, People Counting, PIN Terminal Blur, Mask Check, Container identification, Gun / active shooter detection, On camera VMS, Person re-identification, Smoke detection, Traffic incident detection, Vehicle identification etcetera.

The app may provide processed video data and/or meta data on the basis of the detected and/or tracked objects and/or persons and/or on basis of the other tasks as described above.

Preferably, the app is within a user-space or sandbox of the platform operating system, which is separated from the system-space of the platform operating system, so that non-authorized accesses from the user-space to the system-space are prevented. It is especially preferred, that the platform operating system is adapted to control the user-rights of the app in the user-space.

In a preferred embodiment, the platform operating system comprises a system output interface for providing the video data as the original video data as received or as preprocessed video data. Thus, it is possible that the platform operating system provides functions for preprocessing the video data.

The app comprises an app input interface for receiving the video data, which is the original video data or the preprocessed video data from the system output interface. The app is especially adapted to process the video data and to produce processed video data and/or meta data based on the video data and/or on the processed video data. Optionally, the processed video data and/or meta data is output by the output as described above. The output is preferably part of the app. Preferably, the app is within the user-space or sandbox of the platform operating system, which is separated from the system-space of the platform operating system, so that non-authorized accesses from the user-space to the system-space are prevented. It is especially preferred, that the platform operating system is adapted to control the user-rights of the app in the user-space.

Preferably, the platform operating system provides a shared memory, especially for temporarily storing video data, whereby the system output interface is adapted for providing, especially storing, the video data in the shared memory and the app input interface is adapted for receiving the video data from the shared memory. Especially, the platform operating system and the app are adapted to access the shared memory. It is especially preferred, that the system output interface and the app input interface are adapted to use the shared memory as own memory areas. For example, both interfaces may be adapted to have read and write rights. By using the transfer shared memory and the shared memory, a second copying process in the video pipeline is omitted, improving the efficiency of the video data processing arrangement.

It is further preferred, that the computer hardware comprises a CPU and a GPU, which is managed by the platform operating system, and/or the platform operating system uses a virtual CPU and a virtual GPU, whereby the platform operating system provides a further shared memory. The platform operating system is adapted so that the virtual or physical CPU and the virtual or physical GPU uses both the further shared memory for transporting the video data. The CPU provides a CPU output interface, the GPU provides a GPU input interface, whereby the CPU output interface is adapted for providing the video data in the further shared memory and the GPU input interface is adapted for receiving the video data from the shared memory. Especially, the platform operating system is adapted so that CPU and GPU both are able to access the further shared memory. Instead of providing an CPU output memory means for the CPU output interface and an GPU input memory means for the GPU input interface, whereby the video data must be transferred from the CPU output memory means to the GPU input memory means, it is proposed to use the further shared memory as the CPU output memory means and the GPU input memory means. As a consequence, one copying process of the video data as a time-consuming and resource consuming process is omitted, so that latency is reduced and the available bandwidth of the platform is more efficiently used. The further shared memory may be embodied as an internal system memory (known as DDR, SDRAM, volatile system memory).

In a further embodiment, it is proposed, that the platform operating system is adapted to convert and/or to subsample the video data in the YU V domain as pre- processed video data to decrease the data size per image. It is possible, that the YUV-video data as preprocessed video data is provided in the shared memory. Alternatively, only the Y-channel is provided in the shared memory and the UV- channel is cancelled, because for video-analytics the Y-channel is often sufficient enough to do for computer-vision like applications as described above. The conversion is preferably performed by the GPU. In this embodiment the data stream of the video data is significantly downsized, so that latency is reduced and the available bandwidth of the platform is more efficiently used.

In a preferred realization, the platform operating system is based on an Android™ open source platform and/or is realized as AZENA operating system. Both systems provide the possibility of sharing memory as needed for realization of the claimed subject matter. It is especially preferred, that the transfer shared memory, the shared memory and/or the further shared memory is realized as an Android™ HardwareBuffer.

The Android™ HardwareBuffer has a property to set the timestamp in a respective field. Preferably the video data processing arrangement, especially the platform operating system and/or the app, is adapted to set the timestamp in this field. With this embodiment, the video data processing arrangement is capable to replay prerecorded video data at a much higher frame-rate and still have frame accurate metadata. In the video data processing arrangement it is beneficial to have the timestamp field as a ‘tag’ to easily find back original frames of the video data. The app can use this tag to embed it properly in the metadata.

A further subject matter of the invention is a process for managing video data in the video data processing arrangement as described above, whereby the video data is written in the transfer shared memory as the system host interface by the host operating system and is read from the transfer shared memory as an machine input interface from the virtual machine and/or the platform operating system.

A further subject matter of the invention is a computer program having program code means in order to carry out all steps of the process according to claim 12, especially when the computer program is executed on the computer hardware as described above. Preferably, the computer program comprises the host and the platform operating system and the app.

A further subject matter of the invention is a computer program product having program code means that are stored on a computer readable storage medium. The computer program is realized as described above.

Further embodiments, features and advantages of the invention are described by the following description of preferred embodiments and the figures as attached. The figures show:

Figure 1 a block diagram of a video data processing arrangement as a first embodiment of the invention.

Figure 1 shows a block diagram of a video data processing arrangement 1 as an embodiment of the invention. The video data processing arrangement 1 comprises a computer hardware 2 as a hardware platform of the video data processing arrangement 1. The computer hardware may be embodied as a server or as a computer or another digital processing apparatus. The video data processing arrangement 1 comprises a host operating system 3 running on the computer hardware. The host operating system 3 manages the hardware of the computer hardware 2 and provides common services. .

The video data processing arrangement 1 comprises a virtual machine 21 with a platform operating system 4 running on the virtual machine 21. The virtual machine 21 is running on the host operating system 3 and preferably using the common services of the host operating system 3. The virtual machine 21 and/or the platform operating system 4 manages physical components of the computer hardware 2. Alternatively, the host operating system 3 provides components as virtual components and the platform operating system 4 is adapted to manage the virtual components.

The video data processing arrangement 1 provides a host input interface 22 for receiving video data. The video data may be provided as real-time video data for example from a surveillance camera 23, a network device 24 or a storage 25.

The video data processing arrangement 1 comprises a host output interface 26 for providing the video data as received or as preprocessed to the platform operating system 4 and/or the virtual machine 21.

The platform operating system 4 is adapted to provide common services for apps 5. The apps 5 may be realized as third-party user space applications. The platform operating system 4 and/or the virtual machine 21 comprises a machine input interface 27 as a system input interface for receiving video data capture provided by the host output interface 26.

The video data processing arrangement 1 comprises a transfer shared memory 28, whereby the host output interface 26 is adapted for providing the video data in the transfer shared memory 28 and the machine input interface 27 is adapted for receiving the video data from the transfer shared memory 28. The transfer shared memory 28 is used by the host output interface 26 to write the video data in the transfer shared memory 28 and is used by the machine input interface 27 to read the video data from the shared memory 10.

By using the transfer shared memory 28 by the host output interface 27 as well as by the machine input interface 27, one copying process of the video data during transferring the video data from the host operating system 3 and/or from the host input interface 22 can be saved, so that the bandwidth of the video data processing arrangement 1 is saved and the latency during transferring the video data is reduced.

The platform operating system 4 further comprises a system output interface 7 for providing the video data as the original video data or as preprocessed video data. The platform operating system 4 provides a user space 8 as a sandbox in the virtual machine 21. The video data processing arrangement 1 comprises at least one of the apps 5, which runs on the video data processing arrangement 1 within the user space 8. The user space 8 and thus the app 5 is restricted from accessing the space 9 of the platform operation system 4. The app 5 uses at least a part of the common services provided by the platform operating system 4. The app 5 may run on the same or a further instance of platform operating system 4. The app 5 comprises an app input interface 10 for receiving the video data from the system output interface 7. The tasks of the app 5 are described above and concern especially video data analysis tasks and especially security video analysis tasks.

The platform operating system 4 provides a shared memory 11, whereby the shared memory 11 is used by the system output interface 7 to write the video data in the shared memory 11 and is used by the app input interface 10 to read the video data from the shared memory 10.

By using the shared memory 11 by the system output interface 7 as well as by the app input interface 10, one copying process of the video data during transferring the video data from the space 9 and/or from the platform operating system 4 can be saved, so that the bandwidth of the camera system 2 is saved and the latency during transferring the video data is reduced.

The virtual machine 21 manages a CPU 12 as a physical or virtual component, whereby the CPU 12 manages the transfer including optionally preprocessing the captured images as video data to the system input interface 6. The virtual machine 21 comprises a memory means 13, which is for example represented as a volatile memory of the computer hardware 2, for example a DDR memory and is managed by the host operating system 3 as a virtual or physical component. The platform operating system 4 optionally comprises a further shared memory 14, which is embodied by the memory means 13. The video data is transferred from the system input interface 6 to the memory means 13 and especially to the further shared memory 14.

The virtual machine 21 manages a GPU (graphic processing unit) 15 as a physical or virtual component, whereby the GPU 15 is adapted to preprocess the video data. The GPU 15 provides a GPU input interface 16, which is adapted to read the video data from the further shared memory 14. The GPU 15 can for example be adapted for converting the video data in the YUV-domain or in another domain as a preprocessing in order to reduce the data amount needed to transfer the video data. Optionally the GPU 15 can reduce the data amount by only transferring the Y-channel as video data, because the Y-Channel is often enough for video analysis as used by the apps 5.

By using the further shared memory 14 by the system input interface 6 as well as by the GPU input interface 16, one copying process of the video data during transferring the video data from the camera sensor 3 and/or from CPU 12 to the GPU 15 can be saved, so that the bandwidth of the video data processing arrangement 1 is saved and the latency during transferring the video data is reduced.

The GPU 15 provides a GPU output interface 17, which is equal to the system output interface 7 and which provides the video data as the original video data or the preprocessed video data to the shared memory 11.

The app 5 is adapted to process the video data and to generate process video data and/or meta data based on the processed video data. The video data processing arrangement 1 comprises an output 18 for providing the processed video data and/or meta data based on the processed video data from the app 5. The output 18 can be connected to a storage for storing the processed video data and/or meta data, to a display for displaying the processed video data and/or meta data and/or a signaling device, for example a traffic light, for outputting signals on basis of the processed video data and/or meta data.

In an overall view, the video data processing arrangement 1 comprises a video pipeline 19, starting from host input interface 22 and ending at the app 5, whereby three copying processes are saved by means of the transfer shared memory 28, the shared memory 11 and the further shared memory 14.

In the embodiment as shown, the platform operating system 4 is realized as Azena operating system, which is based on an Android™ platform. All memory copies in software are minimized to the bare-essential of exactly 1 user-space 6 memory copy for the video-data from the source to the destination executed by the CPU 12 and provided as a common service by the platform operating system 4. Thereby, the Hardware Buffer is used.

The Azena OS as an example for the platform operating system 4 can be hosted as a guest in a cloud environment embodied as the cloud server 19. The hardware peripherals like GPU and memory can be shared from the main platform OS to different Guest OS’s.

The physical CPUs will be shared as virtual CPUs to the Azena OS. Multiple instances of the Azena OS which have their own identity can be running in parallel on one physical system in the cloud. The basic thought is to have video entering onto the system interface and metadata is coming back that might be processed further by the system receiving it.

If for example the same pre-recorded footage is presented to different instances of the Azena OS the transfer shared memory 28 as a buffer and the further shared memory 14 as a buffer and optionally the shared memory 11 can be refcount. Basically not making four copies but putting a refcount onto the said two buffers and saving additional memory copies. The guest virtual address of the Android™ Hardware buffer needs to be mapped to the same virtual (physical address) of the platform OS that runs the different Guest OS instances. Additionally a refcount is beneficial over the different guest OS's to control lifetime of the buffers. By saving those memory copies for certain use cases scaling in cloud environments can be improved. The guest id that is unique for each virtual machine can be used as a tag to identify the metadata origin.