Video content analysis or video content analytics (VCA), also known as video analysis or video analytics (VA), is the capability of automatically analyzing video to detect and determine temporal and spatial events.
This technical capability is used in a wide range of domains including entertainment,[1] video retrieval and video browsing,[2] health-care, retail, automotive, transport, home automation, flame and smoke detection, safety, and security.[3] The algorithms can be implemented as software on general-purpose machines, or as hardware in specialized video processing units.
Many different functionalities can be implemented in VCA. Video Motion Detection is one of the simpler forms where motion is detected with regard to a fixed background scene. More advanced functionalities include video tracking[4] and egomotion estimation.[5]
Based on the internal representation that VCA generates in the machine, it is possible to build other functionalities, such as video summarization,[6] identification, behavior analysis, or other forms of situation awareness.
VCA relies on good input video, so it is often combined with video enhancement technologies such as video denoising, image stabilization, unsharp masking, and super-resolution.[citation needed]
Functionalities
editSeveral articles provide an overview of the modules involved in the development of video analytic applications.[7][8] This is a list of known functionalities and a short description.
| Function | Description |
|---|---|
| Dynamic masking | Blocking a part of the video signal based on the signal itself, for example because of privacy concerns. |
| Flame and smoke detection | IP cameras with intelligent video surveillance technology can be used to detect flame and smoke in 15–20 seconds or even less because of the built-in DSP chip. The chip processes algorithms that analyzes the videos captured for flame and smoke characteristics such as color chrominance, flickering ratio, shape, pattern and moving direction. |
| Egomotion estimation | Egomotion estimation is used to determine the location of a camera by analyzing its output signal. |
| Motion detection | Motion detection is used to determine the presence of relevant motion in the observed scene. |
| Shape recognition | Shape recognition is used to recognize shapes in the input video, for example circles or squares. This functionality is typically used in more advanced functionalities such as object detection. |
| Object detection | Object detection is used to determine the presence of a type of object or entity, for example a person or car. Other examples include fire and smoke detection. |
| Recognition | Face recognition and Automatic Number Plate Recognition are used to recognize, and therefore possibly identify, persons or cars. |
| Style detection | Style detection is used in settings where the video signal has been produced, for example for television broadcast. Style detection detects the style of the production process.[9] |
| Tamper detection | Tamper detection is used to determine whether the camera or output signal is tampered with. |
| Video tracking | Video tracking is used to determine the location of persons or objects in the video signal, possibly with regard to an external reference grid. |
| Video error level analysis | Video scene content tamper analysis using free software. Video Error level analysis (VELA) |
| Object co-segmentation | Joint object discovery, classification and segmentation of targets in one or multiple related video sequences |
Commercial applications
editVCA is a relatively new technology, with numerous companies releasing VCA-enhanced products in the mid-2000s.[10][11][12] While there are many applications, the track record of different VCA solutions differ widely. Functionalities such as motion detection, people counting and gun detection are available as commercial off-the-shelf products and believed to have a decent track-record (for example, even freeware such as dsprobotics Flowstone can handle movement and color analysis). In response to the COVID-19 pandemic, many software manufacturers have introduced new public health analytics like face mask detection or social distancing tracking.[13][14][15]
In many domains VCA is implemented on CCTV systems, either distributed on the cameras (at-the-edge) or centralized on dedicated processing systems. Video Analytics and Smart CCTV are commercial terms for VCA in the security domain. In the UK the BSIA has developed an introduction guide for VCA in the security domain.[16] In addition to video analytics and to complement it, audio analytics can also be used.[17]
Video management software manufacturers are constantly expanding the range of the video analytics modules available. With the new suspect tracking technology, it is then possible to track all of this subject's movements easily: where they came from, and when, where, and how they moved. Within a particular surveillance system, the indexing technology is able to locate people with similar features who were within the cameras’ viewpoints during or within a specific period of time. Usually, the system finds a lot of different people with similar features and presents them in the form of snapshots. The operator only needs to click on those images and subjects which need to be tracked. Within a minute or so, it's possible to track all the movements of a particular person, and even to create a step-by-step video of the movements.
Kinect is an add-on peripheral for the Xbox 360 gaming console that uses VCA for part of the user input.[18]
In retail industry, VCA is used to track shoppers inside the store.[19] By this way, a heatmap of the store can be obtained, which is beneficial for store design and marketing optimisations. Other applications include dwell time when looking at a products and item removed/left detection.
The quality of VCA in the commercial setting is difficult to determine. It depends on many variables such as use case, implementation, system configuration and computing platform. Typical methods to get an objective idea of the quality in commercial settings include independent benchmarking[20] and designated test locations.
VCA has been used for crowd management purposes, notably at The O2 Arena in London and The London Eye.
Law enforcement
editPolice and forensic scientists analyse CCTV video when investigating criminal activity. Police use software, such as Kinesense, which performs video content analysis to search for key events in video and find suspects. Surveys have shown that up to 75% of cases involve CCTV. Police use video content analysis software to search long videos for important events.[21][22]
Academic research
editVideo content analysis is a subset of computer vision and thereby of artificial intelligence. Two major academic benchmark initiatives are TRECVID,[23] which uses a small portion of i-LIDS video footage, and the PETS Benchmark Data.[24] They focus on functionalities such as tracking, left luggage detection and virtual fencing. Benchmark video datasets such as the UCF101[25] enables action recognition researches incorporating temporal and spatial visual attention with convolutional neural network and long short-term memory. Video analysis software is also being paired with footage from body-worn and dashboard cameras in order to more easily redact footage for public disclosure and to identify events and people in videos.[26]
The EU is funding a FP7 project called P-REACT[27] to integrate video content analytics on embedded systems with police and transport security databases.[28]
Artificial Intelligence
editArtificial intelligence for video surveillance utilizes computer software programs that analyze the audio and images from video surveillance cameras in order to recognize humans, vehicles, objects and events. Security contractors program is the software to define restricted areas within the camera's view (such as a fenced off area, a parking lot but not the sidewalk or public street outside the lot) and program for times of day (such as after the close of business) for the property being protected by the camera surveillance. The artificial intelligence ("A.I.") sends an alert if it detects a trespasser breaking the "rule" set that no person is allowed in that area during that time of day.
See also
editReferences
edit- ^ KINECT Archived September 12, 2010, at the Wayback Machine, add-on peripheral for the Xbox 360 console
- ^ Dimitrova, Nevenka, et al. "Applications of video-content analysis and retrieval." IEEE multimedia 9.3 (2002): 42-55.
- ^ VCA usage increase in British Security Archived 2014-03-16 at the Wayback Machine, BSIA report
- ^ Cavaliere, Danilo, Vincenzo Loia, and Sabrina Senatore. "Towards an ontology design pattern for UAV video content analysis." IEEE Access 7 (2019): 105342-105353.
- ^ Cavaliere, Danilo; Loia, Vincenzo; Saggese, Alessia; Senatore, Sabrina; Vento, Mario (2019-08-15). "A human-like description of scene events for a proper UAV-based video content analysis". Knowledge-Based Systems. 178: 163–175. doi:10.1016/j.knosys.2019.04.026. ISSN 0950-7051. S2CID 155625544.
- ^ Ma, Yu-Fei, et al. "A user attention model for video summarization." Proceedings of the tenth ACM international conference on Multimedia. 2002.
- ^ Nik Gagvani, Introduction to Video Analytics
- ^ Cheng Peng, Video Analytics
- ^ Style detection Archived 2016-03-03 at the Wayback Machine, Cees G.M. Snoek et al., Detection of TV news monologues by style analysis, ICME'04
- ^ Kwet, Michael (2020-01-27). "The Rise of Smart Camera Networks, and Why We Should Ban Them". The Intercept. Retrieved 2020-10-19.
- ^ "Aimetis", Wikipedia, 2020-01-28, retrieved 2020-10-19
- ^ "Infographic: History of Video Surveillance". IFSEC Global | Security and Fire News and Resources. 2013-12-12. Retrieved 2020-10-19.
- ^ "COVID-19 makes face mask detection essential video analytics - asmag.com". www.asmag.com. Retrieved 2020-10-06.
- ^ Looveren, Pieter van de. "Functionality Beyond Security: The Advent of Open Platform Cameras". www.securityinformed.com. Retrieved 2020-10-06.
- ^ "StackPath". www.securityinfowatch.com. 9 July 2020. Retrieved 2020-10-06.
- ^ British Industry VCA Guide Archived 2018-05-17 at the Wayback Machine, 262 An Introduction to Video Content Analysis Industry Guide
- ^ UK based startup that provides audio analytics into the CCTV industry
- ^ "Project Natal 101". Microsoft. 2009-06-01. Archived from the original on 2012-01-21. Retrieved 2009-06-02.
- ^ "Heat map Intelligent module". Archived from the original on 2017-07-30. Retrieved 2016-07-13.
- ^ i-Lids, Benchmarking initiative by the UK Home Office
- ^ "Northgate offers police forces improved CCTV analysis system". Archived from the original on 4 March 2016. Retrieved 29 Dec 2015.
- ^ "Northgate teams with Dublin tech firm Kinesense to help police video analysis". Risk Manager Online. Retrieved 26 May 2014.
- ^ TRECVID, Academic benchmark initiative by NIST
- ^ PETS Benchmark Data Archived 2006-09-24 at the Wayback Machine, Performance Evaluation of Tracking and Surveillance (PETS) by University of Reading
- ^ Center, UCF (2013-10-17). "UCF101 – Action Recognition Data Set". CRCV. Retrieved 2018-09-12.
- ^ "Police Body Cameras Will Do More Than Just Record You | Fast Company | The Future Of Business". Fast Company. 2017-03-03. Retrieved 2017-03-08.
- ^ P-REACT Project Website
- ^ "Kinesense launches P-REACT, an FP7 project against Petty Crime". 7 April 2014. Retrieved 27 May 2014.