Multi-Modal Representation of Public Safety Events Using Social Media and Surveillance

Abstract

Public safety event is a danger and urgent event that need early detection, fast response and exact recover. The efficient method for responding to a happening public safety event is to collect and describe the related data. In addition to the surveillance cameras from the physical space, the social media data can also be used to collect and describe the related data. In addition to the surveillance cameras from the physical space, the social media data can also be used to collect and describe the related data of a public safety event. In this paper the proposed method concentrates on the steps for describing public safety events. Given a public safety event, videos from the surveillance cameras and social messages from social sensors are collected. The different mode of information including texts, videos and spatial-temporal data is mined to give a description exactly and concisely. At first, the social sensors are associated to surveillance camera by the spatial and temporal information. Second, the social messages are associated to surveillance cameras by semantic information. In third step the social messages are associated to surveillance cameras by the visual features. Apart from the text, social sensors may upload images or videos. Finally the different mode of description is driven based on the three different associations.