LAUSR

In this paper, we initially propose a novel framework for replacing advertisement contents in soccer videos in an automatic way by using deep learning strategies. For this purpose, we begin by applying UNET (an image segmentation convolutional neural network technique) for content segmentation and detection. Subsequently, after reconstructing the segmented content in the video frames (considering the apparent loss in detection), we will replace the unwanted content by new one using a homography mapping procedure. Furthermore, the replacement key points will be tracked into the next frames considering the zoom-in and zoom-out controlling using multiplication of the key point coordinates by the homography matrix between each two consecutive frames. Since the movement of objects in video can disrupt the alignment between frames and correspondingly make the homography matrix calculation erroneous, we use Mask R-CNN algorithm to mask and remove the moving objects from the scene. Accordingly, the replacement will be consistent to the video motion of scene. Such framework is denominated as REP-Model which stands for a replacing model. In addition, we have examined the REP-Model over a large database regarding soccer match videos for removing and replacing the playground billboard contents and the results reveal the discriminative nature of our proposed framework. Furthermore, in order to key out the covered object beneath the new content, we use an unsupervised approach in an adversarial learning set-up by learning object masks with playing a game of cut-and-paste, using a discriminator model to find out whether the covered object has been revealed correctly.