Real-time Visual Object Tracking with Natural Language Description


In recent years, deep learning based visual object trackers have been studied thoroughly, but handling occlusions and/or rapid motions of the target remains challenging. In this work, we argue that conditioning on the natural language (NL) description of a target provides information for longer-term invariance, and thus helps cope with typical tracking challenges. However, deriving a formulation to combine the strengths of appearance-based tracking with the language modality is not straightforward. Therefore, we propose a novel deep tracking-by-detection formulation that can take advantage of NL descriptions. Regions that are related to the given NL description are generated by a proposal network during the detection phase of the tracker. Our LSTM based tracker then predicts the update of the target from regions proposed by the NL based detection phase. Our method runs at over 30 fps on a single GPU. In benchmarks, our method is competitive with state of the art trackers that employ bounding boxes for initialization, while it outperforms all other trackers on targets given unambiguous and precise language annotations. When conditioned on only NL descriptions, our model doubles the performance of the previous best attempt.

The tracking by natural language description task.

The goal is to perform tracking by natural language specifications given by a human. For example, someone specifies track the silver sedan running on the highway and our goal is to predict a sequence of bounding boxes on the input video. We also take advantage of the natural language to better handle the cases of occlusion and rapid motion of the target throughout the tracking process.

Link to arXiv.

title={Real-time visual object tracking with natural language description},
author={Feng, Qi and Ablavsky, Vitaly and Bai, Qinxun and Li, Guorong and Sclaroff, Stan},
booktitle={Proc.\ Winter Conf.\ on Applications of Computer Vision (WACV)},