CityFlow-NL: Tracking and Retrieval of Vehicles at City Scale by Natural Language Descriptions


Natural Language (NL) descriptions can be one of the most convenient or the only way to interact with systems built to understand and detect city scale traffic patterns and vehicle-related events. In this paper, we extend the widely adopted CityFlow Benchmark with NL descriptions for vehicle targets and introduce the CityFlow-NL Benchmark. The CityFlow-NL contains more than 5,000 unique and precise NL descriptions of vehicle targets, making it the first multi-target multi-camera tracking with NL descriptions dataset to our knowledge. Moreover, the dataset facilitates research at the intersection of multi-object tracking, retrieval by NL descriptions, and temporal localization of events. In this paper, we focus on two foundational tasks: the Vehicle Retrieval by NL task and the Vehicle Tracking by NL task, which take advantage of the proposed CityFlow-NL benchmark and provide a strong basis for future research on the multi-target multi-camera tracking by NL description task.

Example frames and NL descriptions from the proposed CityFlow-NL dataset. Crowdsourcing workers annotate the target vehicle using a carefully designed multi-camera annotation platform. NL descriptions we collect tend to describe vehicle color/type (e.gblue Jeep), vehicle motion (e.gturning right and straight), traffic scene (e.gwinding road), and relations with other vehicles (e.gred truck, black SUV, etc.)

Link to ArXiv.

Used as Challenge Track 5 in AI City Challenge Workshop at CVPR 2021.

title={CityFlow-NL: Tracking and Retrieval of Vehicles at City Scaleby Natural Language Descriptions},
author={Feng, Qi and Ablavsky, Vitaly and Sclaroff, Stan},
journal={arXiv preprint arXiv:2101.04741},