Multi-Task Structure-aware Context Modeling for Robust Keypoint-based Object Tracking

Abstract:

In the fields of computer vision and graphics, keypoint-based object tracking is a fundamental and challenging problem, which is typically formulated in a spatio-temporal context modeling framework. However, many existing keypoint trackers are incapable of effectively modeling and balancing the following three aspects in a simultaneous manner: temporal model coherence across frames, spatial model consistency within frames, and discriminative feature construction. To address this problem, we propose a robust keypoint tracker based on spatio-temporal multi-task structured output optimization driven by discriminative metric learning. Consequently, temporal model coherence is characterized by multi-task structured keypoint model learning over several adjacent frames; spatial model consistency is modeled by solving a geometric verification based structured learning problem; discriminative feature construction is enabled by metric learning to ensure the intra-class compactness and inter-class separability. To achieve the goal of effective object tracking, we jointly optimize the above three modules in a spatio-temporal multi-task learning scheme. Furthermore, we incorporate this joint learning scheme into both single-object and multi-object tracking scenarios, resulting in robust tracking results. Experiments over several challenging datasets have justified the effectiveness of our single-object and multi-object trackers against the state-of-the-art.

Publication:

Xi Li, Liming Zhao*, Wei Ji, Yiming Wu, Fei Wu, Ming-Hsuan Yang, Dacheng Tao, and Ian Reid. "Multi-Task Structure-aware Context Modeling for Robust Keypoint-based Object Tracking." TPAMI, accepted, 2018. [paper] [supp] [slides] [dataset (google drive)] [dataset (baidu disk)]

Figure 1: Illustration of our tracking approach

Figure 2: Experimental results

Demo Video:

Youku

YouTube

Executable File:

SMM_Tracker.zip (Executable binary code is available for Windows.)

All files can be downloaded from here (for the users in China, password: hawx)

Single Object Dataset:

OneDrive

Multiple Object Dataset:

[dataset (google drive)] [dataset (baidu disk)]