Ms. Congcong Liu

PhD Candidate

The Hong Kong University of Science and Technology

Miss Congcong Liu is currently a fifth-year Ph.D. candidate in the Electrical and Computer Engineering Department at the Hong Kong University of Science and Technology (HKUST), supervised by Prof. Bertram E. Shi. She received the B.S. degree in Information Engineering in 2015 from Zhejiang University (ZJU), graduated with honor of Zhejiang Province. She was a recipient of the Hong Kong Ph.D. Fellowship (2015-2019). Her research interests include gaze tracking, visual attention, and computer vision. She has done groundbreaking work in the use of gaze interaction to understand the effect of gaze information in gardens on human physiological responses, as well as in proposing a new technique, gaze modulated dropout, which enables deep neural networks to improve their performance based on observing human gaze behavior. She has published two journals and six conferences in the top of the field.

Using Eye Gaze to Enhance the Generalization of Imitation Networks to Unseen Environments

Vision-based autonomous driving through imitation learning mimics the behavior of human drivers by training on driver-view image/action pairs. This paper shows that performance can be enhanced via the use of eye gaze. Previous research has shown that novice human learners can benefit from observing experts' gaze patterns. Deep neural networks can also benefit from this. We trained a conditional GAN to estimate human gaze maps accurately from driver-view image. We describe two approaches to integrating gaze information into imitation networks: eye gaze as an additional input and gaze modulated dropout. Both significantly enhance generalization to unseen environments in comparison to a baseline vanilla network without gaze, but gaze-modulated dropout performs the best. We evaluated performance quantitatively on both single images and in closed-loop tests, showing that gaze modulated dropout yield the lowest prediction error, highest success rate in overtaking cars, longest distance between infractions, lowest epistemic uncertainty, improved data efficiency. Using Grad-CAM, we show that gaze modulated dropout enables the network to focus on task-relevant parts, and ignore task-irrelevant parts.