Deep learning and control algorithms of direct perception for autonomous driving

Abstract

We propose an end-to-end machine learning model that integrates multi-task (MT) learning, convolutional neural networks (CNNs), and control algorithms to achieve efficient inference and stable driving for self-driving cars. The CNN-MT model can simultaneously perform regression and classification tasks for estimating perception indicators and driving decisions, respectively, based on the direct perception paradigm of autonomous driving. The model can also be used to evaluate the inference efficiency and driving stability of different CNNs on the metrics of CNN's size, complexity, accuracy, processing speed, and collision number, respectively, in a dynamic traffic. We also propose new algorithms for controllers to drive a car using the indicators and its short-range sensory data to avoid collisions in real-time testing. We collect a set of images from a camera of The Open Racing Car Simulator in various driving scenarios, train the model using this dataset, test it in unseen traffics, and find that it outperforms earlier models in highway traffic. The stability of end-to-end learning and self driving depends crucially on the dynamic interplay between CNN and control algorithms. The source code and data of this work are available on our website, which can be used as a simulation platform to evaluate different learning models on equal footing and quantify collisions precisely for further studies on autonomous driving.