Automatic assessment of road safety and conditions is essential for improving road infrastructure and reducing fatalities on the roads. The current manual systems used for road safety not only in Australia but around the world are inefficient and prone to many errors. The major challenges are to accurately detect, segment and classify all road objects and also calculate the distance between the objects. Deep learning with a recent breakthrough has the ability to address such major challenges. In this paper, we propose a novel deep learning approach that can analyze video data and assess road safety and conditions. The specific aims are to develop a novel convolutional neural network based segmentation and classification technique for automatically identifying road attributes for Australian Road Assessment Program (AusRAP) and a novel proximity measurement technique for distance measurement between AusRAP attributes. The proposed approach has been evaluated on the roadside video data collected by the Queensland department of transport and main roads and the results are presented.