research papers on autonomous vehicles

Autonomous vehicles: The future of automobiles

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 28 May 2024

Machine vision-based autonomous road hazard avoidance system for self-driving vehicles

• Chengqun Qiu 1 , 2 ,
• Hao Tang 1 ,
• Yuchen Yang 2 ,
• Xinshan Wan 2 ,
• Xixi Xu 1 ,
• Shengqiang Lin 1 ,
• Ziheng Lin 3 ,
• Mingyu Meng 4 &
• Changli Zha 5  

Scientific Reports volume  14 , Article number:  12178 ( 2024 ) Cite this article

233 Accesses

Metrics details

• Computer science
• Mechanical engineering

The resolution of traffic congestion and personal safety issues holds paramount importance for human’s life. The ability of an autonomous driving system to navigate complex road conditions is crucial. Deep learning has greatly facilitated machine vision perception in autonomous driving. Aiming at the problem of small target detection in traditional YOLOv5s, this paper proposes an optimized target detection algorithm. The C3 module on the algorithm’s backbone is upgraded to the CBAMC3 module, introducing a novel GELU activation function and EfficiCIoU loss function, which accelerate convergence on position loss l box , confidence loss l obj , and classification loss l cls , enhance image learning capabilities and address the issue of inaccurate detection of small targets by improving the algorithm. Testing with a vehicle-mounted camera on a predefined route effectively identifies road vehicles and analyzes depth position information. The avoidance model, combined with Pure Pursuit and MPC control algorithms, exhibits more stable variations in vehicle speed, front-wheel steering angle, lateral acceleration, etc., compared to the non-optimized version. The robustness of the driving system's visual avoidance functionality is enhanced, further ameliorating congestion issues and ensuring personal safety.

Similar content being viewed by others

Real-time detection of road manhole covers with a deep learning model

Fast and accurate object detector for autonomous driving based on improved YOLOv5

Study on a risk model for prediction and avoidance of unmanned environmental hazard

Introduction.

Deep learning has made remarkable strides in the field of autonomous driving and Advanced Driver Assistance Systems 1 , 2 . However, the driving safety of such systems has not gained widespread societal acceptance. As a result, research and validation of autonomous driving in complex road conditions continue unabated. It is worth noting that environmental perception in autonomous driving systems heavily relies on deep learning technology 3 . Prior to the application of deep learning in machine vision, visual perception technology was largely stagnant 4 , 5 . Images captured by onboard cameras are critical for intelligent perception in autonomous driving systems.

This paper aims to optimize the YOLOv5s object detection model and employ an optimized camera visual ranging strategy to address the challenges mentioned above. In conjunction with the optimized visual model, a local obstacle avoidance approach is adopted. Additionally, the paper combines the Pure Pursuit algorithm 6 , 7 and the Model Predictive Control (MPC) algorithm 8 , 9 to evaluate the collision avoidance functionality of the experimental autonomous vehicle. The chosen flat terrain in Jiangsu, known for its well-developed manufacturing and commercial centers, is ideal for testing autonomous driving technology. These areas frequently face traffic congestion issues. The approach is designed to tackle complex road conditions, adverse weather scenarios, and improve the accuracy and robustness of visual algorithms for target identification and tracking in complex road environments. Three innovative optimizations ve been applied to enhance the visual model:

(1) EfficiCIoU Loss Function: A new EfficiCIoU loss function is introduced, addressing the limitations of the traditional CIoU function. The traditional CIoU function involves numerous tricks to handle the IoU of predicted and anchor boxes, which increases computational complexity and does not consider the IoU for small targets, leading to sample imbalance issues. The new EfficiCIoU loss function improves the IoU, incorporates a context mechanism to better understand environmental information, and enhances the detection of small or densely packed objects;

(2) Integration of CBAM Attention Mechanism and YOLOv5s C3 Module: The paper integrates the CBAM attention mechanism with the YOLOv5s C3 module. The new CBAMC3 module combines channel attention and spatial attention, enhancing the model’s focus on input features. It is embedded within the CNN framework, improving the inference speed of the visual model;

(3) Upgrade to GELU Activation Function: The traditional ReLU activation function is upgraded to the GELU activation function. Compared to the traditional linear activation function, GELU is more conducive to feature mapping in models because it does not exhibit saturation in input values, thus addressing the gradient vanishing problem.

Machine vision convolutional neural networks

Deep learning frameworks are well-suited for representation learning 10 , 11 , 12 , employing multi-layered nonlinear transformations 13 , 14 in an efficient manner. Due to their cost-effectiveness, high precision, and robustness, they find extensive applications in the field of autonomous driving. In convolutional neural network (CNN) 15 , 16 , 17 , the convolutional kernels extract features while reducing the parameter count. Pooling layers shrink the size of feature maps while retaining essential information. Fully connected layers typically operate at the top of the network, mapping convolutional kernels and feature maps to output categories, connecting one or more hidden layers to the output layer. Activation functions work between the convolutional and fully connected layers, enabling the model to handle more complex tasks. Different activation functions serve various purposes in processing the output layer's tasks. The multi-layered architecture described above facilitates feature extraction by the model.

Figure  1 illustrates the architecture of a deep learning network for image processing. It starts with a Convolution (Conv) layer 18 , which extracts useful features from input images using 5 × 5 filters with a depth of 3 (corresponding to RGB color channels). This generates 20 different feature maps, highlighting specific features from the input. Next is the "Tied Conv Max Pooling" layer 19 , 20 , where “Tied Conv” means parameter sharing, reducing the model's size and preventing overfitting. "Max Pooling" downsizes feature maps by selecting maximum values in different regions, preserving essential features. The "Cross-Input Neighborhood Differences" step enhances matching by computing local differences between input images, useful for stereo vision matching tasks. “Patch Summary Features” are compact features from small patches, often generated by aggregating local features. “In the Across Patch Features” step, features from multiple image blocks are compared and integrated to create a global feature representation, aiding understanding of the entire image scene. Fully connected layers flatten output feature maps into one-dimensional vectors, which enter a network for classification. Finally, the SoftMax layer 21 transforms fully connected layer output into a probability distribution, crucial for classifying images in the vehicle vision system. Neural networks, with their multi-layered architecture and advanced feature extraction capabilities, enable thorough analysis of visual data from vehicles, providing essential environmental perception capabilities for autonomous driving systems.

Structure diagram of a deep learning-based machine vision convolutional neural network nested layer by layer.

YOLOv5s object detection modeling

Machine vision has become a crucial research area in the field of autonomous driving. In object detection algorithms, the YOLOv5s algorithm utilizes convolutional neural networks to calculate the positions of objects to be recognized 22 , 23 , 24 , classifying and localizing them accurately. YOLOv5s is a high-accuracy neural network that surpasses the limitations of traditional image processing algorithms. It is a one-stage algorithm known for its fast inference speed, with frame rates suitable for autonomous driving systems. As decipted in Fig.  2 , the network's input comprises 640 × 640 three-channel images, typically divided into grid regions of 80 × 80, 40 × 40, and 20 × 20. The network's output includes predictions for all grid regions, including classification probabilities, confidence scores, and bounding box information for objects in those regions.

The basic framework of the improved YOLOv5s algorithm.

Key enhancements to the YOLOv5s algorithm include data augmentation using Mosaic 25 , 26 , adaptive anchor box calculations, and adaptive image scaling. The backbone employs operations such as convolution and pooling to reduce feature map dimensions, increase depth, and incorporate the CBAM C3 module for automatic attention to image features. Feature fusion (Neck) is achieved using a PAN + FPN 27 structure, merging feature maps with different resolutions and rich semantic information, creating a feature pyramid.

The convolution layers in the backbone network and feature fusion section (Neck) use the GELU activation function to introduce non-linearity, facilitating the capture of complex image features. The final detection head (Head) is responsible for outputting object detection results, including bounding boxes, confidence scores, and class information. During model training, the EfficiCIoU LOSS function continuously calculates gradients and updates parameters to achieve convergence. The YOLOv5s model is iteratively trained to obtain a set of weight parameters that minimize the loss function 28 . The depth of grid regions in the network relates to their accuracy, with shallower grids providing higher accuracy, particularly for detecting small objects. Careful grid processing is also necessary to avoid semantic ambiguity.

The FPN + PAN structure combines feature maps of different scales in a hierarchical manner, similar to the SSD approach 29 , enabling layer-wise processing and strong feature fusion capabilities. This structure is a crucial module in hierarchical detection methods and facilitates further feature fusion. The vehicle is equipped with a camera that captures RGB images, which are fed into the YOLOv5s algorithm. In the Backbone phase, image preprocessing is performed, including normalization to match the input size of the network. The system employs adaptive anchor box calculations and image scaling, while Mosaic data augmentation is used during grid training to improve model accuracy and speed. In addition to the CBAMC3 module, the Backbone also uses the Focus architecture as a base for general feature extraction. The GELU activation function after each convolutional layer aids in capturing complex image features 30 . The Neck network sits between the base network and the detection head, enhancing feature diversity and robustness. The Head is responsible for outputting the results of object detection, and the number of branches depends on the specific detection method, often used for classification and data regression.

Camera adaptation processing

The system utilizes a stereo camera setup, and prior to conducting experiments, the camera system undergoes a series of processing steps 31 , 32 . Initially, the stereo cameras perform real-time image capture, capturing full stereo images. Subsequently, stereo matching algorithms are applied to these images to calculate disparities, thereby enabling depth estimation. Deep learning algorithms are then employed for object recognition within the images. Based on the identified object classes and their respective spatial extents, distance and orientation information for the objects is extracted from the disparity map. With this information in hand, a well-designed obstacle avoidance strategy is devised to control the vehicle's actions. This strategy includes actions such as emergency braking or steering adjustments to facilitate automatic obstacle avoidance, ensuring safe vehicle operation. The overarching goal is to enable the vehicle to navigate safely in its environment by leveraging the stereo camera system for distance and object recognition.

Given the world coordinates of a real point M = [X,Y,Z] T and its corresponding camera pixel coordinates m = [u, v] T , as shown in Fig.  3 , the transformation from M to m, combined with a scale factor s, the vertical coordinate of the calibration board in the real world Z W  = 0, and simplification using the symmetric matrix B , is used to determine the camera's internal parameters by capturing images of a chessboard pattern. Once the camera's intrinsic parameter matrix is computed, the extrinsic parameter matrix can be obtained.

Algorithm flow of internal and external parameters and distortion coefficients of the camera.

During the process of solving both intrinsic and extrinsic parameters, factors such as lens distortion, image resolution, and noise levels can affect the estimation of these parameters. Therefore, it is necessary to introduce the maximum likelihood estimation to optimize the parameters.

Now, combining with the camera parameters in Table 1 , for each chessboard image containing m corner points, their corresponding image projection points can be represented as:

Subsequently, the probability density function and likelihood function for the chessboard corner points are derived. The Levenberg–Marquardt algorithm (L-M algorithm) is employed to maximize the likelihood function. The L-M algorithm 33 introduces a control parameter λ, which is used to balance the gradient descent and Gauss–Newton steps. In each iteration, λ value is dynamically adjusted based on the current parameter estimate and information from the Hessian matrix to balance the two steps, thereby facilitating a faster convergence towards the optimal solution.

Combining the pixel coordinates under the ideal and actual models, camera optical point coordinates, continuous image coordinates under undistorted and radial distortion conditions, the radial distortion parameters are calculated using the least squares method. By incorporating the undistorted intrinsic and extrinsic parameters with the radial distortion parameter k, the estimation of all parameters is achieved, resulting in:

After completing camera calibration and obtaining the optimal intrinsic and extrinsic parameter matrices along with the distortion coefficients considering radial distortion, the notation m( A ,k 1 ,k 2 , R i , T i , M j ) represents the coordinates corresponding to the j point on the i image, accounting for radial distortion.

Experiments and analysis

Object detection algorithm optimization.

In the calculation of the loss function in the head Section 34 , we introduce the theoretically superior EfficiCIoU_Loss, which improves the components of position loss l box , confidence loss l obj , and classification loss l cls .

Calculation of position loss l box :

Calculation of confidence loss l obj :

Calculation of the confidence of classified losses l cls :

EfficiCIoU_Loss combines the concepts from EfficientDet and CIoU, assuming the presence of two bounding boxes: the predicted box and the target box. EfficiCIoU_Loss comprises the following components:

EfficiCIoU_Loss computes the final loss value as a combination of these error terms, and the specific form may vary depending on different implementations. The design objective of this loss function is to better handle the position and size of the target boxes, ultimately improving object detection performance. Different implementations of YOLOv5s may have varying parameter settings to adapt to different tasks and datasets, allowing for flexibility and customization to suit specific requirements.

Experimental setup

The training of the YOLOv5s algorithm model in this system was conducted on a Windows operating system using the Pytorch framework. The system ran on CUDA 11.0 and Python 3.9, and the hardware configuration of the workstation included an Intel Xeon Gold 6248R CPU and a GeForce RTX 4090 GPU. The dataset used for the model is a self-made dataset of 5607 photos from Car recorder and KITTI, an Open Access dataset for autonomous driving training. The dataset is roughly divided into a training set of 70%, a validation set of 15%, and a test set of 15%. The model is trained with 70% of the images from the self-made dataset, as the self-made dataset is collected for specific driving environments or scenarios, and the model can be better adapted to these specific conditions by using the self-made dataset for training. training can make the model better adapted to these specific conditions. The remaining 30% are randomly selected from KITTI, the Open Access dataset that allows the model to be trained in more driving scenarios. For model validation and testing, more than 90% of the images are from KITTI, and the diversity of samples for validation and testing allows the model to be evaluated under a wider range of driving conditions that have not been seen before, and helps to assess the model's generalization ability. Cross-use of self-made datasets and Open Access datasets balances the performance and applicability of the model and enhances the reliability of the model.

The object detection system divided the collected images from vehicles on the road into multiple objects, annotated the dataset with labels, and then split it into training and validation sets. The model underwent multiple rounds of training where parameters were continually updated through gradient descent. During training, minor changes in model training parameters were amplified with the increase in the number of layers. Furthermore, changes in parameters at different layers altered the data distribution in those layers, posing significant challenges to model training.

Deep learning networks have achieved significant success in large-scale image and video recognition, thanks to the development of large public image databases like ImageNet and high-performance computing systems. Through deep learning, the decision-making process in autonomous driving is achieved in an end-to-end manner, directly mapping sensor-collected image information to vehicle driving behavior without extensive manual feature engineering.

Prof. Karim and others 35 introduced a transformation from existing univariate time series classification models to multivariate time series classification models by incorporating LSTM-FCN. They enhanced the fully convolutional block with squeeze and excitation blocks to further improve accuracy. This framework can be used to build an end-to-end CNN-based steering controller for vehicles, predicting the future distribution of vehicle motion based on current camera observations and vehicle states, and predicting the required steering wheel angle from continuous video images.

Scholars like Li 36 proposed a novel Simultaneous Localization and Mapping (SLAM) method, namely Attention-SLAM, which combines a visual saliency model with traditional monocular visual SLAM. This approach mimics human navigation patterns, and the generated saliency map can focus more on the same salient object.

Drawing from the experiences of previous researchers, we further optimized visual perception under the deep learning framework. The pre-trained YOLOv5s model was deployed on the vehicle's onboard camera for inference. In Fig.  4 ( a,b,c,d ), even in scenarios with multiple objects within the camera's field of view, distant or relatively small and blurry objects, the inference process exhibited strong accuracy. The confidence level of the algorithm's object inference capability remained around 0.95. This makes it highly suitable for application in scenarios involving multiple lanes, multi-vehicle tracking, or traffic congestion.

YOLOv5s targets detect road vehicles and pedestrians.

Optimize results and analysis

We conducted a comprehensive optimization of the traditional YOLOv5s architecture to enhance the accuracy of small object detection and the overall performance of the system. Initially, within the backbone network, we replaced the original C3 module with the CBAMC3 module, which integrates the CBAM 37 , 38 attention mechanism. This modification significantly bolstered the network's ability to extract features in complex scenes, particularly enhancing sensitivity and recognition precision in detecting small objects. In the neck and head networks, we substituted the original activation method with the GELU 39 activation function. Owing to its unique non-linear characteristics, the GELU function improved the feature transmission and transformation processes, rendering the model more effective and precise in handling targets of various sizes. We implemented the EfficiCIoU loss function in the head network, a novel loss function that optimizes the model's convergence behavior during training, especially in the precise computation of localization loss l box , confidence loss l obj , and classification loss l cls . EfficiCIoU significantly heightens localization accuracy by more accurately measuring the similarity between predicted and actual bounding boxes, thereby further enhancing the model's ability to detect small objects. Through these targeted improvements, our model demonstrated superior performance in practical applications, particularly in the identification of road vehicles and the analysis of in-depth positional information within autonomous driving systems.

Since the model's loss function tends to decrease with training iterations, we have selected data from the first 100 training rounds. Please refer to Fig.  5 (a,b,c)for the convergence speed changes in the three loss types for training dataset samples and Fig.  6 for the comparative analysis of our model with other leading object detection models. In comparison to the original YOLOv5s model, our modified architecture exhibits enhancements of 1.9%, 2.1%, and 2.73% in [email protected]:0.95, [email protected], and Precision, respectively. Against the state-of-the-art YOLOv8, our model demonstrates marginal improvements of 0.4%, 0.64%, and 1.33% across these metrics.

( a ) Position loss convergence change; ( b ) Convergence change of confidence loss; ( c ) Convergence change of classification loss.

A Comparative analysis of our model with other leading object detection models.

Ablation experiments

Ablation experiments are a common method used in deep learning research to evaluate the specific contribution of certain parts of the model to the final performance by removing or replacing them. In the YOLOv5s vehicle detection task, ablation experiments help us clarify which features are critical for improving detection accuracy, accelerating inference, and optimizing model size. In this way, we can gain a deeper understanding of how the model works and optimize it effectively in real-world applications. From Table 2 , we can find that model optimization improves the accuracy, recall and mAP of the model to a certain extent, and the model also achieves precise control over the computational resource requirements. Although Params and GFLOPs were increased, they did not put significant pressure on storage resources. In addition, the moderate increase in GFLOPs indicates that the increase in required computation is limited, suggesting that the model can maintain efficient operation even in systems with limited computational resources. The slight improvement in inference Time, which is still maintained in the millisecond range, ensures the model's real-time responsiveness in fast dynamic environments. These improvements highlight the utility of our approach, making it a strong candidate for resource-constrained self-driving vehicle deployments.

Comparison of algorithm complexity

In order to measure the computational complexity of the algorithms, we evaluate the number of parameters, the amount of operations, and the inference time of each algorithm, which will provide us with a more accurate performance comparison of our model. These experiments not only validate the practicality of our model, but also provide reliable data support for future optimization and application. In the following table, it can be seen that our algorithm is only 0.32 M higher than YOLOv8s in terms of the parameters, but the Precision and mAP values have been almost similar to YOLOv8s performance, and the GFLOPs of our algorithm has been significantly reduced by 24.91% from 28.90 to 21.69 in YOLOv8s. It signifies that our algorithm improves computational efficiency while maintaining efficient functionality. The inference time is 0.015 s, which is 8.59% faster compared to 0.0163 s in YOLOv8s. This speedup not only implies faster processing, but also reflects the fact that our model is able to provide more efficient performance in real-world applications, especially when it comes to real-time object detection in driving vehicles (Table 3 ).

Camera vision model testing and analysis

An optimization model equipped with a Stereo Camera is deployed to achieve real-time tracking and target detection functionality for autonomous vehicles. The optimization algorithm is utilized to obtain spatial positioning information of target vehicles. Testing is conducted on selected road segments within the Jiangsu Yancheng High-Tech Industrial Development Zone. This zone encompasses well-developed manufacturing and commercial centers, which often face traffic congestion issues and carry a certain risk of traffic accidents. Therefore, researching an avoidance system for autonomous driving is particularly suitable in this context. The vehicle-mounted camera detects target frames, providing coordinate information of the target vehicle and the distance from the detected vehicle to the center of the vehicle-mounted camera.

By combining deformable convolution modules with depth information selection modules, the YOLOv5s model integrated with the accompanying camera can perform tasks related to target identification, tracking, and provide feedback on the depth information of target positions. As shown in Fig.  7 (a–d), the on-road performance of the detection model is depicted: (b) illustrates a scenario with a road target vehicle located at coordinates (X: − 2.537, Y: − 0.161) in the two-dimensional coordinate system of the vehicle's field of view, at a distance of 7.05 m from the host vehicle. (d) presents a scenario where the target vehicle is parked at coordinates (X: 0.016, Y: 0.451) in the two-dimensional coordinate system of the vehicle's field of view, at a distance of 3.602 m from the host vehicle.

( a ) The target's depth information; ( b ) Location information of the vehicle while it is moving; ( c ) The target's depth information; ( d ) Location information for parked vehicles.

Modeling and analysis of hazard avoidance vehicles

To ensure the experimental feasibility rigorously, this simulation adopts a combined simulation approach, integrating Matlab/Simulink, Carsim, and Prescan. Carsim is used to configure the vehicle's dynamic model, Prescan provides the simulated environmental scenarios, and data processing is performed in Matlab/Simulink 40 , 41 . The algorithms employed include Pure Pursuit and MPC control for local planning and tracking control, respectively.

Obstacle vehicles are placed on the road, categorized as stationary vehicles and moving vehicles, simulating scenarios where the vehicle encounters both stationary obstacles and vehicles in motion while driving on the road. This comprehensive approach ensures a thorough evaluation of the autonomous driving system's performance in various real-world situations.

50 km/h experiment

A simulation scenario was established using Carsim to simulate obstacle avoidance for autonomous vehicles within a predefined environment. In this integrated simulation, the autonomous vehicle was configured to cruise at a speed of 50 km/h while navigating through a simulated scenario that included both static obstacles and moving vehicles on the road. The purpose of this joint simulation was to assess whether the vehicle could still operate effectively under the new system. This comprehensive simulation test aimed to evaluate whether the vehicle could continue to function efficiently and safely, including the ability to avoid obstacles, within this altered operational framework.

As depicted in Fig.  8 , the data showcasing changes in velocity, lateral acceleration, front wheel steering angle, and yaw rate at a speed of 50 km/h demonstrate the performance of the autonomous avoidance model equipped with the new visual algorithm. The model vehicle's velocity remains remarkably stable, staying within the vicinity of the initially set target speed. Within the framework of deep learning, the driving system efficiently detects vehicles ahead, allowing for prompt decision-making in terms of following, decelerating, and lane-changing maneuvers. It is evident that the autonomous driving system, enhanced by the new visual algorithm, possesses a stable and effective obstacle avoidance capability. This reinforces the system's ability to detect and respond to vehicles in its path, ensuring safe and reliable operation.

Speed, lateral acceleration, front wheel rotation angle, and yaw angular velocity vary with position.

70 km/h experiment

The system was configured with a target vehicle speed of 70 km/h, and static obstacles as well as moving vehicles were introduced into the road scenario. Additionally, the threshold for the speed of moving vehicles in the scenario was increased. This adjustment aimed to further evaluate the effectiveness of the obstacle avoidance algorithm under more challenging conditions and higher-speed scenarios. By conducting simulations with these modifications, the system's ability to successfully navigate and avoid obstacles at the increased speed threshold can be thoroughly assessed, providing valuable insights into the algorithm's performance in a variety of driving scenarios.

As shown in Fig.  9 a, after system optimization, the vehicle's velocity curve exhibits reduced fluctuations both in amplitude and frequency. This reduction indicates that the vehicle experiences fewer instances of acceleration and deceleration, resulting in a more stable maintenance of the target vehicle speed. Moving on to Fig.  9 ), the lateral acceleration curve for the optimized system demonstrates fewer fluctuations, indicating reduced occurrences of lateral deviation, with significantly lower peak values compared to the pre-optimization curve. Figure  9 c further supports these improvements, indicating that the system now requires less time for steering, and there are fewer instances of large steering maneuvers.

( a ) The velocity of the vehicle before and after system optimization changes with time; ( b ) The lateral acceleration before and after system optimization changes over time; ( c ) The front wheel rotation angle before and after the system optimization changes over time.

This paper addresses the challenges faced by autonomous vehicle cameras in complex road conditions and the difficulties encountered in target identification and tracking. Leveraging the fusion of camera sensing technology and deep learning, the YOLOv5s algorithm was upgraded in three key aspects: the loss function, CBAMC3 module, and GELU activation function. During the model training process, notable improvements were observed in the convergence speed of the loss function, and the resulting model consistently achieved outstanding metrics in terms of mAP, precision, and recall. Real-world road testing of the visual model revealed that the optimized target detection algorithm efficiently and reliably acquired target position and depth information. In model testing, the optimized visual algorithm enabled the driving system to make path selections that closely approached optimal solutions. This resulted in reduced unnecessary steering and deceleration operations. The system's decision-making precision and robustness were significantly enhanced. Leveraging this exceptional visual model, the system's obstacle avoidance functionality also exhibited greater robustness. In summary, the upgrades made to the YOLOv5s algorithm, combined with rigorous real-world testing, have resulted in an optimized target detection algorithm. This algorithm enhances the system's ability to efficiently and reliably obtain target position and depth information, leading to improved decision-making precision and robustness, ultimately enhancing the system's obstacle avoidance capabilities. It can further improve traffic congestion and ensure personal safety.

Our research focuses solely on optical cameras. Camera vision systems detect and recognize objects by capturing and analyzing images, and are particularly adept at processing color and texture information, which is critical for understanding visual features such as traffic signs and road markings. In addition, camera devices are typically less expensive and smaller than radar systems and can be more easily integrated into existing in-vehicle systems. However, while cameras are uniquely suited to provide these important 2D image and color information, they also have limitations in self-driving vehicle applications. Camera systems are very sensitive to lighting conditions, such as nighttime or backlit environments that may greatly affect their detection capabilities.

LIDAR technology, which acquires the precise distance to an object by emitting laser pulses and measuring their reflection time, is capable of operating in light-free conditions, and its measurement accuracy is greater than that of optical cameras, although its performance may be affected by adverse weather conditions such as rain or fog 42 . This performance impact underscores the importance of employing 3D spatial data from LiDAR when modeling accurately in complex environments, especially in navigation and obstacle avoidance in self-driving vehicles. So subsequently if accuracy is to be improved then additional sensing equipment including LiDAR and radar may still be required.

Despite these challenges, the unique advantages and future potential of camera vision systems in autonomous driving cannot be ignored. Through technological innovation and interdisciplinary collaboration, camera vision will play an increasingly important role on the road to autonomous driving. In response to the limitations of camera vision systems, future research is likely to focus on improving these limitations using advanced algorithms, such as using deep learning to improve detection under unfavorable lighting conditions or enhancing depth perception through sensor fusion techniques. In addition, a combination of complementary e.g. LIDAR and radar sensing devices may still be required for optimal environmental sensing.

Data availability

The collision avoidance experiment datasets generated during the current study are available in the Mendeley Data repository, https://data.mendeley.com/datasets/f2kkskc55s/1 . The self-made datasets are available from the corresponding author on reasonable request.

Abbreviations

channel function

Augmentation matrix of M

Augmentation matrix of m

Measurement factor

Internal reference matrix

Rotation matrix

Pan vectors

Matrix parameters

Symmetry matrix

Distortion coefficient

Figure i Point j coordinates

Camera point coordinates

Matrix of coefficients

The difference in pixel coordinates without distortion

The coordinates after calculating the distortion distortion

Loss of location

Confidence loss

Classified losses

Predict box coordinates

Prediction box’s width

Prediction box’s height

Target box’s width

You only look once

Gaussian error linear unit

Model predictive control

Advanced driver assistance systems

Convolutional neural network

Rectified linear unit

Convolutional block attention module

Single shot multiBox detector

Simultaneous localization and mapping

Long short-term memory

Cross-stage partial networks

Long short-term memory fully convolutional network

Spatial pyramid pooling

mean average precision

Weinberger, S. Driverless: Intelligent cars and the road ahead. Nature 548 , 28–28. https://doi.org/10.1038/548028a (2017).

Article   ADS   Google Scholar  

Li, Y. et al. Deep learning for LiDAR point clouds in autonomous driving: A review. IEEE Trans. Neural Netw. Learn. Syst. 32 , 3412–3432. https://doi.org/10.1109/TNNLS.2020.3015992 (2021).

Article   PubMed   Google Scholar  

Song, H. A. et al. Collaborative processing and data optimization of environmental perception technologies for autonomous vehicles. Assembly Autom. 41 , 283–291. https://doi.org/10.1108/AA-01-2021-0007 (2021).

Article   Google Scholar  

Zhu, H., Yuen, K. V., Mihaylova, L. & Leung, H. Overview of environment perception for intelligent vehicles. IEEE Trans. Intell. Trans. Syst. 18 , 2584–2601. https://doi.org/10.1109/TITS.2017.2658662 (2017).

Huang, G. L. et al. Context-aware machine learning for intelligent transportation systems: A survey. IEEE Trans. Intell. Transp. Syst. 24 , 17–36. https://doi.org/10.1109/TITS.2022.3216462 (2023).

Shan, Y. X. et al. CF-Pursuit: A pursuit method with a clothoid fitting and a fuzzy controller for autonomous vehicles. Int. J. Adv. Rob. Syst. 12 , 134. https://doi.org/10.5772/61391 (2015).

Elbanhawi, M., Simic, M. & Jazar, R. Receding horizon lateral vehicle control for pure pursuit path tracking. J. Vib. Control 24 , 619–642. https://doi.org/10.1177/1077546316646906 (2018).

Choi, Y., Lee, W., Kim, J. & Yoo, J. A variable-sampling time model predictive control algorithm for improving path-tracking performance of a vehicle. Sensors 21 , 6845. https://doi.org/10.3390/s21206845 (2021).

Article   ADS   PubMed   PubMed Central   Google Scholar  

Choi, W. Y., Lee, S. H. & Chung, C. C. Horizonwise model-predictive control with application to autonomous driving vehicle. IEEE Trans. Ind. Inf. 18 , 6940–6949. https://doi.org/10.1109/TII.2021.3137169 (2022).

Zahrawi, M. & Shaalan, K. Improving video surveillance systems in banks using deep learning techniques. Sci. Rep. 13 , 7911. https://doi.org/10.1038/s41598-023-35190-9 (2023).

Article   ADS   CAS   PubMed   PubMed Central   Google Scholar  

Goumiri, S., Benboudjema, D. & Pieczynski, W. A new hybrid model of convolutional neural networks and hidden Markov chains for image classification. Neural Comput. Appl. 35 , 17987–18002. https://doi.org/10.1007/s00521-023-08644-4 (2023).

AlDahoul, N. et al. Parasitic egg recognition using convolution and attention network. Sci. Rep. https://doi.org/10.1038/s41598-023-43068-z (2023).

Article   PubMed   PubMed Central   Google Scholar  

Li, S. P., Luo, J. X. & Hu, Y. M. Nonlinear process modeling via unidimensional convolutional neural networks with self-attention on global and local inter-variable structures and its application to process monitoring. ISA Trans. 121 , 105–118. https://doi.org/10.1016/j.isatra.2021.04.014 (2022).

Khan, S. D. & Basalamah, S. Multi-branch deep learning framework for land scene classification in satellite imagery. Remote Sens. 15 , 3408. https://doi.org/10.3390/rs15133408 (2023).

Zhang, L. W. et al. A novel ground-based cloud image segmentation method based on a multibranch asymmetric convolution module and attention mechanism. Remote Sens. 14 , 3970. https://doi.org/10.3390/rs14163970 (2022).

Article   ADS   CAS   Google Scholar  

Qiu, C. Q., Wang, G. L., Meng, M. Y. & Shen, Y. J. A novel control strategy of regenerative braking system for electric vehicles under safety critical driving situations. Energy 149 , 329–340. https://doi.org/10.1016/j.energy.2018.02.046 (2018).

Zhao, X. Q., Tuo, B. & Hui, Y. Y. Deep learning with CBAM-based CNN for batch process quality prediction. Meas. Sci. Technol. https://doi.org/10.1088/1361-6501/aceb82 (2023).

Cheng, X. H. & Chen, Z. Z. Multiple video frame interpolation via enhanced deformable separable convolution. IEEE Trans. Pattern Anal. Mach. Intell. 44 , 7029–7045. https://doi.org/10.1109/TPAMI.2021.3100714 (2022).

Sun, S., Hu, B., Yu, Z. & Song, X. N. A stochastic max pooling strategy for convolutional neural network trained by noisy samples. Int. J. Computers Commun. Control 15 , 1007. https://doi.org/10.15837/ijccc.2020.1.3712 (2020).

Yildirim, M. Analog circuit architecture for max and min pooling methods on image. Analog Integr. Circuits Signal Process. 108 , 119–124. https://doi.org/10.1007/s10470-021-01842-x (2021).

Shao, H. & Wang, S. F. Deep classification with linearity-enhanced logits to softmax function. Entropy https://doi.org/10.3390/e25050727 (2023).

Article   MathSciNet   PubMed   PubMed Central   Google Scholar  

Yan, B. N., Li, J. X., Yang, Z. Z., Zhang, X. P. & Hao, X. L. AIE-YOLO: Auxiliary information enhanced YOLO for small object detection. Sensors 22 , 8221. https://doi.org/10.3390/s22218221 (2022).

Xi, D. J., Qin, Y. & Wang, S. J. YDRSNet: an integrated Yolov5-Deeplabv3+real-time segmentation network for gear pitting measurement. J. Intell. Manufact. 34 , 1585–1599. https://doi.org/10.1007/s10845-021-01876-y (2023).

Oreski, G. YOLO*C - Adding context improves YOLO performance. Neurocomputing https://doi.org/10.1016/j.neucom.2023.126655 (2023).

Zhao, B. Y., Wu, Y. F., Guan, X. R., Gao, L. R. & Zhang, B. An improved aggregated-mosaic method for the sparse object detection of remote sensing imagery. Remote Sens. 13 , 2602. https://doi.org/10.3390/rs13132602 (2021).

Cao, Z. H., Shao, M. F., Xu, L., Mu, S. M. & Qu, H. C. MaskHunter: real-time object detection of face masks during the COVID-19 pandemic. IET Image Proc. 14 , 4359–4367. https://doi.org/10.1049/iet-ipr.2020.1119 (2020).

Yasir, M. et al. Multi-scale ship target detection using SAR images based on improved Yolov5. Front. Mar. Sci. 9 , 1086140. https://doi.org/10.3389/fmars.2022.1086140 (2023).

Tong, C. H., Yang, X. H., Huang, Q. & Qian, F. Y. NGIoU Loss: Generalized intersection over union loss based on a new bounding box regression. Appl. Sci. Basel 12 , 12785. https://doi.org/10.3390/app122412785 (2022).

Article   CAS   Google Scholar  

Leng, J. X. & Liu, Y. An enhanced SSD with feature fusion and visual reasoning for object detection. Neural Comput. Appl. 31 , 6549–6558. https://doi.org/10.1007/s00521-018-3486-1 (2019).

Lee, M. H. Y. Mathematical analysis and performance evaluation of the GELU activation function in deep learning. J. Math. 2023 , 4229924. https://doi.org/10.1155/2023/4229924 (2023).

Babic, D., Babic, D., Fiolic, M., Eichberger, A. & Magosi, Z. F. A comparison of lane marking detection quality and view range between daytime and night-time conditions by machine vision. Enegies 14 , 4666. https://doi.org/10.3390/en14154666 (2021).

Qi, J. H., Chen, H. & Chen, F. P. Extraction of landslide features in UAV remote sensing images based on machine vision and image enhancement technology. Neural Computing Appl. 34 , 12283–12297. https://doi.org/10.1007/s00521-021-06523-4 (2022).

Rubio, J. D. Stability analysis of the modified levenberg-marquardt algorithm for the artificial neural network training. IEEE Trans. Neural Networks Learn. Syst. 32 , 3510–3524. https://doi.org/10.1109/TNNLS.2020.3015200 (2021).

Article   MathSciNet   Google Scholar  

Talukdar, K., Bora, K., Mahanta, L. B. & Das, A. K. A comparative assessment of deep object detection models for blood smear analysis. Tissue Cell https://doi.org/10.1016/j.tice.2022.101761 (2022).

Karim, F., Majumdar, S., Darabi, H. & Harford, S. Multivariate LSTM-FCNs for time series classification. Neural Netw. 116 , 237–245. https://doi.org/10.1016/j.neunet.2019.04.014 (2019).

Li, J. Q. et al. Attention-SLAM: A visual monocular SLAM learning from human gaze. IEEE Sens. J. 21 , 6408–6420. https://doi.org/10.1109/JSEN.2020.3038432 (2021).

Wang, Y. C., Li, J. D., Chen, Z. M. & Wang, C. L. Ships’ small target detection based on the CBAM-YOLOX algorithm. J. Mar. Sci. Eng. 10 , 2013. https://doi.org/10.3390/jmse10122013 (2022).

Wang, W., Tan, X. A., Zhang, P. & Wang, X. A CBAM Based Multiscale Transformer Fusion Approach for Remote Sensing Image Change Detection. IEEE J . Sel. Top. Appl. Earth Obs. Remote Sens. 15 , 6817–6825. https://doi.org/10.1109/JSTARS.2022.3198517 (2022).

Tai, S. K. et al. Deep learning for traffic sign recognition based on spatial pyramid pooling with scale analysis. Appl. Sci. Basel 10 , 6997. https://doi.org/10.3390/app10196997 (2020).

Qiu, C. Q. & Wang, G. L. New evaluation methodology of regenerative braking contribution to energy efficiency improvement of electric vehicles. Energy Convers. Manag. 119 , 389–398. https://doi.org/10.1016/j.enconman.2016.04.044 (2016).

Qiu, C. Q. et al. A novel regenerative braking energy recuperation system for electric vehicles based on driving style. Energy https://doi.org/10.1016/j.energy.2023.129055 (2023).

Dreissig, M., Scheuble, D., Piewak, F. & Boedecker, J. Survey on LiDAR Perception in Adverse Weather Conditions. 2023 IEEE Intelligent Vehicles Symposium (IV) , 1–8 https://doi.org/10.1109/IV55152.2023.10186539 , (2023).

Download references

Acknowledgements

This study is supported by the National Key R&D Program of China (No.2018YFB0106405), the Jiangsu Natural Science Foundation of China (No. BK20211364), the open project of Jiangsu Intelligent Optoelectronic Devices and Measurement and Control Engineering Research Center (No.2022IODM004), the Teaching Research Foundation of Yancheng Teachers University (Project No. 2023YCTCJGZ06), the Postgraduate Research and Practice Innovation Project of Yancheng Institute of Technology (No. SJCX22_XZ026).

Author information

Authors and affiliations.

School of Automotive Engineering, Yancheng Institute of Technology, Yancheng, 224051, China

Chengqun Qiu, Hao Tang, Xixi Xu & Shengqiang Lin

Jiangsu Province Intelligent Optoelectronic Devices and Measurement-Control Engineering Research Center, Yancheng Teachers University, Yancheng, 224007, China

Chengqun Qiu, Yuchen Yang & Xinshan Wan

School of Automotive and Transportation Engineering, Hefei University of Technology, Anhui, 230009, China

Interdisciplinary Graduate School of Science & Engineering, Tokyo Institute of Technology, Yokohama, 2268502, Japan

Mingyu Meng

School of Electronic Engineering and Intelligent Manufacturing, Anqing Normal University, Anhui, 246133, China

Changli Zha

You can also search for this author in PubMed   Google Scholar

Contributions

Conceptualization, C.Q. and C.Z.; Methodology, C.Q.; Software, H.T.; Validation, H.T, Y.Y. and X.W.; Formal Analysis, X.X.; Investigation, S.L.; Resources, M.M.; Data Curation, M.M.; Writing – Original Draft Preparation, C.Q.; Writing – Review & Editing, M. M.; Visualization, C.Z.; Supervision, Z.L.; Project Administration, Z.L.; Funding Acquisition, C.Q.; All authors commented on and reviewed paper.

Corresponding author

Correspondence to Changli Zha .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Qiu, C., Tang, H., Yang, Y. et al. Machine vision-based autonomous road hazard avoidance system for self-driving vehicles. Sci Rep 14 , 12178 (2024). https://doi.org/10.1038/s41598-024-62629-4

Download citation

Received : 28 December 2023

Accepted : 20 May 2024

Published : 28 May 2024

DOI : https://doi.org/10.1038/s41598-024-62629-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Deep learning
• Machine vision
• Self-Driving
• Risk avoidance
• Control algorithm

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

• Open access
• Published: 06 May 2023

Exploration of issues, challenges and latest developments in autonomous cars

• B. Padmaja   ORCID: orcid.org/0000-0001-5327-9795 1 ,
• CH. V. K. N. S. N. Moorthy   ORCID: orcid.org/0000-0001-7209-8017 2 , 3 ,
• N. Venkateswarulu   ORCID: orcid.org/0000-0003-1782-579X 4 &
• Myneni Madhu Bala   ORCID: orcid.org/0000-0003-4734-5914 5  

Journal of Big Data volume  10 , Article number:  61 ( 2023 ) Cite this article

14k Accesses

9 Citations

2 Altmetric

Metrics details

Autonomous cars have achieved exceptional growth in the automotive industry in the last century in terms of reliability, safety and affordability. Due to significant advancements in computing, communication and other technologies, today we are in the era of autonomous cars. A number of prototype models of autonomous cars have been tested covering several miles of test drives. Many prominent car manufacturers have started investing huge resources in this technology to make it commercialize in the near future years. But to achieve this goal still there are a number of technical and non-technical challenges that exist in terms of real-time implementation, consumer satisfaction, security and privacy concerns, policies and regulations. In summary, this survey paper presents a comprehensive and up-to-date overview of the latest developments in the field of autonomous cars, including cutting-edge technologies, innovative applications, and testing. It addresses the key obstacles and challenges hindering the progress of autonomous car development, making it a valuable resource for anyone interested in understanding the current state of the art and future potential of autonomous cars.

Introduction

Autonomous cars are becoming more pragmatic from year to year as multi-national companies are racing ahead to produce intelligent vehicles. The projected value for autonomous vehicles in the global market is at $615 Billion by 2026. According to the U.S. Department of Transportation's National Highway Traffic Safety Administration (NHTSA) fully automated or autonomous vehicles are "those in which operation of the vehicle occurs without direct driver input to control the steering, acceleration, and braking and are designed so that the driver is not expected to constantly monitor the roadway while operating in self-driving mode.". The advancement and rise of autonomous cars are due to the significant research results obtained in the arenas of wireless and embedded systems, sensors, communication technologies, navigation, data acquisition, and analysis.

The initial thought of autonomous cars was initiated in the year 1920 with the "phantom autos" concept, which means a remote-control device, which was used to control the vehicle [ 1 ]. Later in the 1980s self-managed autonomous cars were developed. Further, NavLab of Carnegie Mellon University contributed majorly in this field by developing an Autonomous Land Vehicle (ALV) [ 2 ]. In a major breakthrough in 1987, the “Prometheus project” of Mercedes [ 3 ] gave the design of their first automated car with the capability of tracking lanes. At that time, it was not completely autonomous, but it had the ability to automatically switch lanes. In the twenty-first century, there is a huge demand for low-cost, high-performance autonomous cars. There is a fine line of difference between the two terminologies: the automated car and the autonomous car. The term automated car refers to a vehicle with little human intervention, whereas the term autonomous car refers to a vehicle without any human intervention. The autonomous car is a fully computer-controlled car which can instruct (guide) itself, make its own decisions, familiar with its surrounding without any human interference (intervention).

The concept of connected car technology is influenced by autonomous cars [ 4 ] as both technologies are related to each other. Layered architectures are being proposed to address challenges faced due to the internet response time and the compatibility of various components that are being used in connected car technology [ 5 ]. Autonomous vehicles need to be connected to each other to improve overall autonomy when driving on the road. For example, the connected car works on a vehicular ad hoc network (VANET) technology and a dedicated short-range communication (DSRC) standard protocol [ 6 ] using which communication between vehicles is possible when they are in range. The VANET [ 7 ] provides 2 types of applications; one regarding safety and the other one related to infotainments. In Autonomous cars, communication related security measures are stringent whereas in connected cars, security measures are moderately relaxed. In the latest developments regarding the connected cars and VANET technologies, many multinational technology companies such as Google along with car manufacturers such as Tesla and Audi are working together and we see a solid collaboration among the technology companies and vehicle manufacturers to facilitate the development and design of cars.

Similarly, Microsoft has begun a coalition with Volvo and Toyota for building autonomous cars. Also, companies like NVidia have shown their dedication to making autonomous cars by launching NVidia Drive PX2, a dynamic supercomputer GPU and a deep learning-based computing platform for autonomous cars [ 8 ]. A few more Asian companies such as TATA, KIA, and Hyundai are funding in design, development, and research regarding automated cars. Also, the European auto market (Mercedes and BMW) is in the race for autonomous cars, and their goal is the development of full-fledged commercial versions by 2020. Many companies such as Volkswagen and the French PSA group are focusing on developing autonomous cars and they have started test-drive since 2016. The PSA group brought together many car manufacturers together and drove their autonomous cars covering hundreds of kilometers from Paris to Amsterdam without any driver supervision in 2016 on a level-3 autonomy [ 9 ].

In the development of an autonomous system, there are several issues that must be addressed properly by the car manufacturing companies such as governments’ regulations, consumer satisfaction, cost, reliability, and safety. Further, an important role is played by federal regulations in achieving novel technologies, and autonomous cars are no exemption from that. The automatic transmission system plays a vital role in autonomous cars and these days as most automobiles use this technology due to its reduction in cost and improvement in quality and management of a fleet of electric vehicles [ 10 ]. In brief, this technology will consume some more time period till it is affordable, and reachable to customers in terms of cost and reliability. This paper highlights all the issues and challenges involved in the development of autonomous cars and conducts a survey on the latest autonomous car technologies that are trying to overcome these issues and challenges in an efficient way.

Related work

Till today, a number of works have been done to explore multiple issues of autonomous car system [ 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 ]. But the majority of surveys focus only on certain aspects of the autonomous car and none of these surveys present a comprehensive (holistic) method towards autonomous car technology.

Campbell et al. [ 11 ] discussed the approaches to challenges faced in urban environments by autonomous vehicles. Okuda et al. [ 12 ] did a thorough survey on the usage of advanced driving assistance (ADAS) in autonomous cars and the trends in the technology. Fagnant et al. [ 13 ] helped in surveying the required policies to make a nation ready for autonomous vehicles. Moreover, Bagloee et al. [ 14 ] focused on a few challenges that such different policies provide to autonomous cars. Other surveys regarding its functionalities include planning and motion control [ 15 ], long-term maps’ constructions [ 18 ] and visual perception from both implementation and operators’ perspectives [ 20 , 21 ]. Furthermore, Abraham et al. [ 16 ] conducted a survey on consumer trust and also the preferences of the consumer on already available alternatives. Gupta et al. [ 22 ] aimed to understand the public's attitude towards autonomous vehicles by analyzing large amounts of data from Twitter without the need for manual labeling. The authors used advanced machine learning techniques to analyze the data and identify patterns in the public's perception of autonomous vehicles. The study found that the majority of tweets about autonomous vehicles were positive in nature, but there were also concerns about safety and privacy. Joy et al. [ 17 ] looked over security and privacy issues in autonomous cars. Madhav et al. [ 23 ] presented a study on how to improve human trust in autonomous vehicles through the use of Explainable Artificial Intelligence (XAI). The authors focused on bridging the gap between artificial decision-making and human trust by providing an explanation for the decisions taken by autonomous vehicles. Bairy et al. [ 24 ] presented a model for explaining the decision-making process of autonomous vehicles. The model is based on integrated formal methods, which are mathematical methods used to model and verify systems. The authors claim that providing explanations for the actions of autonomous vehicles is important for building trust in the technology of autonomous vehicles and for ensuring accountability in the event of an accident.

Parkinson et al. [ 19 ] have expansively analyzed cyber threats in autonomous cars and the challenges they pose on the future of connected vehicles. Mazri et al. [ 25 ] proposed a self-defense mechanism against security attacks for autonomous vehicles. The mechanism is designed to protect against various types of cyberattacks, including those targeting the communication systems and decision-making processes of the vehicles. The authors presented a detailed analysis of the potential vulnerabilities and risks of autonomous vehicles, and provide a comprehensive evaluation of the proposed mechanism using various simulation scenarios. Li et al. [ 26 ] investigated the preferences of drivers when it comes to performing secondary tasks and how the vehicle's level of autonomy affects the driver's task engagement. The study also explored the potential impact of secondary tasks on safety and comfort while driving. The findings of the study can be used to design interfaces and interactions that better support drivers’ needs and preferences in highly autonomous vehicles.

The use of blockchain technology for training autonomous cars has been proposed by G. M. Gandhi et al. [ 27 ]. In this process, all connected cars can share their experience with each other. Blockchain can also be used to maintain energy transactions at charging stations [ 28 ]. Many recent surveys on autonomous cars have majorly focused on a few topics of autonomous cars. Table 1 presents a summary of related works. Figure  1 depicts the prediction of Autonomous vehicle implementation by 2060.

Predicted Development of Autonomous Vehicles from 2010 to 2060. This figure illustrates the expected advancement of autonomous vehicle technology over the course of the next several decades. The x-axis represents the years, starting in 2010 and ending in 2060. The y-axis represents the level of autonomy, with 0% being no automation and 100% being fully autonomous. The line represents the projected trajectory of the technology, starting at 0% in 2010 and gradually increasing to 100% by 2060. The graph illustrates the rapid pace of technological advancement in the field of autonomous vehicles and the significant impact it is expected to have on transportation in the coming years

This ground-breaking survey delves deep into the latest developments in the exciting field of autonomous cars. From cutting-edge technologies and innovative applications to in-depth simulations, this paper provides a comprehensive overview of the current state of the art. Additionally, it addresses the key obstacles and challenges that are hindering the progress of autonomous car development, including both technical and non-technical issues.

Organization

Organized in a clear and logical manner, this survey begins by summarizing existing works in the field and outlining the different levels of autonomy according to the standards set by the Society of Automotive Engineers (SAE).

The heart of the paper is dedicated to exploring the underlying technology behind autonomous cars, including detailed descriptions of their design, components, and functionalities. The benefits of autonomous cars are also discussed, along with a survey of state-of-the-art research outcomes.

The survey goes on to explore the implementation and design challenges that must be overcome in order to bring autonomous cars to reality. Finally, the paper concludes with a look at the technology and challenges for autonomous car deployment, making it a must-read for anyone interested in this rapidly evolving field.

Contribution

This survey provides a thorough and up-to-date overview of the latest developments in the field of autonomous cars, including cutting-edge technologies, innovative applications, and in-depth simulations. Additionally, it addresses the key obstacles and challenges that are hindering the progress of autonomous car development, and provides an in-depth look at the technology, benefits, and challenges behind autonomous car deployment. It is a valuable resource for anyone interested in understanding the current state of the art and future potential of autonomous cars.

The autonomous car technology

The autonomous car has been in the spotlight during the last few decades and prototype models have been improved by various car manufacturers. But, the commercial actualization of autonomous cars is a major challenge. As a basic task, gearing of every autonomous car is performed with numerous actuators and sensors that produce vast amounts of data which must be handled and analyzed to take decisions on time. The amount of data that the car must handle depends on the levels of autonomy [ 38 ] as shown in Table 2 . The key point is the requirement of various sensors and actuators to function autonomously, so that the car should foresee, decide and maneuver cautiously according to some strategy. A number of fundamental characteristics of autonomous car design are outlined in this section. Figure  2 depicts a number of elevated functional components of a typical autonomous car.

This image displays the functional architecture of an autonomous car. The architecture is divided into several functional blocks, each of which is responsible for a specific set of tasks. The main functional blocks of the architecture are Data Acquisition, Data Processing and Actuation – The components that carry out the commands from the control system like acceleration, braking and steering

The functional architecture of an autonomous car is a layered structure which contains data acquisition, data processing and actuation. Data acquisition is performed by the hardware components, such as sensors, radars, cameras, LIDAR, and transceivers [ 40 ]. The collected data is processed by a central computer system, which is later implemented by the decision-support system (DSS). The DSS activates the autonomous car and the situational awareness is actualized through both short- and long-range imaging devices [ 41 ]. Figure  3 shows the system architecture of the autonomous car, where it shows the areas covered by different components in the car design. Different ranges of situational awareness vary from application to application and it is achieved through multiple components. For instance, prevention of front and rear bumper collisions is done through infrared devices, whereas short-range radars are used for object detection, lane-change cautioning, and traffic view. Equipping autonomous cars with a series of cameras for the surrounding views and LIDAR helps in collision avoidance and emergency brakes. Long-range radars help in cooperative cruise control and long-range traffic view construction. Altogether the aforementioned components are networked and work firmly with each other, as shown in Fig.  2 .

This figure displays the basic architecture of an autonomous car. The architecture consists of various components that work together to enable the car to drive itself. The main components of the architecture are Sensors, Perception and Localization, Planning and Control, Communication and Cloud

For movement of an autonomous car from one point A to B, the car performs a number of important steps in an iterative manner until it reaches its destination such as Perceive and aware about its surrounding environment, Path planning and navigation and Controlled movements on the road [ 42 ].

After perceiving its surrounding environment, it makes path planning along with its destination information and then starts navigating to reach the destination. A number of controlled movements are exercised for a smooth, safe ride on the road with the help of actuators and sensors [ 43 ]. Electronic Control Units (ECUs) control most of the components electronically. ECUs communicate with each other and with the decision support system through the controller area network (CAN) bus inside each car. During a drive, the autonomous car exhibits a number of manoeuvres which needs both software/hardware support, extensive coordination, and data sharing among different components of the car.

Autonomous car benefits

In spite of several complexities and difficulties, the autonomous car provides security, user-friendliness, comfort and value-added services to its customers.

Safety is one of the major concerns and is given top priority in the automotive domain. Every year many millions of people lost their lives in road accidents according to the National Highway Traffic Safety Administration (NHTSA). More than 90% of accidents [ 44 ] happen due to human errors and these errors are caused by various factors such as carelessness, aggressiveness, intoxication and distraction. So, it’s essential to have an alternative and safer mechanism like autonomous driving cars to eliminate human errors and save lives.

Daniel Zelle et al. [ 32 ] developed a security evaluation platform for autonomous driving (SPEAD) to enable researchers to develop, implement and evaluate new security solutions for autonomous vehicles. SPEAD allows us to model realistic autonomous vehicle architectures and test their security mechanisms. There is another aspect of safety that is how to protect the car from thieves. Due to high-end on-board sensors, recognition of the car owner is done by the car itself and it sends the owner an alert in case of any unwanted situation. Unlike ordinary cars, autonomous cars do not require keys, but they operate with finger prints, retina scan and voice recognition software. It is also equipped with finger-print enabled door-lock system to provide more security.

Increase in revenue and business opportunities

Mobility-as-a-Service (MaaS) benefits the customers by saving multiple resources including money, time, and space. Car sharing and car-pooling are two popular applications among customers today. With the advent of autonomous cars, these applications will become more effective by making use of car resources more efficiently. These applications create economic benefits, and also reduce air pollution produced by vehicles in urban areas. This also generates huge business opportunities for the customers. In future, autonomous cars will transform taxi and car rental business. There is no longer requirement of drivers in taxis and rental cars, therefore it will reduce the cost and increase the revenue of owners. Further, it will impact the software industry to develop many smart applications for cars. Hence, autonomous cars will help in increase in revenue by reducing maintenance and labour costs.

User-friendliness and convenience

User-friendliness and convenience are other additional benefits of autonomous cars. In some situations, physically disabled or intoxicated persons can’t drive a car manually. Similarly, for aged, young people without licence, autonomous cars will be a suitable mode of transportation. In such situations, the autonomous car provides a safe and economical way of transportation.

Improving traffic conditions

In autonomous cars, car sharing and car-pooling can be a major advantage, hence it will increment per-vehicle occupancy and decrement the number of vehicles on the road, thereby improving the traffic conditions. Inter-vehicle distance is strictly followed in autonomous cars to improve passenger safety; hence in turn it reduces road traffic. These cars follow traffic rules more precisely, thereby condensing the requirement for more traffic personnel on the road. Fuel efficiency is also improved by automatically choosing the best and shortest path from source to destination [ 45 ]. Hence there is also a decrease in air pollution.

Bo Liu et al. [ 29 ] proposed a redundancy concept for autonomous vehicle functions using microservice architecture in which vehicle-vehicle and cloud-vehicle communication ensure that important vehicle functions keep executing even in the case of a hardware failure. These cars also make use of a fuel-efficient mode through programming by eliminating unnecessary braking situations on the road. One such example is regenerative-braking [ 46 ] in electric autonomous vehicles which uses the kinetic energy of the car to convert it into electrical energy (fuel) until the car stops naturally. Qiao et al. [ 36 ] proposed a hybrid model for traffic assignment and control that combines the strengths of both traditional traffic assignment models and control methods for autonomous vehicles. The model uses a combination of traffic simulation, machine learning, and optimization techniques to assign traffic and control the movement of autonomous vehicles in a given road network. The authors evaluated the performance of the proposed model using real-world traffic data and show that it can improve the overall efficiency and safety of traffic flow in comparison to traditional methods.

Autonomous parking

In metropolitan cities, there are a number of issues related to vehicle parking such as getting parking space in peak hours, increased population and maintaining inter-vehicle distance in parking slots. The advantage of autonomous cars is they can park themselves even in a narrow available parking slot, which is very difficult for a driver to park a car manually. Marvy Badr Monir Mansour et al. [ 47 ] implemented a project to demonstrate autonomous parallel car parking that can be used efficiently in metropolitan cities. Baramee Thunyapoo et al. [ 48 ] proposed a simulation framework for autonomous-car parking for moderate complexity scenarios. Shiva Raj Pokhrel et al. [ 49 ] developed (and evaluated) an experience-driven, secure and privacy-aware framework of parking reservations for automated cars. Since autonomous-cars can communicate with each other, they can reduce traffic congestion by coordinating their movement on the road.

Consumer-centric experience

In autonomous cars passengers and drivers can sit, relax and enjoy the ride. They can also simultaneously do their personal work or utilize the car’s entertainment system [ 50 ]. When the autonomous car is paired with a smart phone then a passenger can command the car to perform some important tasks automatically like picking up children from school, picking up or dropping someone at the airport. These kinds of sophisticated features called “ Summon ” of autonomous cars are introduced by Tesla Company in its high-end models [ 51 ]. Using the “ Summon” application, the car owner can instruct the car to go to the designated parking place even to the basement of the building.

Further research is going on to understand various driving patterns by analysing the behaviour of the drivers by considering drivers’ age, gender, driving experience, personality, emotion, and history of accidents [ 52 ]. A number of features of autonomous cars can be customized based on the human behaviour. For example, overtaking and speed while driving depends on gender, age, emotion etc. Young people drive faster than elders, whereas female drivers drive more cautiously. Similarly, people with families and kids drive more carefully than others. So, during autonomous car customization, one should take these factors into consideration. In future, autonomous cars will lead to attract the developers to build a number of customer-driven applications, where customers can personalize their travel experience according to their choice of speed, in-car entertainment, and level of risk Travelling in autonomous vehicles also increases productivity as the user can focus on work instead of driving the vehicle during transit.

Technology behind autonomous cars

In this section, various technological researches conducted so far in the area of autonomous cars are presented. This section also covers the software components and algorithms used in this technology. Broggi et al. [ 53 ] discussed regarding a number of tests performed on driverless cars in different scenarios such as roads with less to heavy traffic during 1990–2013. The outcomes witnessed changes in the behaviours of the drivers in different environments such as traffic lights, freeway junctions, pedestrian crossing. This car test was named PROUD (Public Road Urban Driverless) and it was one of the biggest achievements in autonomous car technology. PROUD resulted in a few important observations such as precise route maps, efficient learning and perception mechanism. Jo et al. [ 54 , 55 ] concentrated on the generalization of the autonomous car development procedure independent of any specific environment. FlexRay was utilized as a communication protocol and software platform to increase the system performance. Traditionally CAN (Controller Area Network) technology is used to communicate among different ECUs (Electronic Control Unit) in normal cars. But this technology was not suitable for autonomous cars due to its low speed and vulnerability to different attacks [ 56 , 57 ]. FlexRay is a faster and more efficient technology, but it’s expensive. Some technologies used in fully automated cars are Lane Keep Assist (LKA), Park Assist (PA), Automatic Emergency Braking (AEB), Driver Monitoring (DM), Traffic Jam Assist (TJA), Dead Reckoning (DR) [ 38 ]. Most of these rely on sensor data and processing of this sensor data using machine learning/deep learning algorithms.

Computer vision

Autonomous cars require two essential and critical features such as object detection and computer vision. Using object detection techniques, autonomous cars should see the road and detect the object in the road. These two important features along with other modules help to drive safely on the road by avoiding unwanted situations. Janai et al. [ 58 ] conducted an accurate survey on computer vision algorithms such as perception, object detection and tracking, and motion planning used in autonomous cars. But still there are a number of unpredictable scenarios creating challenges for autonomous cars. Consequently, the success of computer vision algorithms will presumably take longer time. In addition to these algorithms, the decision support system has an important role in analysing the behaviour, learning mechanism of drivers. Also, AI has an important part in the prediction and perception aspects of autonomous cars. Shi et al. [ 20 ] studied on computer vision algorithms used for lane detection, object detection and surface detection experimented with GPU (Graphics Processing Unit), FPGA (Field-programmable Gate Array) and ASIC (Application-specific Integrated Circuit) Among these, the performance of ASICs is better than GPUs and FPGAs.

Computer vision is a boundless field of study, where autonomous cars use only object detection and motion estimation algorithms. These cars detect both static and dynamic objects and accurate object detection is always a challenging task due to multiple factors like shadows, identical objects, background lighting, and the size of the object. The detection algorithms accept these multiple factors into consideration and perform object detection with different sensors. Recent Autonomous cars use sensor fusion mechanisms to combine different types of sensors to detect day/night time, living/non-living organisms [ 59 , 60 ]. Sensor-fusion based object detection techniques have more accuracy than traditional object detection methods. Chen et al. [ 61 ] proposed a CNN (Convolutional Neural Network) deep learning approach to detect 3D objects with a single camera, which takes generated date in LIDAR and images as input and predicts 3-D representation of that data. This method first detects objects using dissimilar features and then improves it for the identification of true objects. Then with the help of sensors data, it categorizes the objects into different types. This process of categorization is termed as pixel level semantic segmentation [ 62 ]. To support semantic segmentation, shallow and DL approaches are used for classification and prediction [ 62 , 63 , 64 ]. The approaches used in autonomous cars for object detection, semantic segmentation, and classification give improved accuracy, but have some drawbacks such as algorithm complexity, computational overhead, latency and complex design features. Therefore, compared to the shallow learning approach, DL (deep learning) approaches (such as auto-encoders and CNN) are preferred for object detection and classification due to its automatic feature selection process [ 65 , 66 ]. DL-based approaches are also used in the construction of 3D images from the 2D image which are used for motion planning and actuation process in autonomous cars [ 67 , 68 ].

Deep and machine learning approaches in autonomous cars

AI, ML and DL techniques are essential for autonomous cars as the object behaviour and surrounding environment are unpredictable. ML and DL techniques are used by most computer vision mechanisms. A deep Neural Network (DNN) approach learns features automatically from large data sets. Perception is an important aspect of autonomous cars and analysis of huge data collected from sensors for decision making is done by deep learning techniques. Tian et al. [ 69 ] introduced a DNN model called “ DeepTest ” to assess the behaviour of autonomous cars and found erroneous behaviour multiple times during testing when the car comes under erratic traffic and environmental conditions. These test results were not satisfactory for the current challenges and increased the need for more meticulous measures for the accurate functioning of autonomous cars. Chen et al. [ 70 ] used a novel mechanism to learn automatically the features from an image to calculate affordance in autonomous cars. Chen computed affordance for each driving action, as a substitute for individual driving scenes, it depends on multiple factors like static and dynamic objects on the road, pedestrian and vegetation Duanfeng Chu et al. [ 33 ] proposed a combined trajectory planning and tracking algorithm for autonomous vehicle control. Muhammad Mobaidul Islam et al. [ 71 ] proposed an efficient training strategy for pedestrian detection by occlusion handling by classifying body parts. Mohammed Ikhlayel et al. [ 72 ] made a car prototype for traffic sign detection using convolutional neural network (CNN).

There are two types of perceptions used in autonomous cars namely mediated perceptron and behaviour reflex mode. In mediated perceptron, current surrounding is not known and in behaviour reflex mode, deep neural networks are used to train the system based on human behaviour [ 70 , 73 ]. Furthermore, Chen et al. also proposed another direct perception technique which is based on CNN. This system systematically learns mapping from a captured image to various features related to driving actions. TORCS, an open-source car racing simulator was used to test the car. Laddha et al. [ 74 ] proposed an algorithm to identify features from road images required for autonomous driving. In this algorithm an automated labelled training dataset was taken to make the process more scalable. It takes multiple sensor inputs which are mounted on vehicles including localization and camera sensors. This algorithm effectively utilized CNN to detect obstacles along the way with a moderately good accuracy. Dairi et al. [ 74 ] introduced another DL technique to identify obstacles on the road based on using a hybrid deep autoencoder and stereovision. More than 98% of accuracy was shown on different datasets by this method and it outperformed the DBN (Deep Belief Network) and SDA (Stacked Auto-encoders). Tam et al. [ 35 ] proposed a probability-based artificial potential field method for autonomous vehicles to avoid uncertain obstacles. The authors argued that this method is more effective in avoiding uncertain obstacles than traditional methods, and that it can improve the safety of autonomous vehicles.

Qizwini et al. [ 73 ] made a system and trained with 5 affordance parameters and tested by simulating with some realistic assumptions. XU et al. [ 75 ] used a Long Short-Term Memory Fully Convolutional Network (LSTM-FCN) by training the multi-modal driving-behaviours to predict future ego-motion problem in autonomous driving. The LSTM-FCN model was trained by a large video dataset of vehicular actions by the authors [ 76 ]. The author addressed the problems of traditional end-to-end learning by using a very large crowd-sourced dataset and the learning results were promising. Similar to this model, a number of popular DL architectures such as VGG-16, Google LeNet, AlexNet and ResNet are used accurately in semantic segmentation and scene understanding in autonomous cars [ 77 ]. Among these models, AlexNet achieved an accuracy of 84.6% and VGG achieved an accuracy of 92.7%, Google LeNet 93.3% and ResNet with 96.4% respectively. So, DL architectures played important roles in the multiple aspects of autonomous cars.

Sensors, communications, and control

Computing unit implement all the logic and it is the heart of an autonomous car. Sensors and actuators are highly needed in autonomous car systems. Basically, both known and unknown situations and environments are dealt with by autonomous cars and it needs ML, DL, AI techniques. These ML/DL techniques are data-intensive and these data are collected using various sensors placed within the car. Therefore, it follows a series of procedures such as data acquisition, data processing, communication and controlling among different modules inside the car and its surroundings. In addition to that it has to take autonomous decisions based on the circumstances and this feature requires a lot of interaction data with neighbours, infrastructure and the Internet. Communication among different modules and the environment is a vital function in autonomous cars and this data helps in sensor data analytics.

Autonomous cars deal with a huge number of sensors which generates a huge volume of data and these data are processed to get maximum utilization of it. Sensor Fusion is one of the commonly used techniques to gather data from multiple sensors intelligently to assist in the decision support systems. A number of algorithms have been proposed to deal with heterogeneous data used in autonomous cars. Oliveria et al. [ 78 ] proposed a more accurate technique to visualize a scene from 3D data gathered from sensors by using large scale polygonal primitives. Also, a visual scene may change constantly when obstacles come on the road, so there is a need for a stable mechanism to deal with unanticipated environments. So, the reconstructed scene should be calibrated continuously with the current scene to increase the efficiency as always, the new data from sensors is processed. Polygonal primitives-based scene reconstruction algorithms are incremental in nature and detection of polygon and reconstruction of new scene is performed using old data. Therefore, time required for detecting these polygons increases with an increase in the number of polygons, hence there is an increase in efficiency.

Identifying the road is one of the important aspects in autonomous cars and it is done using different sensors, camera and LIDAR (Fig. 4 ). Xiao et al. [ 79 ] applied a Hybrid Conditional Random Field (HCRF) to overcome the disadvantages of LIDAR and camera sensors. The proposed technique used a multi-modal approach and applied a binary labelling mechanism for separating road and background. This approach showed a maximum degree of accuracy then the existing point-wise CRF. Jo et al. [ 54 , 55 ] used a number of existing functionalities along with sensor fusion algorithms that deal with sensors used inside the car, communication systems and also stressed upon communication of autonomous cars with the outside world by implementing different algorithms and schemes to establish communication with other entities. There exists efficient coupling between connected vehicles and autonomous cars as they are orthogonal to each other and reinforce towards making of intelligent transportation system. Vehicle-to-vehicle (V2V) and Vehicle-to-Infrastructure (V2I) communication technologies are popularly used to exchange their mobility data in platooning and cooperative traffic information system applications. Hobert et al. [ 80 ] worked on applications of cooperative autonomous cars such as convoy management, intersection management, low delay, cooperative sensing and enhanced reliability. It also helps in sharing the information about surrounding environment to operate more efficiently, timely and safely.

Source – Velodyne. This image shows a 3D map of the surroundings of an autonomous car, constructed using lidar sensor data. The map illustrates the precision and accuracy of the lidar sensor in capturing the geometry of the environment, including buildings, roads, and other obstacles. The use of lidar sensors in autonomous cars is crucial for creating a detailed and accurate map of the car's surroundings, which is then used for navigation and localization

Peng et al. [ 81 ] scrutinized the presentation of IEEE 802.11p protocol and considered inter-platoon communication for different platoons in terms of network performance, delay, packet loss and other parameters. Platoon management is an important task in platoon communication, but it is LTE-driven to reduce the communication delay in autonomous cars. In recent years, transmitters and receivers are fixed in the front and back lights of the cars which uses visible light communication (VLC) technology, but it is in its initial stage and structured channel modelling is mandatory to meet the requirements of autonomous cars. VLC technology is also affected by the massive amount of data and network bandwidth. So numerous researchers are working on finding new techniques to reach out the demands of bandwidth requirement. Chang et al. [ 82 ] experimented on cooperative motion planning to communicate with pedestrians called “Eyes” on car, which means how the autonomous car makes an eye contact with the walker and evaluate the intention of the walker while crossing the road. The author tested with the real-world users and obtained the results of 86.6% in deciding intention of the users while crossing the road. It is observed that the real users made faster decision around 0.287 seconds then no eyes on the car.

Sensors used in an autonomous- car depends on its autonomy level as per the standards of SAE (table –2). There are a total of 6 levels of autonomy ranging from 0-5. Level 0 (Driver only) has no automation whereas level 5 refers to full autonomy in which the car performs all the Dynamic Driving Tasks (DDT) and also achieves the minimum risk condition (DDT fallback) [ 38 ]. A wide range of sensors are used until level 3 of autonomy, from level 4 sensors remain the same but the algorithms and processing of sensor fusion data achieve more autonomy. The U.S. Department of Transportation's National Highway Traffic Safety Administration (NHTSA) has adopted the standards set by the Society of Automotive Engineers (SAE) in the September of 2016.

Autonomous vehicle control (AVC) module is an important feature in autonomous cars that controls the behaviour and environments in various situations by controlling the hardware level component. AVC module is responsible for trajectory during motion by handling both predicted and unforeseen circumstances in autonomous cars. This module is also used in steering wheels to calculate the steering angle for the next action based on the control algorithm. This module is also responsible for speed control, distance travelled and emergency brakes. Apart from this module, the predictive control module is also used in autonomous cars to optimize the predicted motion for inter-vehicular communications. Immense amount of real-time data is required to communicate with neighbours and surroundings to stabilize the operations of autonomous cars. Autonomous cars need a powerful control mechanism to avoid transmission delays and communication errors that arise due to wireless communication. Zeng et al. [ 83 ] suggested an integrated control mechanism, where the communication delays were analysed to find the stability in autonomous platoons. The authors suggested to use the adaptive control mechanism to improve the reliability of the communication mechanism. The traditional control system is quite different then the control mechanism used in autonomous cars. The central control mechanism of autonomous cars is context oriented and uses adaptive control system to act immediately as the information of the context is highly required. Liu et al. [ 84 ] suggested a joint control communication mechanism to understand different road situations in autonomous cars.

Autonomous decision making

Autonomous decision making is a complex task in autonomous cars and certain cars are known as “ego-vehicles” which focus only on their local surroundings, current speed, direction and destination. But the communication mechanism must be strong enough in autonomous cars, so the concepts of “ego-vehicles” are fading away. So, there is a need of different prediction algorithms which gives predictions of high probability. And the final decision is a combined decision by taking the inputs from sensory data and other modules. But there is a number of other challenges related to environment, noise and limitations in the sensor data, mysterious state of neighbours. Therefore, autonomous cars should have accurate information about the neighbours in order to make predictions with high probability. But sometimes the neighbour information is neither available in public nor shared and this restriction causes serious problems for doing predictions and taking decisions accurately. Mockery of human behaviour in autonomous cars is very difficult, therefore the decision-making process is challenging. Numerous factors affect the decision-making process in autonomous cars such as behaviour of the car, prediction and perception, information about the neighbours, data processing done by sensors, calibration of equipment’s A number of present decision-making mechanisms are based on machine/deep learning, AI (Artificial Intelligence), Markov decision process. Hubmann et al. [ 85 ] highlighted the decision-making process for uncertain prediction of the surrounding vehicles from the sensor data. Many such AI based decision making approaches have been discussed by Claussmann et al. [ 86 ]. Several external factors impact the decision-making process in autonomous cars and these issues are addressed in a comprehensive way.

Issues such as ego-motion and inter-vehicle communication play a key role in crowd sensing and perception of neighbour’s behaviour. For commercializing autonomous cars in future, a major role will be played by connected car technology and it is seen today in many high-end cars [ 87 ]. The cloud infrastructure connected to the autonomous-car can also provide more computational resources for the car to execute its software processing functions, ensuring the car functions even in case of a hardware failure. Cloud infrastructure can also provide many new applications including entertainment [ 88 ]. But few issues arise while using the cloud services such as communication delay between cloud infrastructure and autonomous car. Any kind of delay can’t be tolerated by the decision-making module of autonomous cars due to cloud infrastructure, hence mostly decision-making is done locally in real-time environments. Similarly, fog computing techniques are also implemented which gives low delays in providing real-time services [ 89 ].

Testing of autonomous cars in real-time

In today’s scenario, numerous tests of autonomous cars have been carried out to evaluate its performance. Campbell et al. [ 11 ] took part in the DARPA grand challenge (DGC) and conducted 3 rounds of test for autonomous cars and documented the results from those tests. Campbell tried to incorporate the autonomous car mechanism in a regular car and tested it without any human intervention. In DGC challenge a route network definition file (RNDF) was given for self-driving using a road map to follow [ 90 ]. This challenge helped Campbell to understand the problems and issues that need to be addressed and how to commercialize the vehicle. Endsley et al. [ 91 ] considered Tesla autonomous car Model S for a period of 6 months and analysed different issues related to situation awareness, response to unanticipated circumstances on the road He also studied a number of other parameters related to consumer behaviour, trust, complexity, interface design and feedback from the customers. Broggi et al. [ 92 ] invented a BRAiVE (Brain Drive) system to conduct tests on autonomous car at Intelligent Systems and Artificial Vision Lab. Broggi’s test accumulated a huge amount of test data of real-time driving that is now used for upgrading the existing autonomous cars. Further Broggi tested autonomous cars on streets and he named his project as PROUD. Jo et al. [ 54 , 55 ] designed an autonomous car and participated in South Korea in 2012. Jo developed the architecture of the car and experimented in various environments.

AUTOSTAR [ 93 ] is one of the open standard architectures popularly used in many autonomous cars, but it has its limitations due to high cost and high complexity to implement. Later AUTOSTAR-lite came into market as a replacement to AUTOSTAR. Jo tried a distributed architecture instead of a centralized architecture to reduce the complexity and to group the functional components. This architecture also increased the efficiency and performance through parallel computations. A number of automotive industries along with academia experimented and “ Drive Me ” is one of the special projects done by Volvo in autonomous cars. This project in Sweden collected information from 100 consumers about their daily routines, driving behaviour, their preferences and other important aspects of driving. This consumer data helped researchers to improve the commercial autonomous cars. Blockchain technology can also be used for shared training of autonomous-cars where cars can share their experience over a blockchain network [ 27 ]. Blockchain technology can also be used to charge electric vehicles with no human intervention as shown in [ 28 ] with the help of a public ledger recording each transaction.

Design and implementation issues

A number of factors such as safety, robustness, hardware / software designs, customer behaviour will decide the future of autonomous cars. And also, these cars need to be designed with utmost precision, safety and reliability features [ 94 ]. The major components of autonomous cars are LIDAR, sensors, radar, positioning systems and various optimized software’s. A number of issues have direct impact on the autonomous car industry such as cost, maps, software complexity and simulation. First, the software/hardware cost is a major barrier in manufacturing autonomous cars. LIDAR is one of the expensive products in the car. Second, the maps used in autonomous cars contain a number of road details and it differs from the normal maps generated by the GPS systems. Memory requirements and processing power are enormous to store all the road details. The log files generated from these cars also requires memory to store and it contains detailed information about localization and mapping. Third, the software program of the car decides the various operations of the car such as move, stop, and lane change and overtake. So, a highly accurate and reliable software program is required which takes inputs from different sensors [ 95 ]. The huge data acquired from the sensors, environment and its surroundings create a real challenge for the autonomous car [ 96 ]. The algorithms processing this huge amount of sensor data should be efficient as well. An example stated in [ 38 ] is the use of stereo-camera over normal camera. A stereo-camera can take 3D images that map the environment accurately. Even though the data generated by stereo-camera is huge, it takes less time to process this data then conventional image processing.

Autonomous cars still need to be tested in adverse conditions such as mist, rainstorms, night-time and densely populated cities. The software used in autonomous cars also records the driving patterns and behaviours such as obstacle management, pedestrians crossing, and overtaking. Simulation technology is becoming the self-driving technology for autonomous car designers and Google is one among the top leading company in the market along with hardware giants like Nvidia announcing the launch of support hardware for autonomous cars [ 8 ]. To check the software reliability and safety, large-scale simulation is necessary. After simulation it is tested on real hardware with all built-in functionalities. Simulation tools developed specifically for the requirement of autonomous vehicles are utilized to simulate diverse aspects such as path planning and testing, mobility dynamics, fuel economy in urban scenarios [ 97 ]. Sajjad et al. [ 49 ] proposed an efficient and scalable simulation model for autonomous vehicles with economical hardware. A reduced reality gap for testing autonomous vehicles has been proposed by Patel et al. [ 98 ]. The simulation tool designed by Buzdugan et al. [ 34 ] provides a realistic environment for students to learn about the behaviour of autonomous vehicles and how they interact with their surroundings. The authors also evaluated the effectiveness of the tool in teaching the behaviour of autonomous vehicles. The study can help in the development of new methods of teaching autonomous vehicles and to improve the understanding of the behaviour of autonomous vehicles. When such simulation and testing models are combined, the industry can use various models for training autonomous cars both efficiently accurately.

Challenges for autonomous car deployment

A number of challenges still exist and it must be resolved by various stakeholders, manufacturers, developers, academicians, policy makers and designers [ 99 ]. These challenges can be categorized into technical, non-technical, social and policy related. A number of technical challenges arise during car deployment such as validation and testing, hardware / software resources, quality, safety, privacy and security. Validation and testing are time consuming process and it varies from model to model and it also depends on the degree of sophistication. Different of testing techniques are used such as simple bug fixing to quality testing. A number of safety and mission-critical testing are performed to fine tune the performance of autonomous cars and to check whether the rigorous necessities are met or not. Koopman et al. conducted widespread overview of the validation and testing trials for autonomous cars when deployed at scale [ 100 ]. The author applied ISO 26262 development V process which maps each design of autonomous vehicles to an appropriate testing method while focusing more on specific advanced challenges and divided the work into 3 phases namely phased deployment, monitor architecture, and fault inoculation. This testing framework is popularly used in in automotive industry for validation and testing and it checks the specific requirements of vehicles minutely and test all expected functionalities. Autonomous cars have specific set of complex requirements, and it is different from traditional validation and testing process. The author also discussed the challenges in the testing of autonomous vehicles as vehicle level testing is not thorough enough to ensure systems with ultimate dependencies which, in this case, are the various technologies employed in autonomous cars discussed in the prior sections. Younang et al. [ 37 ] examined how the concerns of users and government policies regarding autonomous vehicles compare and contrast. They analysed data from surveys and government policy documents to understand user's concerns and government policies on autonomous vehicles (Fig. 5 ).

His image is a flowchart that classifies the challenges faced in the development of autonomous cars into four categories: Technical, Non-technical, Social, and Policy. The chart illustrates the complex nature of autonomous car development and the various factors that must be considered. The Technical challenges include issues related to sensor technology, machine learning, and artificial intelligence. This flowchart is a useful tool to understand and address the different challenges that autonomous car development faces, and it helps to identify the areas that need further research and development

Safety and reliability are the second core aspects to be addressed in autonomous cars. The degree of reliability can be identified to some extent by statistical analysis by using a huge database of distance travelled by the car. Safety is one of the major concerns for autonomous cars, so a number of new methods are required for measuring the reliability parameter. Similarly, legislation is another prime challenge in autonomous cars. Safety is an important and interdisciplinary issue, so a lot amount time and money are spent on safety measures of the car [ 101 ].

Software quality is third aspect in autonomous cars as the car is run by complex and sophisticated software. Validation and testing of these software’s play a major role due to mission-critical and complex software system used by these cars. These software’s responds to a number of unanticipated situations and in all situations, it must be fail-safe.

Computational resource requirement is the fourth aspect in autonomous cars due to a number of high-resolution cameras used for monitoring various operations of the car. Apart from that a number of sensors are used for object recognition such as LIDAR. So autonomous cars require GPU processors with FPGA (field-programmable gate array) and SoC (system on a chip) to meet the functional and operational requirements of the car. Optionally these can be connected to a cloud infrastructure which provides the necessary computational power as a service. But in case of a connection timeout, it can cause problems. A number of sophisticated algorithms are used for data processing and provide an efficient and reliable system.

Security requirement is the fifth and one of the important aspects in autonomous cars. The data shared between various components and among other vehicles and infrastructure must be kept safe and inaccessible to unauthorized people. All type of communication information must be safe from hackers, so a number of cryptographic techniques are used to provide internal security and privacy.

Automotive industry is fast changing and a number of auto maker’s companies are now making autonomous cars. A number of new business opportunities are there for car makers along with few challenges such as safety for car and passengers. In this paper, we discussed the current solutions regarding design, operation issues, and forthcoming challenges. A number of major benefits and technical challenges are thoroughly discussed. A number of research areas to be focussed for autonomous cars have been outlined such as computer vision using deep learning, decision-making using machine learning, navigation, planning, control, perception, blockchain technology, cloud infrastructure integration, A number of real-life tests conducted so far are also outlined on autonomous cars. This work has focussed on the recent technologies behind autonomous cars like AI decision-making-based path planning, layered architectures, cyber threat protection mechanism, use of blockchain for transaction management and shared training of autonomous vehicles and also provides a valuable reference to the field of autonomous cars. This paper focussed on a number of technical and non-technical issues that exist in the design and implementation, including the analysis of the consumer and driver preferences, of autonomous cars. A number of issues were reviewed regarding object tracking and detection, data acquisition from sensors, safety and reliability, decision support, security, simulation models and privacy. Even though notable results are achieved in the research and development of autonomous cars and now that we have entered the commercialization phase, but still this field still has several areas of research like user experience and precautions and hardware for efficient decision making on the vehicle. The future of autonomous cars is promising, but further research and development is needed to overcome the remaining obstacles like autonomous vehicle testing for component compatibility, accurate simulations, large scale training and to fully realize their potential. The paper concludes with a call for further research in the field of autonomous cars to address the remaining technical and non-technical issues and to maximize the potential and adoption of autonomous cars in the future.

Availability of data and materials

Not applicable. For any collaboration, please contact the authors.

Lafrance A. Our grandmother’s driverless car. https://doi.org/https://www.theatlantic.com/technology/archive/2016/06/beep-beep/489029/ ; 2016.

Kanade T, Thorpe C, Whittaker W. Autonomous land vehicle project at cmu. In: Proceedings of the 1986 ACM Fourteenth Annual Conference on Computer Science, CSC ’86, (New York, NY, USA), pp. 71–80, ACM, 1986.

Schmidhuber J. Robot car history. https://doi.org/http://people.idsia.ch/~juergen/robotcars.html .

Guerrero-ibanez JA, Zeadally S, Contreras-Castillo J. Integration challenges of intelligent transportation systems with connected vehicle, cloud computing, and internet of things technologies. IEEE Wirel Commun. 2015;22:122–8.

Article   Google Scholar  

Contreras-Castillo J, Zeadally S, IbÃaÃsez JAG. A seven layered model architecture for internet of vehicles. J Inf Telecommun. 2017;1(1):4–22.

Kenney JB. Dedicated short-range communications (dsrc) standards in the United States. Proc IEEE. 2011;99:1162–82.

Zeadally S, Hunt R, Chen Y-S, Irwin A, Hassan A. Vehicular ad-hoc networks (vanets): status, results, and challenges. Telecommun Syst. 2012;50:217–41.

Lopez N. Nvidia announces a ’supercomputer’ gpu and deep learning platform for self-driving cars.” https://doi.org/https://thenextweb.com/author/napierlopez/#.tnw_G6F0jhzi , 2016.

P. Group, “Two psa group autonomous cars drive from Paris to Amsterdam in "eyes off" mode.” http://www.businesswire.com/news/home/20160414006039/en/PSA-Group-Autonomous-Cars-Drive-Paris-Amsterdam , 2016.

Mehar S, Zeadally S, Ralmy G, Senouci SM. Sustainable transportation management system for a fleet of electric vehicles. IEEE Trans Intell Transport Syst. 2015;16:1401–14.

Campbell M, Egerstedt M, How JP, Murray RM. Autonomous driving in urban environments: approaches, lessons and challenges. Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences. 2010;368(1928):4649–72.

Google Scholar  

Okuda R, Kajiwara Y, Terashima K. A survey of technical trend of adas and autonomous driving. In: Proceedings of Technical Program - 2014 International Symposium on VLSI Technology, Systems and Application (VLSI-TSA), pp. 1–4, 2014.

Fagnant DJ, Kockelman K. Preparing a nation for autonomous vehicles: opportunities, barriers and policy recommendations. Transportation Research Part A: Policy and Practice. 2015;77:167–81.

Bagloee SA, Tavana M, Asadi M, Oliver T. Autonomous vehicles: challenges, opportunities, and future implications for transportation policies. Journal of Modern Transportation. 2016;24:284–303.

Paden B, Änãap M, Yong SZ, Yershov D, Frazzoli E. A survey of motion planning and control techniques for self-driving urban vehicles. IEEE Trans Intell Vehicles. 2016;1:33–55.

Abraham H, Lee C, Brady S, Fitzgerald C, Mehler B, Reimer B, Coughlin JF. White paper: Autonomous vehicles, trust, and driving alternatives: A survey of consumer preferences. Tech. Rep. 2016–6, MIT AgeLab, Massachusetts Institute of Technology, Cambridge, MA; 2016.

Joy J, Gerla M. Internet of vehicles and autonomous connected car - privacy and security issues. In: 2017 26th International Conference on Computer Communication and Networks (ICCCN), pp. 1–9, 2017.

Bresson G, Alsayed Z, Yu L, Glaser S. Simultaneous localization and mapping: A survey of current trends in autonomous driving. IEEE Transactions on Intelligent Vehicles. 2017;2:194–220.

Parkinson S, Ward P, Wilson K, Miller J. Cyber threats facing autonomous and connected vehicles: Future challenges. IEEE Trans Intell Transport Syst. 2017;99:1–18.

Shi W, Alawieh MB, Li X, Yu H. Algorithm and hardware implementation for visual perception system in autonomous vehicle: A survey. Integr VLSI J. 2017;59:148–56.

Hulse LM, Xie H, Galea ER. Perceptions of autonomous vehicles: Relationships with road users, risk, gender and age. Saf Sci. 2018;102:1–13.

Gupta AS, Sharma S. Analysis of Public Perception of Autonomous Vehicles Based on Unlabelled Data from Twitter. In: Tuba M, Akashe S, Joshi A, editors. ICT Infrastructure and Computing Lecture Notes in Networks and Systems, vol. 520. Singapore: Springer; 2023.

Madhav, A.V.S., Tyagi, A.K. (2023). Explainable Artificial Intelligence (XAI): Connecting Artificial Decision-Making and Human Trust in Autonomous Vehicles. In: Singh, P.K., Wierzchoń, S.T., Tanwar, S., Rodrigues, J.J.P., Ganzha, M. (eds). Proceedings of Third International Conference on Computing, Communications, and Cyber-Security Lecture Notes in Networks and Systems. Springer, Singapore.

Bairy A. (2022). Modeling Explanations in Autonomous Vehicles. In: ter Beek, M.H., Monahan, R. (eds) Integrated Formal Methods. IFM 2022. Lecture Notes in Computer Science, vol 13274. Springer, Cham. https://doi.org/10.1007/978-3-031-07727-2_20

Mazri T, Tibari S. The Proposed Self-defense Mechanism Against Security Attacks for Autonomous Vehicles. In: Ben Ahmed M, Boudhir AA, Karaș İR, Jain V, Mellouli S, editors. Innovations in Smart Cities Applications Volume 5 SCA 2021 Lecture Notes in Networks and Systems. Cham: Springer; 2022.

Li Q, Wang Z, Wang W, Yuan Q. Understanding Driver Preferences for Secondary Tasks in Highly Autonomous Vehicles. In: Long S, Dhillon BS, editors. Man-Machine-Environment System Engineering. MMESE 2022. Lecture Notes in Electrical Engineering, vol. 941. Singapore: Springer; 2023.

Gandhi GM, Salvi. Artificial Intelligence Integrated Blockchain For Training Autonomous Cars. In: 2019 Fifth International Conference on Science Technology Engineering and Mathematics (ICONSTEM), 2019, pp. 157–161, https://doi.org/10.1109/ICONSTEM.2019.8918795 .

Aguilar Cisneros JR, Fernández-y-Fernández CA, Juárez Vázquez J. Blockchain Software System Proposal Applied to Electric Self-driving Cars Charging Stations: A TSP Academic Project. In: 2020 8th International Conference in Software Engineering Research and Innovation (CONISOFT), 2020, pp. 174–179, https://doi.org/10.1109/CONISOFT50191.2020.00033 .

Liu B, Betancourt VP, Zhu Y, Becker J. Towards an On-Demand Redundancy Concept for Autonomous Vehicle Functions using Microservice Architecture. IEEE International Symposium on Systems Engineering (ISSE). 2020;2020:1–5. https://doi.org/10.1109/ISSE49799.2020.9272016 .

Qiu H, Ayara A, Glimm B. A Knowledge Architecture Layer for Map Data in Autonomous Vehicles. In: 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), 2020, pp. 1–6, https://doi.org/10.1109/ITSC45102.2020.9294712 .

Coicheci S, Filip I. Self-driving vehicles: current status of development and technical challenges to overcome. In: 2020 IEEE 14th International Symposium on Applied Computational Intelligence and Informatics (SACI), 2020, pp. 000255–000260, https://doi.org/10.1109/SACI49304.2020.9118809 .

Zelle D, Rieke R, Plappert C, Krauß C, Levshun D, Chechulin A. SEPAD – Security Evaluation Platform for Autonomous Driving. In: 2020 28th Euromicro International Conference on Parallel, Distributed and Network-Based Processing (PDP), 2020, pp. 413–420, https://doi.org/10.1109/PDP50117.2020.00070 .

Li H, Wu C, Chu D, Lu L, Cheng K. Combined Trajectory Planning and Tracking for Autonomous Vehicle Considering Driving Styles. IEEE Access. 2021;9:9453–63. https://doi.org/10.1109/ACCESS.2021.3050005 .

Buzdugan ID, Roșu IA, Antonya C. Development of a Simulator Tool for Teaching the Autonomous Vehicles Behavior. In: Auer ME, El-Seoud SA, Karam OH (eds) Artificial Intelligence and Online Engineering. REV 2022. Lecture Notes in Networks and Systems. Springer, Cham. 2013.

Tam PM, Anh HPH. A Probability-Based Artificial Potential Field for Autonomous Vehicles in Avoiding Uncertain Obstacles. In: Huang YP, Wang WJ, Quoc HA, Le HG, Quach HN, editors. Computational Intelligence Methods for Green Technology and Sustainable Development GTSD 2022 Lecture Notes in Networks and Systems. Cham: Springer; 2023.

Qiao J, de Jonge D, Zhang D, Sierra C, Simoff S. A Hybrid Model of Traffic Assignment and Control for Autonomous Vehicles. In: Aydoğan R, Criado N, Lang J, Sanchez-Anguix V, Serramia M, editors. PRIMA 2022: Principles and Practice of Multi-Agent Systems. PRIMA 2022. Lecture Notes in Computer Science. Cham: Springer; 2023.

Wakam Younang VC, Yang J, Jacuinde LG, Sen A. A Comparative Analysis of User’s Concerns and Government Policies on Autonomous Vehicles. In: Tekinerdogan B, Wang Y, Zhang LJ, editors. Internet of Things –ICIOT 2022 Lecture Notes in Computer Science. Cham: Springer; 2023.

Zanchin BC, Adamshuk R, Santos MM, Collazos KS. On the instrumentation and classification of autonomous cars. 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2017, pp. 2631–2636, https://doi.org/10.1109/SMC.2017.8123022 .

Preliminary statement of policy concerning automated vehicles", NHTSA. https://doi.org/https://www.transportation.gov/briefing-room/us-department-transportation-releases-policy-automated-vehicle-development .

Domínguez R, Onieva E, Alonso J, Villagra J, González C. LIDAR based perception solution for autonomous vehicles. In: Intelligent Systems Design and Applications (ISDA), 2011 11th International Conference on, 2011, p. 790–795.

Hasch, E.Topak, R. Schnabel, T. Zwick, R.Weigel, and C.Waldschmidt, “Millimeter-wave technology for automotive radar sensors in the 77 GHz frequency band,” IEEE Transactions on Microwave Theory and Techniques, vol. 60, no. 3, pp. 845–860, 2012

Fu M, Song W, Yi Y, Wang M. Path planning and decision making for autonomous vehicle in urban environment. In: 2015 IEEE 18th International Conference on Intelligent Transportation Systems, pp. 686–692, 2015.

Lee M-H, Chen Y-J, Li THS. Sensor fusion design for navigation and control of an autonomous vehicle. In: Systems, Man, and Cybernetics (SMC), 2011 IEEE International Conference on, 2011, p. 2209–2214.

“Critical reasons for crashes investigated in the national motor vehicle crash causation survey.” https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/812115 , 2015. [Online].

Anderson MJ, Nidhi K, Karlyn DS, Sorensen P, Samaras C, Oluwatola OA. Autonomous vehicle technology: A guide for policymakers. In: RAND Corporation; 2016.

Heydari S, Fajri P, Sabzehgar R, Asrari A. Optimal Blending of Regenerative and Friction Braking at Low Speeds for Maximizing Energy Extraction in Electric Vehicles. IEEE Energy Conversion Congress and Exposition (ECCE). 2019;2019:6815–9. https://doi.org/10.1109/ECCE.2019.8913117 .

Mansour MBM, Said A, Ahmed NE, Sallam S. Autonomous Parallel Car Parking. In: 2020 Fourth World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4), 2020, pp. 392-397.

Pokhrel SR, Qu Y, Nepal S, Singh S. Privacy-Aware Autonomous Valet Parking: Towards Experience Driven Approach. IEEE Trans Intell Transp Syst. 2021;22(8):5352–63. https://doi.org/10.1109/TITS.2020.3006337 .

Sajjad M, et al. An Efficient and Scalable Simulation Model for Autonomous Vehicles With Economical Hardware. IEEE Trans Intell Transp Syst. 2021;22(3):1718–32. https://doi.org/10.1109/TITS.2020.2980855 .

Kim HS, Yoon HS, Kim MJ, Ji YG. Deriving future user experiences in autonomous vehicle. In: Adjunct Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, AutomotiveUI ’15, (New York, NY, USA), pp. 112–117, ACM, 2015.

Pierce D. Tesla summon hints at how the world of self-driving cars will work. https://doi.org/https://www.wired.com/2016/01/tesla-summon/ , 2016.

Li L, Liu Y, Wang J, Deng W, Oh H. Human dynamics based driver model for autonomous car. IET Intel Transport Syst. 2016;10(8):545–54.

Broggi A, Cerri P, Debattisti S, Laghi MC, Medici P, Molinari D, Panciroli M, Prioletti A. Proud: Public road urban driverless-car test. IEEE Trans Intell Transp Syst. 2015;16:3508–19.

Jo K, Kim J, Kim D, Jang C, Sunwoo M. Development of autonomous car (part i): Distributed system architecture and development process. IEEE Trans Industr Electron. 2014;61:7131–40.

Jo K, Kim J, Kim D, Jang C, Sunwoo M. Development of autonomous car (part ii): A case study on the implementation of an autonomous driving system based on distributed architecture. IEEE Trans Industr Electron. 2015;62:5119–32.

Woo S, Jo HJ, Lee DH. A practical wireless attack on the connected car and security protocol for in-vehicle can. IEEE Trans Intell Transp Syst. 2015;16:993–1006.

Woo S, Jo HJ, Kim IS, Lee DH. A practical security architecture for in-vehicle can-fd. IEEE Trans Intell Transp Syst. 2016;17:2248–61.

Janai J, Güney F, Behl A, Geiger A. Computer vision for autonomous vehicles: Problems, datasets and state-of-the-art. CoRR, vol. abs/1704.05519, 2017.

Chen X, Kundu K, Zhu Y, Ma H, Fidler S, Urtasun R. 3d object proposals using stereo imagery for accurate object class detection. CoRR, vol. abs/1608.07711, 2016.

Gonzalez A, Vãazquez D, Lãspez AM, Amores J. On-board object detection: Multicue, multimodal, and multiview random forest of local experts. IEEE Trans Cybern. 2017;47:3980–90.

Chen X, Ma H, Wan J, Li B, Xia T. Multi-view 3d object detection network for autonomous driving. CoRR, abs/1611.07759, 2016.

Baek J, Kim J, Kim E. Fast and efficient pedestrian detection via the cascade implementation of an additive kernel support vector machine. IEEE Trans Intell Transp Syst. 2017;18:902–16.

Bilal M. Algorithmic optimisation of histogram intersection kernel support vector machine-based pedestrian detection using low complexity features. IET Comput Vision. 2017;11(5):350–7.

Hattori H, Boddeti VN, Kitani K, Kanade T. Learning scene-specific pedestrian detectors without real data. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3819–3827, 2015.

Sermanet P, Kavukcuoglu K, Chintala S, Lecun Y. Pedestrian detection with unsupervised multi-stage feature learning. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3626–3633, 2013.

Xu D, Ouyang W, Ricci E, Wang X, Sebe N. Learning cross-modal deep representations for robust pedestrian detection. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4236–4244, 2017.

Luo W, Schwing AG, Urtasun R. Efficient deep learning for stereo matching. in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5695–5703, 2016.

Mayer N, Ilg E, HÃdusser P, Fischer P, Cremers D, Dosovitskiy A, Brox T. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4040–4048, 2016.

Tian Y, Pei K, Jana S, Ray B. Deeptest: Automated testing of deep-neural-network-driven autonomous cars. CoRR, vol. abs/1708.08559, 2017.

Chen C, Seff A, Kornhauser A, Xiao J. Deepdriving: Learning affordance for direct perception in autonomous driving. In: 2015 IEEE International Conference on Computer Vision (ICCV), pp. 2722–2730, 2015.

Islam MM, Newaz R, Gokaraju B, Karimoddini A. Pedestrian Detection for Autonomous Cars: Occlusion Handling by Classifying Body Parts. In: 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2020, pp. 1433–1438, https://doi.org/10.1109/SMC42975.2020.9282839 .

Ikhlayel M, Iswara AJ, Kurniawan A, Zaini A, Yuniarno EM. Traffic Sign Detection for Navigation of Autonomous Car Prototype using Convolutional Neural Network. In: 2020 International Conference on Computer Engineering, Network, and Intelligent Multimedia (CENIM), 2020, pp. 205–210, https://doi.org/10.1109/CENIM51130.2020.9297973 .

Al-Qizwini M, Barjasteh N, Al-Qassab H, Radha H. Deep learning algorithm for autonomous driving using googlenet. in 2017 IEEE Intelligent Vehicles Symposium (IV), pp. 89–96, 2017.

Laddha A, Kocamaz MK, Navarro-Serment LE, Hebert M. Map-supervised Road detection. In: 2016 IEEE Intelligent Vehicles Symposium (IV), pp. 118–123, 2016.

Xu H, Gao Y, Yu F, Darrell T. End-to-end learning of driving models from large-scale video datasets. CoRR, vol. abs/1612.01079, 2016.

Daftry S, Bagnell JA, Hebert M. Learning transferable policies for monocular reactive (mav) control. CoRR, vol. abs/1608.00627, 2016.

Milioto, Andres & Behley, Jens & Mccool, Chris & Stachniss, Cyrill. (2020). LiDAR Panoptic Segmentation for Autonomous Driving. https://doi.org/10.1109/IROS45743.2020.9340837 .

Oliveira M, Santos V, Sappa AD, Dias P, Moreira AP. Incremental scenario representations for autonomous driving using geometric polygonal primitives. Robot Auton Syst. 2016;83:312–25.

Xiao L, Wang R, Dai B, Fang Y, Liu D, Wu T. Hybrid conditional random field based camera-lidar fusion for road detection. Inf Sci. 2018;432:543–58.

Article   MathSciNet   Google Scholar  

Hobert L, Festag A, Llatser I, Altomare L, Visintainer F, Kovacs A. Enhancements of v2x communication in support of cooperative autonomous driving. IEEE Commun Mag. 2015;53:64–70.

Peng H, Li D, Abboud K, Zhou H, Zhao H, Zhuang W, Shen X. Performance analysis of ieee 802.11p dcf for multiplatooning communications with autonomous vehicles. IEEE Trans Veh Technol. 2017;66:2485–98.

Chang C-M, Toda K, Sakamoto D, Igarashi T. Eyes on a car: An interface design for communication between an autonomous car and a pedestrian. In: Proceedings of the 9th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, AutomotiveUI ’17, (New York, NY, USA), pp. 65–73, ACM, 2017.

Zeng T, Semiari O, Saad W, Bennis M. Joint communication and control for wireless autonomous vehicular platoon systems. CoRR, vol. abs/1804.05290, 2018.

Liu J, Zhang S, Sun W, Shi Y. In-vehicle network attacks and countermeasures: Challenges and future directions. IEEE Network. 2017;31(5):50–8.

Hubmann C, Becker M, Althoff D, Lenz D, Stiller C. Decision making for autonomous driving considering interaction and uncertain prediction of surrounding vehicles. In: 2017 IEEE Intelligent Vehicles Symposium (IV), pp. 1671–1678, 2017.

Claussmann L, Revilloud M, Glaser S, Gruyer D. A study on al-based approaches for high-level decision making in highway autonomous driving. In: 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 3671–3676, 2017.

Ilas C. Electronic sensing technologies for autonomous ground vehicles: A review. In: Advanced Topics in Electrical Engineering (ATEE), 2013 8th International Symposium on, 2013, p. 1–6.

Hussain R, Rezaeifar Z, Oh H. A paradigm shift from vehicular ad hoc networks to vanet-based clouds. Wireless Pers Commun. 2015;83(2):1131–58.

Aazam M, Zeadally S, Harras K. Fog computing A ¸S architecture, evaluation, and future research directions. IEEE Communications Magaz (in press), 2018.

Knight W. The Future of self-driving cars. MIT Technology Review, Massachusetts Institute of Technology, 2013.

Endsley MR. Autonomous driving systems: A preliminary naturalistic study of the tesla models. Journal of Cognitive Engineering and Decision Making. 2017;11(3):225–38.

Broggi A, Buzzoni M, Debattisti S, Grisleri P, Laghi MC, Medici P, Versari P. Extensive tests of autonomous driving technologies. IEEE Trans Intell Transp Syst. 2013;14:1403–15.

Bunzel S. Autosar – the standardized software architecture. Informatik-Spektrum. 2011;34:79–83.

Chakraborty S, Laware H, Castanon D, Zekavat SR. High precision localization for autonomous vehicles via multiple sensors, data fusion and novel wireless technologies. In: Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON), IEEE Annual, 2016, p. 1–9.

Li Q, Chen L, Li M, Shaw S-L, Nuchter A. A Sensor-Fusion Drivable-Region and Lane-Detection System for Autonomous Vehicle Navigation in Challenging Road Scenarios”. IEEE Trans Vehic Technol. 2014;63(2):540–55.

Google, “Waymo: On the road.” https://waymo.com/ontheroad/ , 2017.

Figueiredo MC, Rossetti RJF, Braga RAM, Reis LP. An approach to simulate autonomous vehicles in urban traffic scenarios. In: 2009 12th International IEEE Conference on Intelligent Transportation Systems, pp. 1–6, 2009.

Kaival Kamleshkumar P. A Simulation Environment with Reduced Reality Gap for Testing Autonomous Vehicles. 2020. Electronic Theses and Dissertations. 8305.

The Road Ahead: The Emerging Policy Debates for IT in Vehicles", Information Technology & Innovation Foundation, https://doi.org/http://www2.itif.org/2013-road-ahead.pdf .

Koopman P, Wagner M. Challenges in autonomous vehicle testing and validation. SAE Int. J. Trans. Safety, vol. 4, pp. 15–24, 2016.

de Lemos R, et al., Software Engineering for Self-Adaptive Systems: A Second Research Roadmap, pp. 1–32. Berlin: Springer, 2013.

Download references

Acknowledgements

We would like to acknowledge Prof. G Chandrasekhar, Prof. E Krishna Rao Patro and Mr. Habeeb for their assistance in editing and proofreading the paper.

This research received no specific grant from any funding agency. There is no funding information available for this research work.

Author information

Authors and affiliations.

Department of Computer Science and Engineering, Institute of Aeronautical Engineering, Hyderabad, Telangana, India

Department of Mechanical Engineering, Vasavi College of Engineering, Hyderabad, Telangana, India

CH. V. K. N. S. N. Moorthy

Engineering and Architecture Faculty, Nisantasi University, Istanbul, Turkey

Department of Computer Science and Engineering, G. Narayanamma Institute of Technology & Science (Autonomous), Hyderabad, Telangana, India

N. Venkateswarulu

VNR Vignana Jyothi Institute of Engineering and Technology, Hyderabad, Telangana, India

Myneni Madhu Bala

You can also search for this author in PubMed   Google Scholar

Contributions

BP has made substantial contributions towards literature survey and compiling the contents of this paper. CHVKNSNM has performed the analysis and manuscript preparation. NV has given significant contributions in writing and arranging the contents in proper order. MMB contributed to data collection and analysis on tools and techniques. All authors read and approved the final manuscript.

Authors' information

Dr. B Padmaja is currently working as an Associate Professor and Head, CSE (Artificial Intelligence and Machine Learning), Institute of Aeronautical Engineering, Hyderabad, Telangana, India. She has received her B.Tech from North Eastern Regional Institute of Science and Technology (NERIST), Arunachal Pradesh, India in May 2001. She completed her M.Tech from School of IT, JNTUH, and Hyderabad, India in 2010. She was awarded the Ph.D. degree in Computer Science and Engineering in 2021 by JNTUH, Hyderabad. She has vast teaching and research experience of 20 years. She has published more than 20 research papers in various International journals and presented more than 15 papers in various International conferences. She is also a reviewer for 08 International journals. Her current areas of research interest include Machine Learning, Deep Learning, Computer Vision, and Social Network Analysis. She is a life member of ISTE, CSI, IAENG and CSTA.

Prof. Dr. CH V K N S N Moorthy is working as Director R&D, Vasavi College of Engineering, Hyderabad, Telangana, India. He is a multidisciplinary and cross domain researcher having experience in the fields of Computer Science and Mechanical Engineering. He received Master of Technology both in the fields of Computer Science Engineering and Heat Power Refrigeration & Air Conditioning. He received Doctoral degree for research in the field of Thermo-Nano Fluid Heat Transfer from GITAM University, Vishakhapatnam and pursuing his Doctoral degree in the field of Machine Learning too. He has nearly two decades of teaching and research experience with a total research grant of 424.46 K USD from Department of Science and Technology, Ministry of Science and Technology, Government of India for various projects under cross domain research, more than 40 research publications, International Research Collaborations, Awards and Patents to his credit. He is a Chartered Engineer and Fellow Member of Institution of Engineers, India (IEI), a Life Member of Indian Society for Technical Education (ISTE), Member of American Society of Mechanical Engineers (ASME) and Institute of Electrical and Electronics Engineers (IEEE). His thrust areas of research include Cognitive Science, Data Analytics and Data Science, Machine Learning, Artificial Intelligence, Thermo-Nano fluid Heat Transfer, Nanotechnology, Carbon Nano Tubes, Computational Fluid Dynamics.

Mr. N. Venkateswarulu is working as Assistant Professor, Department of Computer Science and Engineering, G. Narayanamma Institute of Technology & Science, Shaikpet, Hyderabad, Telangana, India. He is a research scholar currently persuing Ph.D in KL University, Andhra Pradesh, India. He has received his master’s from Sree Vidyanikethan Engineering College, Tirupathi, Andhra Pradesh, India. He has received his bachelor’s degree from Sreenivasa Institute of Technology and Management Studies, Chittoor, Andra Pradesh, India in June 2003. He has total thirteen years of teaching experience. He has six research publications in international journals out of which one got indexed in SCOPUS and one got published in SCI journal JSIR in February 2023. His thrust areas of research include Computer Algorithms, Data Mining, Data Science, Machine Learning, and Artificial intelligence.

Dr Madhu Bala Myneni is working as a professor and Head of computer science and engineering at Institute of Aeronautical Engineering, Hyderabad. She received her Ph.D in Computer Science and Engineering from JNTUH. She has Twenty-one years of academic and research experience. Her research interests are Data Science frameworks, Image Mining, Text mining, Machine learning, Artificial Intelligence, Deep Learning and Data Analytics. She has published 57 articles in reputed Journals indexed in SCOPUS, SCI etc. She has published 2 patents. She is the Principal Investigator of a DST-funded sustainable smart city development project. And has received various grants from AICTE for organizing Short Term Training Programs; Infrastructure Development; and Faculty Development Programs. And selected a part of AICTE national mission programs such as Student Learning Outcomes Assessment (SLA); Technical Book Writing (TBW). She is a reviewer for Elsevier, Springer, and more indexed journals. She acted as session chair, organizing member, and advisory member for various International Conferences. She delivered various invited talks on Data Modelling, Data Science, and Analytics. She is a Life member of professional bodies like CSI and ISTE, Sr. Member for IEEE, WIE & International association IAENG, ICST, and SDIWC.

Corresponding author

Correspondence to B. Padmaja .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Padmaja, B., Moorthy, C.V.K.N.S.N., Venkateswarulu, N. et al. Exploration of issues, challenges and latest developments in autonomous cars. J Big Data 10 , 61 (2023). https://doi.org/10.1186/s40537-023-00701-y

Download citation

Received : 13 January 2022

Accepted : 17 February 2023

Published : 06 May 2023

DOI : https://doi.org/10.1186/s40537-023-00701-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Autonomous Cars
• Driverless Cars

A Review of a Research in Autonomous Vehicles with Embedded Systems

• Conference paper
• First Online: 20 February 2024
• Cite this conference paper

• Fulya Akdeniz 13 ,
• Mert Atay 13 ,
• Şule Vural 13 ,
• Burcu Kır Savaş 13 &
• Yaşar Becerikli 13  

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 906))

Included in the following conference series:

• The Proceedings of the International Conference on Smart City Applications

153 Accesses

Designing an embedded system for practical applications needs objectives such as high accuracy, low power consumption and cost, and secure environment. NVIDIA’s Jetson is the most popular platform for embedded system applications which promises to achieve a balance between all these objectives. This paper focused on autonomous vehicles and embedded system studies evaluating the Jetson card features used in smart cars. The characteristics of autonomous vehicles and presents the data was compared for researchers categorized by author names, vehicle names, platform, camera, purpose and environment. In paper shows that most of studies had different purposes and tested area in different environments. We also developed a cost-effective lane detection model for developing and testing the autonomous vehicle JetRacer AI Kit and detail the overall structure of the system in this work. We tested the lane detection performance for system feasibility. In addition, we presented our ongoing work to further develop the features of this vehicle based on the proposed framework.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Akdeniz, F., Kayikçioğlu, İ., Kaya, İ., Kayikçioğlu, T.: Using WignerVille distribution in ECG arrhythmia detection for telemedicine applications. In: 2016 39th International Conference on Telecommunications and Signal Processing (TSP), pp. 409–412. IEEE (2016)

Google Scholar  

DARPA Grand Challenge. https://en.wikipedia.org/wiki/DARPA_Grand_Challenge 2018–2021. Accessed 10 Jun 2022

Savaş, B.K., Becerikli, Y.: Real time driver fatigue detection system based on multi-task ConNN. IEEE Access 8 , 12491–12498 (2020)

Article   Google Scholar  

Stocco, A., Pulfer, B., Tonella, P.: Mind the Gap! A Study on the Transferability of Virtual vs Physical-world Testing of Autonomous Driving Systems. arXiv preprint arXiv:2112.11255 (2021)

Shi, W., Alawieh, M.B., Li, X., Yu, H.: Algorithm and hardware implementation for visual perception system in autonomous vehicle: a survey. Integration 59 , 148–156 (2017)

Bonnefon, J.F., Shariff, A., Rahwan, I.: The social dilemma of autonomous vehicles. Science 352 (6293), 1573–1576 (2016)

Akdeniz, F., Becerikli, Y.: Performance comparison of support vector machine, k-nearest-neighbor, artificial neural networks, and recurrent neural networks in gender recognition from voice signals. In: 2019 3rd International Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT), pp. 1–4, IEEE

Kır Savaş, B., Becerikli, Y.: Behavior-based driver fatigue detection system with deep belief network. Neural Comput. Appl. 2022 (2019)

Karabulut, G.: Mini autonomous car architecture for urban driving scenarios (Master’s thesis, Middle East Technical University) (2019)

Castro, O., Céspedes, A., Ubaldo, R., Ramos, O.E.: Lane following with a duckiebot vehicle using visual feedback. In: 2019 IEEE Sciences and Humanities International Research Conference (SHIRCON), pp. 1–4, IEEE (2019)

Roberts, A., Maennel, O., Snetkov, N.: Cybersecurity test range for autonomous vehicle shuttles. In: 2021 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW), pp. 239–248, IEEE (2021)

Kawakura, S., Shibasaki, R.: Deep learning-based self-driving car: JetBot with NVIDIA AI board to deliver items at agricultural workplace with object-finding and avoidance functions. Eur. J. Agric. Food Sci. 2 (3) (2020)

Bingöl, M.S., Kaymak, Ç., Ayşegül, U.Ç.A.R.: Derin Öğrenme Kullanarak Otonom Araçların İnsan Sürüşünden Öğrenmesi”. Fırat Üniversitesi Mühendislik Bilimleri Dergisi 31 (1), 177–185 (2019)

Yuhas, M., Feng, Y., Ng, D.J.X., Rahiminasab, Z., Easwaran, A.: Embedded out-of-distribution detection on an autonomous robot platform. In: Proceedings of the Workshop on Design Automation for CPS and IoT, pp. 13–18 (2021)

Loeminto, E.S., Khoswanto, H., Lim, R.: Pengenalan Traffic Light Pada Robot Mobil Duckietown. Teknika 7 (2), 89–93 (2018)

Kumar, C.S., Anjali, T., Rao, S.N.: A cost-effective framework for developing and testing autonomous RC cars. In: 2021 Sixth International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET), pp. 26–29, IEEE (2021)

Seth, A., James, A., Mukhopadhyay, S.C.: 1/10th scale autonomous vehicle based on convolutional neural network. Int. J. Smart Sens. Intell. Syst. 13 (1), 1–17 (2020)

Sandha, S.S., Garcia, L., Balaji, B., Anwar, F.M., Srivastava, M.: Sim2Real transfer for deep reinforcement learning with stochastic state transition delays. In: 4th Conference on Robot Learning, Cambridge MA, USA.CoRL (2020)

Katyal, K.D., Polevoy, A., Moore, J., Knuth, C., Popek, K.M.: Highspeed robot navigation using predicted occupancy maps. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 5476–5482, IEEE (2021)

O’Kelly, M., et al.: F1/10: An open-source autonomous cyber-physical platform (2019). arXiv preprint arXiv:1901.08567

O’Kelly, M., Zheng, H., Karthik, D., Mangharam, R.: F1tenth: an open-source evaluation environment for continuous control and reinforcement learning. Proc. Mach. Learn. Res. 123 , 77–89 (2020)

Liu, Y., Schofield, H., Shan, J.: Navigation of a self-driving vehicle using one fiducial marker. In: 2021 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), pp. 1–6, IEEE (2021)

Hu, J., Zhang, Y., Rakheja, S.: Adaptive trajectory tracking for car-like vehicles with input constraints. IEEE Trans. Ind. Electron. 69 (3), 2801–2810 (2021)

Sukhil, V., Behl, M.: Adaptive Lookahead Pure-Pursuit for Autonomous Racing (2021). arXiv preprint arXiv:2111.08873

Balaji, B., et al.: DeepRacer: educational autonomous racing platform for experimentation with sim2real reinforcement learning (2019). arXiv preprint arXiv:1911.01562

Chiba, S., Sasaoka, H.: Effectiveness of transfer learning in autonomous driving using model car. In: 2021 13th International Conference on Machine Learning and Computing, pp. 595–601 (2021)

Karaman, S., et al.: Project-based, collaborative, algorithmic robotics for high school students: programming self-driving race cars at MIT. In: 2017 IEEE Integrated STEM Education Conference (ISEC), pp. 195–203, IEEE (2017)

Scheffe, P., et al.: Networked and autonomous model-scale vehicles for experiments in research and education. IFAC-PapersOnLine 53 (2), 17332–17337 (2020)

Verschueren, R., De Bruyne, S., Zanon, M., Frasch, J.V., Diehl, M.: Towards time-optimal race car driving using nonlinear MPC in real-time. In: 53rd IEEE Conference on Decision and Control, pp. 2505–2510, IEEE (2014)

Hassanein, A.S., Mohammad, S., Sameer, M., Ragab, M.E.: A survey on Hough transform, theory, techniques and applications (2015). arXiv preprint arXiv:1502.02160

Chen, M.: Recognition and localization of target images for robot vision navigation control. J. Robot. 2022 , 8565913 (2022)

Macdonald, I.: Probabilistic Hough Transform. Notes, 1–4 (2016)

Stephens, R.S.: Probabilistic approach to the Hough transform. Image Vis. Comput. 9 (1), 66–71 (1991)

Cass, S.: Nvidia makes it easy to embed AI: the Jetson nano packs a lot of machine-learning power into DIY projects-[Hands on]. IEEE Spectr. 57 (7), 14–16 (2020)

Download references

Acknowledgements

The study was carried out in the Image Processing and Computer Vision Laboratory of the Computer Engineering Department of Kocaeli University, the study is supported by the BAP research and development center with Project ID 2337 and also supported by the TÜBİTAK 2209-A Project with project no:1919B012112773.

Author information

Authors and affiliations.

Kocaeli University, Umuttepe Campus, Kocaeli, 41001, Turkey

Fulya Akdeniz, Mert Atay, Şule Vural, Burcu Kır Savaş & Yaşar Becerikli

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Burcu Kır Savaş .

Editor information

Editors and affiliations.

Computer Science and Smart systems Laboratory, Faculty of Science and Techniques of Tangier, Abdelmalek Essaadi University, Tangier, Morocco

Mohamed Ben Ahmed

Anouar Abdelhakim Boudhir

École Spéciale des Travaux Publics, Paris, France

Rani El Meouche

Computer Engineering Department, Karabük University, Karabük, Türkiye

İsmail Rakıp Karaș

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Cite this paper.

Akdeniz, F., Atay, M., Vural, Ş., Savaş, B.K., Becerikli, Y. (2024). A Review of a Research in Autonomous Vehicles with Embedded Systems. In: Ben Ahmed, M., Boudhir, A.A., El Meouche, R., Karaș, İ.R. (eds) Innovations in Smart Cities Applications Volume 7. SCA 2023. Lecture Notes in Networks and Systems, vol 906. Springer, Cham. https://doi.org/10.1007/978-3-031-53824-7_21

Download citation

DOI : https://doi.org/10.1007/978-3-031-53824-7_21

Published : 20 February 2024

Publisher Name : Springer, Cham

Print ISBN : 978-3-031-53823-0

Online ISBN : 978-3-031-53824-7

eBook Packages : Engineering Engineering (R0)

Share this paper

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Help | Advanced Search

Computer Science > Robotics

Title: robust perception and navigation of autonomous surface vehicles in challenging environments.

Abstract: Research on coastal regions traditionally involves methods like manual sampling, monitoring buoys, and remote sensing, but these methods face challenges in spatially and temporally diverse regions of interest. Autonomous surface vehicles (ASVs) with artificial intelligence (AI) are being explored, and recognized by the International Maritime Organization (IMO) as vital for future ecosystem understanding. However, there is not yet a mature technology for autonomous environmental monitoring due to typically complex coastal situations: (1) many static (e.g., buoy, dock) and dynamic (e.g., boats) obstacles not compliant with the rules of the road (COLREGs); (2) uncharted or uncertain information (e.g., non-updated nautical chart); and (3) high-cost ASVs not accessible to the community and citizen science while resulting in technology illiteracy. To address the above challenges, my research involves both system and algorithmic development: (1) a robotic boat system for stable and reliable in-water monitoring, (2) maritime perception to detect and track obstacles (such as buoys, and boats), and (3) navigational decision-making with multiple-obstacle avoidance and multi-objective optimization.

Submission history

Access paper:.

• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

Special Features

Vendor voice.

Personal Tech

Two big computer vision papers boost prospect of safer self-driving vehicles

New chip and camera technology bring closer potential of hands-free road time.

Like nuclear fusion and jet-packs, the self-driving car is a long-promised technology that has stalled for years - yet armed with research, boffins think they have created potential improvements.

Citizens of Phoenix, San Francisco, and Los Angeles are able to take one of Waymo's self-driving taxis, first introduced to the public in December 2020. But they have not been without their glitches. Just last month in San Francisco, for example, one of the taxi service's autonomous vehicles drove down the wrong side of the street to pass a unicycle. In December last year, a Waymo vehicle hit a backwards-facing pickup truck , resulting in a report with the US National Highway Traffic Safety Administration (NHTSA) and a software update.

But this week, not one but two groups of researchers bidding to improve the performance of self-driving cars and other autonomous vehicles have published papers in the international science journal Nature.

A design for a new chip geared towards autonomous vehicles has arrived from China. Tsinghua University's Luping Shi and colleagues have taken inspiration from the human visual system by both combining low-accuracy, fast event-based detection with more accurate, but slower visualization of an image.

The researchers were able to show the chip — dubbed Tianmouc — could process pixel arrays quickly and robustly in an automotive driving perception system.

In a paper published today, the authors said: "We demonstrate the integration of a Tianmouc chip into an autonomous driving system, showcasing its abilities to enable accurate, fast and robust perception, even in challenging corner cases on open roads. The primitive-based complementary sensing paradigm helps in overcoming fundamental limitations in developing vision systems for diverse open-world applications."

In a separate paper, Davide Scaramuzza, University of Zurich robotics and perception professor, and his colleagues adopt a similar hybrid approach but apply it to camera technologies.

Youtube Video

Cameras for self-driving vehicles navigate a trade-off between bandwidth and latency. While high-res color cameras have good resolution, they require high bandwidth to detect rapid changes. Conversely, reducing the bandwidth increases latency, affecting the timely processing of data for potentially life-saving decision making.

To get out of this bind, the Swiss-based researchers developed a hybrid camera combining event processing with high-bandwidth image processing. Events cameras only record intensity changes, and report them as sparse measurements, meaning the system does not suffer from the bandwidth/latency trade-off.

The event camera is used to detect changes in the blind time between image frames using events. Event data converted into a graph, which changes over time and connects nearby points, is computed locally. The resulting hybrid object detector reduces the detection time in dangerous high-speed situations, according to an explanatory video.

Among AI infrastructure hopefuls, Qualcomm has become an unlikely ally

• GhostStripe attack haunts self-driving cars by making them ignore road signs
• Boston Dynamics' humanoid Atlas is dead, long live the ... new commercial Atlas
• US military pulls the trigger, uses AI to target air strikes

In their paper , the authors say: "Our method exploits the high temporal resolution and sparsity of events and the rich but low temporal resolution information in standard images to generate efficient, high-rate object detections, reducing perceptual and computational latency."

They argue their use of a 20 frames per second RGB camera plus an event camera can achieve the same latency as a 5,000-fps camera with the bandwidth of a 45-fps camera without compromising accuracy.

"Our approach paves the way for efficient and robust perception in edge-case scenarios by uncovering the potential of event cameras," the authors write.

With a hybrid approach to both cameras and data processing in the offing, more widespread adoption of self-driving vehicles may be just around the corner. ®

Narrower topics

• Large Language Model
• Machine Learning
• Neural Networks
• Tensor Processing Unit

Broader topics

• Self-driving Car

Send us news

Other stories you might like

Ai smartphones must balance promise against hype and privacy concerns, will windows drive a pc refresh everyone's talking about ai, ai chip sales predicted to jump by a third this year – then cool off, the sky’s the limit for 5g app developers.

Prepare your audits: EU Commission approves first-of-its-kind AI Act

Using ai in science can add to reproducibility woes, say boffins, google finally addresses those bizarre ai search results, top ai players pledge to pull the plug on models that present intolerable risk, mit professor hoses down predictions ai will put a rocket under the economy, by 2030, software developers will be using ai to cut their workload 'in half', big brains divided over training ai with more ai: is model collapse inevitable.

• Advertise with us

Our Websites

• The Next Platform
• Blocks and Files

Your Privacy

• Cookies Policy
• Privacy Policy
• Ts & Cs

Copyright. All rights reserved © 1998–2024