Enhancing Test Driver Response Skills through Accident Scenario Analysis in Real-World Evaluation of Autonomous Vehicles
Copyright Ⓒ 2024 KSAE / 219-10
This is an Open-Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License(http://creativecommons.org/licenses/by-nc/3.0) which permits unrestricted non-commercial use, distribution, and reproduction in any medium provided the original work is properly cited.
Abstract
This paper aims to propose a method that can help improve test drivers’ ability to avoid accidents and response time by analyzing accident recordings from dashcams. This is done to prevent safety accidents that may occur during the actual road evaluations of autonomous vehicles. By analyzing accident videos that occurred during a driving situation, safety accidents can be prevented by identifying the traffic situation and the scenario in advance, and accident symptom training can be provided to autonomous driving test operators. Thus, test operators can understand traffic situations and scenarios with a high probability of accidents, and can improve safety measures during the testing of an autonomous vehicle during an open road evaluation. The proposed method is expected to contribute to strengthening the safety standards of self-driving vehicle evaluations on actual roads.
Keywords:
Open road testing, Accident recording analysis, Traffic scene recognition, Ontology, Accident prevention1. Introduction
In the modern world, self-driving vehicles are leading a revolutionary paradigm shift in driving. These vehicles are equipped with sensors and algorithms to detect road conditions and make decisions, allowing them to drive themselves without driver intervention. Accordingly, the evaluation of self-driving vehicles is becoming increasingly important.
On-road evaluation of self-driving vehicles is an essential step in evaluating their performance and safety. The performance of a self-driving vehicle is determined by how well it can cope with driving situations on real roads, which is crucial for safety and reliability. However, testing and evaluation of self-driving systems carry risks that may cause accidents on the road.
These risks arise from the limitations of self-driving systems and the potential for accidents in unpredictable situations. Therefore, test drivers must be able to recognize the signs of an accident situation in a self-driving vehicle and react appropriately.
Test drivers’ role is crucial in evaluating and improving the performance of self-driving vehicles. Specifically, they play a key role in validating self-driving systems in real-world road conditions. They need to utilize their real-world driving experience and abilities to ensure that self-driving vehicles are safe and reliable in various situations. Hence, they need to have the following competencies:
Real-world driving experience: Test drivers have real-world driving experience in different driving situations and road conditions. Their ability to cope with and anticipate complex situations in the road environment helps evaluate the performance of self-driving vehicles.
Ability to prevent accidents: Test drivers should have excellent accident prevention skills. They should quickly recognize dangerous situations and take appropriate actions to evaluate the response and performance of the self-driving system.
Understanding driver intent: Test drivers should understand and mimic the driver’s intentions. This helps to accurately model driver behavior, such as the vehicle’s driving patterns, lane changes, and signal recognition, to evaluate the self-driving system’s response.
Efficient problem-solving skills: Test drivers should react quickly if a problem arises. This helps assess safety and resolve issues immediately by taking effective action in situations such as unexpected road conditions or system errors.
Feedback and reporting capabilities: Test drivers should provide accurate and detailed feedback on the behavior and performance of the self-driving system. This is important information for improving and enhancing the system.
To improve the test driver’s ability to prevent accidents and quickly resolve problems when they occur, this paper aims to organize and classify the traffic conditions and scenes at the time of the accident through accident video. In addition, through the development of an accident scene catalog, we intend to classify various accidents that may occur in real-world road tests of self-driving vehicles.
Guidelines related to on-road evaluation6-11) suggest the need for safety accident prevention training for test drivers; however, no specific training materials exist. Therefore, this study can be distinguished from other studies as it attempts to develop materials for accident prevention and response to address real road accidents. The originality and usefulness of this study can be seen in the analysis of actual accident situations and the development of materials to improve accident response.
This thesis is organized as follows: Chapter 2 analyzes studies that utilize accident videos. Chapter 3 examines selected accident videos, summarizes the traffic situations that occurred in the real-world evaluation, and analyzes the causes of each accident. Finally, Chapter 4 provides conclusions and future research.
2. Related Studies
This section describes related studies that use traffic accident data and categorize traffic scenes.
2.1 Research on Traffic Accident Data Analysis
Kang1) studied an AI model that classifies dangerous situations from dashcam videos of traffic accidents. The study’s main purpose was to improve the ability of self-driving systems to avoid accidents. Since dashcam footage was uncalibrated, distortions may occur, and recording performance of each product may vary, there are certain limitations in creating a sophisticated and accurate model based on the study’s findings.
Lee2) analyzed traffic accident data on how to generate critical scenarios to verify the collision prevention system of self-driving vehicles. The significance of this study is that it calculated the scenario space based on the road shape and collision traffic flow for possible accident situations based on accident data and conducted tests based on it. However, the study has limitations in that the actual traffic data was not reflected in the simulation, and the number of traffic participants considered in the simulation was limited. Another limitation is that the simulation environment considered is essentially an open-loop simulation rather than a closed-loop simulation.
To analyze traffic accidents, Kim’s3) research aimed to accurately model the accident scene using a drone and apply PC-Crash, an accident analysis program, to reproduce accidents based on this.
2.2 Research on Traffic Scene Classification
Bagschik’s study4) proposed a method for organizing all information in a traffic scene based on a knowledge base and structuring it based on an ontology using a hierarchical model. The paper aimed to enable the automatic generation of traffic scenes, which is proposed to solve the difficulty of scenario generation. This research is presented at the proof-of-concept stage, and further research is needed to determine the utility of the research results. However, an approach to quantify traffic scenes is currently developed by ISO TC22/SC33/WG9 as a standard for categorizing scenarios.5)
3. Traffic Accident Data Analysis and Traffic Scene Classification
This paper analyzed traffic accident videos collected from the Korean Agency for Technology and Standards’ project to build a data platform for storing automobile accident videos in a standard format (2020~2023) and classified them by accident cause. There can be many causes of accidents, not just one, but the most important ones were categorized. As specified in Table 1, 10 criteria were classified, 200 videos were analyzed, and 98 traffic scenes were selected for test driver training.
Classification of dominant accident causes
3.1 Traffic Accident Scenes
This section describes nine typical accident scenes that can occur in real-world road conditions. For each case, the accident location is summarized, as presented in Table 2.
Representative hazardous scenarios analyzed from the 200 accident recordings
The obstruction of vision by neighboring vehicles in a crosswalk is a crucial factor that may increase the risk of a collision between a pedestrian and a vehicle. In the given traffic accident scene, a bicyclist is traveling through a crosswalk. Since there are no other vehicles in the right lane, some vehicles may attempt to change lanes and turn right through the crosswalk. However, the preceding vehicle may obscure the cyclist, causing the vehicle to fail to detect the person and collide with him.
In these situations, the test driver should be particularly mindful of visibility obstruction by surrounding vehicles. First, to minimize the risk of collisions with pedestrians or cyclists moving in crosswalks, the test driver should carefully monitor that the self-driving car has enough free time before performing lane changes and right turns. The test driver should check to see if there are any pedestrians or cyclists obscured by the preceding vehicle and should be prepared to take over control immediately if their safety is not considered a priority.
Second, it must be possible to monitor whether the self-driving vehicle’s sensors and systems are accurately detecting pedestrians or bicyclists obscured by visibility obstructions. The self-driving vehicle must be able to use technology that can comprehensively perceive its surroundings to determine whether a visibility obstruction is likely to cause a detection failure.
Cyclist occluded by other vehicles on the crosswalk
A collision with an oncoming vehicle in an unprotected left turn is one of the most dangerous situations. In the presented traffic situation, there are only a few vehicles on the road traveling at a relatively high speed. The vehicle in the opposite lane is waiting to make an unprotected left turn and will subsequently attempt to make a left turn. The traffic light is marked straight ahead, so the car attempts to drive straight ahead, but it is too late to see the car turning left on the other side and attempt to slow down, resulting in an inevitable collision.
Intersection with an unprotected left turn
In this situation, the test driver should be particularly mindful of the risk of crashing in an unprotected left turn situation. First, the test driver should observe the traffic signal indications and ensure that the driver is following them properly. When attempting to make a left turn, be extra careful if there is no protected signal or priority. If necessary, a separate safety management plan may be required if the driving route includes unprotected left turns. For example, you may consider utilizing additional personnel to determine if an oncoming vehicle is making a dangerous entry and communicate this to the test driver.
Second, the test driver must be able to accurately identify its surroundings and anticipate the behavior of the other vehicle. The test driver must also be able to quickly recognize whether the other vehicle is attempting to turn left or continue driving straight and react accordingly. This requires the self-driving vehicle to have a wide field of vision and the ability to utilize sensor technology to detect and predict the movements of other vehicles. The test driver can determine whether the self-driving vehicle has these capabilities by looking at the separate status information provided by the test vehicle.
Making a right turn in merging lanes, or any situation where a vehicle without the right of way must yield to another vehicle in a situation where it has the right of way, is an important road situation for self-driving vehicles. Because the right of way is a passively given concept, self-driving vehicles need to be able to understand it and respond appropriately.
The test driver needs to be very careful to avoid collisions in these situations. First, he needs to monitor whether the self-driving vehicle is yielding appropriately to the vehicle with the right of way when it does not have the right of way. This can be done by keeping an eye on the behavior of the vehicle with the right of way and making sure it takes appropriate right turns or lane merging actions accordingly. The test driver must constantly observe and anticipate the movements of surrounding vehicles to accurately recognize the right of way and respond to the situation.
Second, the test driver should observe that the self-driving vehicle attempts to turn right or merge into a lane only when there are no passing vehicles. He should observe whether the self-driving vehicle maintains a sufficiently safe distance to prevent collisions with other vehicles and whether it continuously detects changes in the surrounding environment and moves at the appropriate time.
Priority and privilege to move
In congested traffic, the test driver must be very careful to avoid a collision. One lane is free of vehicles, but the other lanes are congested with vehicles. A vehicle attempting to make a U-turn in the opposite lane may encroach vertically into the ego lane, where the cut-in vehicle may be obscured by other vehicles and exposed at close range.
Collision with a vehicle making a U-turn in a congested section
The test driver must take precautions to avoid crashing in such situations. First, he should monitor whether the distance to the neighboring vehicle is properly maintained. He should also constantly monitor the movement of the surrounding vehicles to anticipate and prepare for the aforementioned encroachment.
This traffic scene requires a difficult decision. The preceding vehicle, a truck, enters the left-turn lane to make a left turn, but the ego lane is occupied. The car next to it has also stopped to enter the left-turn lane to make a left turn. This is a situation that could lead to a collision if the self-driving vehicle in front of it does not understand the intentions of the other vehicles. Therefore, the test driver should be prepared to take over control at any time when a situation like this arises.
Partial occupation of the adjacent lane and sudden slowing down
School zones are areas where traffic conditions can be particularly hazardous and accidents can cause very serious problems. Children may be obscured by parked cars in the surrounding area, trucks may be moving forward, and cars in the opposite lane may be waiting for them to pass. Children may be running across the crosswalk because the pedestrian light has turned blue, regardless of whether the surrounding traffic is moving.
Children running in a crosswalk
These hazardous situations are ones that the test driver should be especially careful about. First, he needs to be aware of and alert for children who are obscured by parked cars. He needs to keep a close eye on his surroundings to make sure he has a clear view and keep a sharp lookout to make sure that the obscured children do not appear on the road.
Second, the test driver should be aware that children can run into the road at any time in school zones and should always be prepared to bring the vehicle to an emergency stop. To increase this reactivity, the self-driving car’s capabilities should be reviewed in advance of the test, as the test driver may not be able to react if the self-driving car is traveling too fast in a school zone.
School zones are road environments where the safety of children should be a top priority. The test driver should be aware of these dangerous situations, drive carefully, and take appropriate actions to ensure the safety of children.
Traffic on high-speed roads can be a pressing challenge for self-driving vehicles. In particular, a lack of accurate obstacle recognition can increase the risk of a collision. Therefore, the test driver should be able to recognize and respond to obstacles in highway traffic.
Various obstacles may occur on a highway. Tables 9 to 11 describe the footage collected from actual traffic scenes, and it is unlikely that a self-driving vehicle will be able to recognize these situations. Therefore, the test driver should be prepared to take over the self-driving vehicle if such a situation is encountered. The test driver must also identify and anticipate these highway obstacles. He should be able to constantly monitor lane change signals, the movement of nearby vehicles, and changes in road conditions to proactively recognize potential obstacles and take appropriate and immediate action.
A falling object from a truck
Overturned truck
Shattering of the windshield by a thrown stone
During self-driving road tests, the test driver may encounter road users who violate traffic laws, which can increase the risk of an accident. Road users who violate traffic laws may exhibit unpredictable behavior and violations, such as running red lights, speeding, or crossing lanes.
Road users who violate a traffic law
These traffic violations pose a major challenge in testing the performance and reactions of self-driving systems. Thus, the test driver is expected to anticipate and react appropriately to such traffic violations. The test driver must be able to detect violations of road traffic laws, react quickly to prevent accidents, and evaluate the performance and safety of the self-driving system.
4. Future Plans and Conclusions
To reduce the occurrence of safety accidents for the evaluation of self-driving vehicles, this study analyzed traffic accident videos and the signs of accident occurrence and traffic scenes. We intend to utilize the results in developing training materials to improve the accident response-ability of test drivers in real-world situations.
Test drivers play a critical part in evaluating the performance of self-driving vehicles through real-world driving experience in multifarious driving situations and road conditions. They evaluate and help improve the performance of self-driving vehicles based on their ability to cope with and predict road conditions. Through the analysis of accident videos, this study aimed to guide test drivers develop the ability to prevent accidents, determine driver’s intentions, and solve problems quickly.
Therefore, the role and capabilities of test drivers should be strengthened and developed for the evaluation and improvement of self-driving vehicles. This will improve the performance and safety of self-driving systems and reduce the risk of safety incidents that may occur in real-world evaluations.
This study also analyzed accident videos, developed an accident scene catalog to identify traffic conditions and scenes in advance, and systematically classified various accident situations that may occur during real-world road tests to improve the accident response capabilities of test drivers.
A limitation of this study is that due to the limited data collected, it was not possible to analyze all possible situations; therefore, the paper does not cover a wide range of situations with a high risk of accident occurrence. Consequently, we believe that this study needs to be expanded by increasing the number of analysis data aligned with the risk assessment proposed in ISO 26262. In other words, it is necessary to compensate for the shortcomings of this study by evaluating the severity, exposure, and controllability of the accident scenarios in question.
In the future, we plan to analyze additional data to develop a catalog of dangerous traffic scenes for test drivers. To date, we have analyzed 200 pieces of data, but we are aiming to analyze and organize more than 20,000 pieces of data in the future.
The study’s findings can be used to create more complex traffic scenarios that reflect real-world traffic conditions. We believe that we can improve on the work done by Lee2) by reflecting on the surrounding traffic environment rather than simply evaluating the movement of one neighboring vehicle and the ego vehicle.
A lesson learned from analyzing traffic accidents is that accidents are always likely to happen as long as there are humans involved in traffic. This means that sometimes, humans violate traffic laws, but even when they don’t, they can still cause accidents due to negligence in looking ahead and aggressive driving that results from the physical and psychological effects of distraction, fatigue, impatience, irritability, and selfishness.
Therefore, when self-driving vehicles are operated on actual roads in the future, issues such as negligence, which may occur during an accident, and the imposition of criminal penalties for self-driving vehicles may arise. Thus, further research on the topic is recommended.
Acknowledgments
This research was supported by the Korea Institute of Police Technology with funding from the government (National Police Agency) in 2023 (No.RS-2023-00238253, Development of real-world road-based Lv.4 self-driving vehicle driving ability evaluation technology).
References
-
M. Kang, W. Lee, K. Hwang and Y. Yoon, “Vision Transformer for Detecting Critical Situations and Extracting Functional Scenario for Automated Vehicle Safety Assessment,” Sustainability, Vol.14, No.15: 9680.
[https://doi.org/10.3390/su14159680]
-
J. Lee, U. Jung and B. Song, “Critical Scenario Generation for Collision Avoidance of Automated Vehicles Based on Traffic Accident Analysis and Machine Learning,” Transactions of KSAE, Vol.28, No.11, pp.817-826, 2020.
[https://doi.org/10.7467/KSAE.2020.28.11.817]
-
J. Kim, S. Kim, J. Choi and J. Park, “3D Reconstruction of Traffic Accident Scene Using Drone & Pix4D,” Transactions of KSAE, Vol.27, No.12, pp.941-947, 2019.
[https://doi.org/10.7467/KSAE.2019.27.12.941]
-
G. Bagschik, T. Menzel and M. Maurer, Ontology based Scene Creation for the Development of Automated Vehicles,” 2018 IEEE Intelligent Vehicles Symposium (IV). IEEE Press, pp.1813-1820, 2018.
[https://doi.org/10.1109/IVS.2018.8500632]
- ISO, ISO/DIS 34504:2022, “Road Vehicles – Test Scenarios for Automated Driving Systems – Scenario Categorization”, https://www.iso.org/standard/78953.html
- NHTSA, “Automated Driving Systems (ADS): A Vision for Safety 2.0,” Department of Transportation, https://www.nhtsa.gov/node/36071, , Accessed on 23 May 2023.
- NHTSA, “Automated Vehicle 3.0: Preparing for the Future of Transportation,” Department of Transportation, https://www.transportation.gov/av/3/preparing-future-transportation-automated-vehicles-3, , Accessed on 23 May 2023.
- NHTSA, “Automated Vehicle 4.0: Ensuring American Leadership in Automated Vehicle Technologies,” Department of Transportation, 2020, https://www.transportation.gov/sites/dot.gov/files/2020-02/EnsuringAmericanLeadershipAVTech4.pdf, , Accessed on 23 May 2023.
- National Police Agency (Japan), Guidelines for Public Road Testing of Automated Driving Systems, https://www.npa.go.jp/english/bureau/traffic/guideline.pdf, , Accessed on 22 May 2023.
- National Transport Commision (Australia), Lessons learned from automated vehicle trials in Australia December 2020, https://www.ntc.gov.au/sites/default/files/assets/files/AV-trial-guidelines-2020.pdf, , Accessed on 22 May 2023.
- National Transport Commision (Australia), Guidelines for trials of automated vehicles in Australia 2020, https://www.ntc.gov.au/sites/default/files/assets/files/AV-lessons-learned-2020_0.pdf, , Accessed on May 2023.
