Training AI always depends on the quality of input data: the better your data, the better the AI.
So, when it comes to autonomous driving, it’s always been clear that simulation-based training would only be successful if the model could realistically mirror the real-world environment.
In recent years, digital twins have transformed the research and development process, helping engineers to test and train prototypes at scale without breaking the bank. Now, digital twins are being developed of cities to make it easier for automotive companies to train AI for autonomous vehicles (AVs). With a digital twin of a cityscape, companies can iteratively evaluate the effect of real-world variation on their AI tools, driving down the cost of prototype testing and development. But training AI for AVs is only scratching the surface of the potential application of digital twin cityscapes.
Why Use Simulations and Digital Twins for AI Training?
Simply having 2D maps of roads for a given area is not representative of all the random, unpredictable events that can occur in the environment. Although humans use predominately audio and visual cues while driving, AVs integrate a full suite of sensors that can detect many more elements of the physical environment. This includes everything from the length of vegetation, the material of surrounding buildings, active precipitation and how all these variables impact road conditions and safety. But for sensors to function properly and for AVs to operate safely, AI needs to be trained to handle the chaotic and unpredictable real-world driving environment.
Digital twins also facilitate AI training to troubleshoot potential issues that arise due to the properties of AVs. For example, glass can reflect LiDAR signals and disrupt vehicle sensing. By training for these scenarios, companies can improve their AI development before vehicles ever hit the road.
Making a digital twin of a cityscape can be time-consuming and challenging, but the end result can accelerate the overall development process and reduce the time and cost necessary for AI training.
Within the simulation map, engineers can layer in dynamic elements they want to use to test and train their AI. For example, weather conditions can be simulated, including: rain, snow, fog, and daylight; traffic density for both vehicles and pedestrians can be modified, and other dynamic elements such as cyclists, children, pets and wildlife can also be added or subtracted from the city model.
As an example, let’s consider fleet vehicles operating in one metropolitan area, such as San Francisco. Using a digital twin to train the AI can ensure the vehicles are prepared for any possible scenario within the city of San Francisco. Prior to any physical testing, AI can be trained to the exact dimensions of the city and the type of events that might disrupt vehicle function: fog, rain, steep hills, etc. Investing more in the preliminary training phase can help improve success when it comes to physical prototyping.
How To: Build a Digital Twin of a City
In conversation with Matthias Schlegel, Product Manager Autonomous Simulation at Hexagon, he outlined the two steps required to create a digital twin of a cityscape: (1) reality capture and (2) feature capture.
Step 1: Reality Capture
As part of the reality capture, three sub-steps need to be considered: measurement, classification and data fusion. The first sub-step during the reality capture is the most important: collecting ground measurements. A vehicle with an onboard mobile measurement system can gather detailed information required for making a digital twin. To develop an accurate, detailed cityscape, it’s important to collect high-quality original measurement data. Everything that follows the collection process, from extraction to layering in dynamic elements, will ultimately deteriorate the original information. The more detailed the initial measurements can be, the better. Hexagon currently collects aerial measurements to a 5-centimeter resolution. For ultra-high-resolution mapping of cities, aerial data is captured using the LiDAR airborne sensor, Leica CityMapper-2.
Next, the collected data is classified and vectorized. Dynamic objects such as pedestrians, cyclists and other vehicles are removed from the initial map. The final 3D model will therefore only include static objects such as road signs, buildings, benches and more. Other dynamic elements can be layered in later as part of diverse simulation scenarios. This often-tedious process involves a combination of manual and automatic detection of objects to be removed from the ground-level and aerial scans.
The final stage of reality capture is fusing the data to increase confidence in the final digital twin. As part of this process, engineers need to combine ground-level measurements with aerial imagery such as photos, digital surface models, GPS data and digital terrain models. For Hexagon, all the data used in their digital twins is collected and integrated in-house, as they do not use any public data for these purposes. They use the software tool Vadstena, from Melown Technologies, to construct the final 3D cityscape, incorporating all the images taken during the measurement stage.
Step 2: Feature Capture
This next step in the production of a cityscape digital twin is used to mature the model and facilitate real-world scenario testing.
First, road logic is extracted in an open drive format; this includes lane markings, streetlights, traffic signs or any other elements relevant to road safety. This stage aims to extract as much information as possible to enable the simulation to respect traffic rules. Next, the buildings and physical environment are extracted, and textures and materials are layered on top of this information. Is a certain building covered in glass? Is a road cobble-stoned, paved or concrete? This information will facilitate training of the AI and how the physical environment and its materials may disrupt input signals.
The final step is termed ‘adaptation.’ Here, engineers aim to improve the overall visualization and tailor the map to match specific use cases. For example, an engineer might modify road lines, traffic islands or textures to increase the complexity of the simulation and test certain scenarios. Engineers can ask questions like ‘what would happen if the vegetation lining a road was overgrown and beginning to occlude traffic?’ Similar scenarios can be simulated in the digital twin, and engineers can make additional modifications and iteratively test and train the AI as needed. Over time, this process can also become increasingly automated as engineers develop strategies to train AI within the digital cityscape.
Where can Engineers Access Maps to Generate Digital Twins?
Unfortunately, there is no public catalog of integrated aerial and 3D maps at the resolution necessary to generate a digital twin. In conversation, Schlegel mentioned that Hexagon might offer maps-as-a-service in the future, but this service is currently unavailable. For Hexagon clients, however, engineers can utilize aerial maps of several metropolitan areas, including Munich, NYC, Barcelona, and more. However, the available maps are not in the complete simulation stage, and ground-level measurements are still required for each city. The aerial maps are also only available at the end of the reality capture stage and need a complete feature extraction before they can be used as a true digital twin.
Hexagon wants to operate as a solution provider to generate these digital twins. Currently, the HxGN Content Program includes one of the largest (private) databases of high-resolution 2D aerial and 3D ground-level information for digital twins.
Looking Beyond Autonomous Vehicles
Digital cityscapes are useful for training much more than AVs. These digital twins can also train driver assistance functions such as adaptative cruise control and parking assistance.
Beyond the automotive industry, cities have also expressed interest in digital cityscapes for asset management, simulating maintenance of public vegetation, road planning and other city planning initiatives. This is also true for construction planning, traffic modeling and weather modeling to assess how cities can appropriately plan for specific scenarios. Construction companies can even use digital twins to provide more accurate cost estimates for large- and small-scale projects.
As AVs, driver-assist features and other technologies continue to expand in the consumer market, the development of digital cityscapes will help shift the validation of these tools earlier in the development process.
Over the next few years, digital twins of cityscapes will continue to be developed to improve autonomous AI training. Currently, the digital construction process is time-consuming, but Hexagon is working on making the overall process as automated as possible. As they work towards streamlining digital twin development, it will hopefully make these tools more accessible, as other companies work on generating their own digital twins for AI training.