What did we do those past 3 months? I didn’t quite remember myself before deciding to write it up, so let’s all dive in and see what’s new.

Vulnerable Road Users

In a previous entry I mentioned our work around Vulnerable Road Users (VRUs), which are a critical piece of our research. Over time we’ve worked a lot on pedestrians, but the past months we focused on “rides”, mostly bicycles and E-scooters.

Rides are somewhat tricky to implement properly, as they’re the result of the interaction of two bodies: the ride (vehicle) and the rider (character). To get a realistic and customizable behavior, both are implemented as Skeletal Meshes, with the ride being driven by Unreal’s vehicle framework (PhysX or Chaos), and the rider being animated by a home-made Animation Blueprint, which actually works across all rides’ types (e.g., bikes, scooter).

But when you have a wide range of rides and riders (e.g., child, adult), it starts to become quite complex to ensure the compatibility between all of those: a child and an adult aren’t riding a scooter the same way. A lot of things can change: handle height, hands/feet position, elbow orientation, and so on.

And that’s where Unreal “Data Asset” come into the picture. We can easily create assets that will hold all the configuration values for a character, so that it perfectly fits our rides, and so we can easily reuse it. Since we have a rather large collection of characters, coming from various sources (e.g., Marketplace, CARLA), we’ve been able to build a library of Data Asset that can fit all of them into our rides. And Data Assets can even be edited at runtime, meaning it’s super easy to fine tune every setting.

And as the video above shows: once you’ve defined your “rider” Data Asset, there’s nothing stopping you from adding more than one rider on a single scooter! You just need to create Data Assets for both the “rider” and the “passenger”, each having their own position offsets so that they both fit on the vehicle. This will be especially useful for an upcoming experiment, which aims at recreating complex and emerging car-VRU interactions.

Data Assets aren’t particularly well documented, but they sure are a powerful Unreal feature. Since we discovered them for the use case mentioned above, we’ve already started using them in other parts of our tool.

Unreal Engine 5

When UE5.1’s roadmap got published, it motivated me enough to migrate our simulation platform from UE4.27 to UE5.

It wasn’t a full migration from scratch: I had already prototyped things in the Early Access version, and most of our Marketplace products had already been updated to UE5. The main issue to solve was two-wheeled vehicles, as those work quite differently from PhysX to Chaos; and nearly all of the ones we use are custom made, so I couldn’t rely on someone from the Marketplace just pushing an update!

In the end, vehicles were migrated rather easily, except for a couple minor issues… and a big one. I spent days narrowing down an issue where off-screen vehicles would just randomly go completely off-track. You can see from the video below: the red car passing by somehow does a 180° flip when off-screen and comes back the wrong way!

I got help from Marco Ghislanzoni to figure this one out, so many thanks to him. The issue was related to wheel colliders not being updated when the vehicle was not visible, which lead to lots more issues down the line. I reproduced the issue on the vehicle demo map, sent a bug report to Epic, and it now seems that this is solved in UE5.1! I’m not sure if my bug report helped, but in any case it’s great to have that fixed.

In the end, I managed to run our most complex UE4 scenario in UE5, which was my main goal. I still have some performance issues to figure out, as well as testing on our nDisplay clusters, but I’ll wait for 5.1 to test that.

Since we have a very unusual “forked template project” workflow, with two active projects in development, the master branch on our repository is still using UE4. So merging Blueprint development done in master with changes in the UE5 branch isn’t exactly easy nor fun, but at least we’re moving closer to actually using Unreal Engine 5 in our driving simulation research.

And to conclude that UE5 section, I’ll mention that Switchboard scripting is coming to UE5.1, which is a feature no-one is talking about (not even Epic), but which I’m very excited about: we’ll be able to fully automate Switchboard, so no more needing to use the User Interface to launch the experiment, all that will be handled by background scripts. Experimenters will be able to focus on their work instead of having to learn the Switchboard UI!

NEWMOB

We’ve mentioned this project before, but this is still where most of our development time is spent.

The experiment’s “training scenario” is now mostly done, and even though it’s only for training, it’s probably one of our most ambitious scenario yet. Around 15 minutes of driving, through various environment styles (e.g., rural, urban, industrial), with lots of interaction with VRUs.

The past few weeks were spent on optimization and polishing. Even though this experiment will use our largest driving simulator, we did a lot of testing on our mid-range one which is the only one available at the moment. Our unified workflow across all our cabins allows us to run any project on any of our platform, which is hugely beneficial during testing.

We started using Level Streaming to improve performance and stability (we were running out of GPU memory!). We’re still figuring things out, but overall it’s very positive. The picture above shows some recent rework: sidewalks. For quite some time now we’ve felt that our junctions were weird, but we couldn’t quite put our finger on why. It turned out that our sidewalks didn’t have a realistic geometry, which is not something obvious if you’re not working with sidewalks in real life. Thankfully, RoadRunner is so awesome that tweaking sidewalks takes a few seconds.

Next up in this project is the “main” scenario. From a design perspective, it will be quite similar to the one we’ve done for the training, except we’re now all much more experienced in this workflow, so everything should be done much quicker. But we like to challenge ourselves, so we’ll still try to add new and innovative things in it.

Varjo

I moved to a new, much bigger, office; and one of the reason is to have more space to test and prototype with our XR tools. The headliner of which is the Varjo XR-3.

A colleague helped me make this sweet "#Varjo-mobile". Now I can easily move our whole #XR setup around campus for experiments, demos or simply prototyping in my office. pic.twitter.com/lcRzMrYzug

— Bertrand Richard (@brifsttar) September 22, 2022

Our new Varjo-mobile will allow us to use our XR setup along with other scientific equipment. For instance, we plan on trying mixed reality on our experimental seat used to study (dis)comfort. And I’m still planing on using mixed reality in our main driving simulator by throwing a green sheet over it and chroma-keying all windows to display a virtual worlds over them. So stay tuned!

My new office has ample room for #VR stuff, so it's time to experiment new things!#UnrealEngine + #Varjo + #eScooter + #Simulation + #MixedReality = ? pic.twitter.com/XhZtrT8sKp

— Bertrand Richard (@brifsttar) August 2, 2022

Another Varjo-related development is scooter simulator. We’re not quite sure we’ll go forward with this, but it’s really great to be able to prototype things with the XR-3: with depth occlusion and a scooter borrowed from a colleague, we’ve been able to simulate riding through a virtual world while seeing our very real body and vehicle. It looks very promising and opens up a whole lot of opportunities for all our researchers.

Misc

Sometimes you get into the simulator, turn the key to start the vehicle, press the gas pedal and nothing happens. And then you’re left wondering what possible thing could have gone wrong so that nothing is working anymore. To help debug this, and a whole lot of other things, I added a simple “Input Values Debug” display on our UI.

Built using UMG and Katan Charts for the plots, it was actually fun (and easy) to implement, and I’m usually dreading any UI work. But UMG is really intuitive, and with Blueprint it’s fairly clean and simple to get the desired values from the controller to the display. We have tools outside Unreal to visualize inputs, but it’s great to also have them in-engine to quickly get a glimpse of everything, as well as to help figure out where issues lie when debugging.

On a completely unrelated subject, we moved all of our repositories from GitLab SaaS to a new self-managed instance at our university. This is mainly due to GitLab’s new free tier restrictions, even though we were using the GitLab for Education offer. The migration was no easy feat, as our main super-repository contains over 300 submodules which totals around a whopping 500GB. But it all went rather smoothly, and we’ve been able to continue our daily operations without any downtime. I’m still waiting on Epic’s Skein to improve our version control workflow, which is something I’m really not fully satisfied with.

Marketplace

My three favorite Marketplace creators have each been very productive, and their latest additions are absolute must-have for us. They’re all high quality and very low price.

lyoshko added an absolutely amazing articulated city bus, which comes with pretty much every possible option I can think of.

Der Sky’s new French Highway pack is a dream come true for our driving simulation highway scenes. And the creator does signage for a lot of countries, so they probably have products tailored for you!

Meik.W continues to add more and more buildings that are perfect for our roadsides: Gas Station, Coffee Shop, Car Wash Box and much more.

I can’t praise those creators enough for their work, and I highly recommend you check them out.