I hate car mirrors

Driving simulation development is fun and all, right until you remember that actual cars have mirrors. So today I’ll rant about that, and maybe explore solutions.

The problems with…

Mirrors in games

Driving simulation is kind of “just a video game”, and game developers will tell you: they don’t like mirrors.

Why? Well, unless you do ray-tracing, which is still quite uncommon, rendering a mirror is basically rendering another image, from another point of view, on top of the actual main image you’re already rendering. So in terms of computation cost, it’s quite significant.

Of course, modern game engines, such as Unreal, have implemented various methods of rendering reflections, which all come with different cost/benefit ratios, so you can choose which one is best for you. But obviously, better quality means higher cost, there’s no way around that.

Car mirrors

So, how are car mirrors different from classic video game mirrors, and what challenges do they bring?

To start: there are 3 of them. As if rendering one reflection wasn’t enough, we have to deal with 3, at all times, covering a pretty wide field of view. That’s just… great.

Then, you have the fact that most of the time, cars are driven outdoors (shocking, I know). Indoor mirrors are easier to deal with: limited (and usually static) amount of space and objects. If you’re far from the mirror, you don’t have to reflect much; if you’re close to it, then you don’t have many things in direct view, so it kind of evens out.

But the outdoor? On roads? You’re dealing with long and wide field of views, many items and light sources, weather effects and what have you.

And last but not least, car mirrors aren’t planar (unless you’re in North America apparently). They’re actually convex, or even aspheric. Their actual shape isn’t standardized, so usually it changes between car models. So if we want to simulate accurate car mirrors, we have to figure out their shape and how to render reflection following those. Tricky.

Driver

Not only are mirrors tricky, but the driver adds a layer on top that.

When driving a real car and looking in a mirror, sometimes you’ll slightly move your head one way or the other to catch a glimpse of something that’s out of the mirror’s field of view. That’s the nice thing about (real) mirrors: depending on where you’re looking at them from, they’ll show a different point of view!

But replicating that effect in some simulator setups is quite difficult, especially when considering some of the other challenges mentioned above, like the fact that mirrors aren’t planar.

Another issue regarding the driver is more related to why we use simulation in the first place: to study driver’s behavior and cognition. To do that, we sometimes rely on eye-tracking and other measurements that can help us understand what the driver perceived from the (simulated) environment.

One recent example is covered on our “Angular size of partially occluded actors” blog entry, which illustrated how we need to compute indicators based on actor visibility. And as mentioned in the article, mirrors are again here to ruin our day: not only do we need to measure visibility in the main “direct” view, but we also have to do that for each mirror.

How we do it

We’ve laid out some of the challenges related to car mirrors in driving simulation. Now that it’s out, let’s focus on how we tackle some of those in our various simulator setups.

No cabin

First we’ll talk about small-scale simulators that don’t have an actual cabin with mirrors. Those include common desktop setups with gaming steering wheel and pedals.

In those, we usually display ego’s virtual car, which include the mirrors’ mesh. To get the reflection displayed in there, we first have to ensure that the mirrors have their own material slot (ideally, one slot per mirror, but you can make it work with a single one) and properly unwrapped UVs. Our go-to ego car for mirrors is lyoshko’s Luxury Car from their 7 Cars Pack, which I highly recommend.

The actual reflection is generated using a Scene Capture 2D at a fixed point of view. This means both no dynamic point of view based on driver’s head position (we don’t use head-tracking), and also no convex or aspheric reflection, only planar. Using scene capture allows us to fine tune the reflection settings. But that can be tricky, especially with our trade-offs: we use a wider capture field of view than the theoretical planar mirror would give to compensate for the lack of both head movement and convex/aspheric mirror.

With some shader magic, you could probably emulate convex/aspheric using the simple Scene Capture 2D, or you could use a Scene Capture Cube, but the performance hit isn’t worth it in my opinion.

With the recent progress on realtime ray tracing, it could also be interesting to explore that solution for car mirrors.

With cabin

If your simulator cockpit is large enough to include physical mirrors, there are a few ways to go about integrating virtual mirrors; and some time ago, arguing about each and every one of those was a (albeit annoying) pastime of mine.

However, let’s start by saying that there’s also a very wrong way of doing it, and I’ve seen it implemented on some of the most expensive driving simulator out there, so I think it deserves a warning. If you have a continuous 360° display, don’t just use real actual mirrors that reflect the display. Mirrors’ images should be computed from a point of view that is not the “direct” view. The only valid application of real physical mirror is if they point toward a display area that is generated from the mirrored driver point of view, and not their actual point of view.

Anyway, the most common solution is to use small monitors that are integrated within the mirror frame. For side mirrors you can use standard ratios (16:9). Even though it’s not very common to have perfectly rectangular side mirrors, it’s not that problematic (and if you think it is, you can rework the frame to cover part of the monitor).

For the rear-view mirror, it’s a bit harder to find ~10 inches ~32:9 monitors; but I know they exist, I just don’t know where. Otherwise, for this and only this mirror, you can actually keep the real mirror and reflect the “direct” view that is displayed in the rear (if you have rear display). The difference in point of view between direct and mirrored is rather small and has very little impact on overall scene perception (unless you’re working on very specific use cases).

If you’re integrating monitors inside mirrors’ frames, you’ll want to flip the image (“mirror” it). I’ve previously said that this was not possible in nDisplay, but it turns out I was wrong: it’s very much possible, it was just poorly documented at the time. I’ve been told that it’s even easier to set up in Unreal Engine 5, so rejoice!

Virtual Reality headsets

I’ve talked about VR+mirrors in a previous entry, but to summarize: it’s hard enough rendering 1 frame with 3 mirrors at 60Hz, so 2 frames with 3 mirrors each at 90Hz is just… no.

My workaround so far has been to use the same “Scene Capture” solution as discussed above, but with a custom layer of foveated rendering over it. Since our VR headsets have eye-trackers, the idea is to render a super low-res low-FPS mirror most of the time, and to only render high-res high-FPS mirror when the driver is actively looking at it.

At the moment this is just a prototype, and we’ve never actually used it (we don’t do much VR-headset driving simulation), but I was rather impressed by the initial results: the switch from low to high quality is almost undetectable, and the low quality mirrors aren’t “so bad looking” that they become salient in the peripheral vision (unless you dip below 1 FPS).

Car mirrors: a thing of the past?

A side note to conclude this article: will car mirrors disappear in the near future?

A few years ago, back when I was looking for any argument supporting my “embedded monitors instead of mirrors” argument, I noticed a trend starting to emerge: replacing mirror with monitors… in real cars!

First it was Lexus, then Audi, maybe more followed. The benefits? Adaptative field of view, improved visibility at night and in difficult weather condition, in-monitor contextual display… and reduced drag, though I doubt you save much fuel (or battery).

Though this trend didn’t really seem to catch on, it can also be found in trucks: Mercedes-Benz offers this option for its Actros series.

So once again, Betteridge’s law of headlines prevails: mirrors will keep on saving lives on the road, and keep on causing me headaches.

Written on March 1, 2023