Previsouly I talked about our workflows or automation, but I never took the time to explain our overall production pipeline. How do we manage the production of multiple experiments per year, each running on different simulation platforms? How do we capitalize on each project’s new features? How do we ensure everyone is working with up-to-date tools? Let’s explore under the hood and see how we get all of that done.

Creating the project

Requirements

The first thing needed to get started is an Unreal project. But since we own hundreds of Marketplace products, we don’t really want to start from a blank Unreal project.

Also we’ll need a RoadRunner project. And as with Unreal, we also have quite a lot of custom assets (e.g., markings, road types) that we’d need to import to any new blank project.

And depending on the experiment, we might also need other projects in other tools, or have some dependencies to internal tools or frameworks. That could include for example custom LabStreamingLayer apps, tools to manage our clusters or a framework to interact with in-car displays.

So overall, any experiment isn’t just an Unreal project. It’s a lot of projects, with lots of dependencies.

Superproject

Our answer to that is to use a superproject. And to not start from scratch, we have a template superproject stored as a Git repository.

This project, at its core, is an Unreal project. On top of the usual Unreal project structure, we include the following

• All our Marketplace products, which are kept up-to-date (as submodules)
• Custom assets and code, that we built over the years for our specific use cases
• Preset configuration files (Config/*.ini) with settings that work for us
  • Those may vary across branches, depending on the target hardware (e.g., nDisplay, XR headsets)
• Dependencies (e.g., template RoadRunner project) are included, also as subs, in a modules/ directory
• A collection of PowerShell scripts, more on that later

This superproject is constantly updated to include new features, assets, bugfixes, and so on. It contains a few homemade sample levels, and is ready to be used for any project. As long as you…

Fork it!

For us, the actual first step to start a new project is to fork this template superproject. This workflow is really beneficial:

• Always start with a full and ready-to-use to project, no need to bother with boilerplate
• You can work on multiple projects at the same time without having to clone them all individually
  • Any other project is just a git checkout away with the properly configured remote
  • Since our template projet is ~500GB, having multiple instances of that would quickly add up in local space and bandwidth
• Synchronize with the upstream superproject, both ways
  • You can merge new features/fixes coming from upstream
  • The upstream repository can cherry-pick specific changes from the forks

This is really useful since we’re working on multiple projects per-year, each having a lot of similarities (“they’re all racing games”) while still requiring very specific features. And since we’re working with very experimental stuff, having the ability to propagate any new development from one project to another is really beneficial.

Beware that this workflow requires great attention to not end up in merge hell. If all forks start editing the same Blueprint, and then those changes get cherry-picked into the upstream repo, things are going to break pretty fast. But with a clean design, good communication and clear processes, things tend to work out pretty good here. Blueprint merge conflicts are very rare, and we don’t even use file-locking!

Do Unreal things!

Once the fork is setup, you can jump straight into RoadRunner and Unreal and work on your actual project. All assets are there, everything’s configured.

A big part of my job therefore is to maintain this template superproject

• Ensure it’s always up to date
  • It’s been migrated from Unreal 4.25 to 5.4 thus far!
• Include new Marketplace/third-party products and updates
• Cherry-pick useful developments coming from the forks
• Ensure compatibility with our hardware (e.g., simulators, XR headsets)
• Develop new features to make everything better and easier for the forks!

“Building” the project

Let’s jump over the fun part, which is using Unreal to create our virtual worlds, and go to the boring part, which is generating a usable end-product from that. By this, I mean either an executable for interactive experiment, or videos for non-interactive.

Thankfully, Unreal is pretty of good at both packaging and offline rendering, so as usual, most of the heavy lifting is handled for us. But we still need to complete the last mile ourselves, to tune all of those pipelines to our needs.

We wrote a bunch PowerShell scripts that handle all of that, from bundling other dependencies (e.g., sensors acquisition tools) alongside Unreal packages, to efficiently robocopying all of that to our deployment servers.

And I say servers plural, because we have a specific one for each of our simulator (hardware), because each one of those has its own local network. So usually, each fork will also have to create its own main entry-point PowerShell script to at least specify which server the product should be deployed to (along with other possible specific requirements). But that’s usually a one-liner, and nothing other than this is preventing the project from running an another simulator.

We also have similar scripts for other “side-build” stuff, like static lighting (but now Lumen exists…). Maybe in the future we’ll also add HLOD builds for example.

””“Continuous integration”””

Having scripts to build a bunch of boring stuff is really great, but running them locally on your computer is really not, as those can run for hours. That’s why continuous integration exists, with tools like Jenkins, TeamCity, QuickBuild or Epic’s very own Horde.

However, we didn’t chose any of them. Why?

• I’m very lazy and didn’t want to set those up
• None of them is really suited for Unreal projects (Horde didn’t exist at that time)
• I’d prefer something simple with minimal setup, ideally that could wholly fit into our version control
• We already had Discord bots and webhooks…

So instead we built a rather small Discord bot, running on a build machine in my office. It just runs the PowerShell scripts detailed above, either as nightly builds, or on-demand. It’s a really smooth process as Discord is an integral part of our daily work (don’t tell that to our bosses).

And we also have Discord bots for deployment of builds into each of our nDisplay cluster, so if we want to test on our simulator the level we’re currently working on on our desktop, it’s just a Discord command away.

Conclusion

The production pipeline outlined above allows us to really focus our work on what’s important: building immersive worlds and scenarios, tailored made for each research subject we’re working on. The template superproject means we can start working immediatly, without any other requirements than clicking the Fork button. And the build scripts included within the template, along with the Discord bots, make it so that what you see on your desktop can be running on any of our simulation platform in just a few minutes, with just a Discord message needed to trigger the process.

This pipeline works really well for us, and we’re constantly looking for new ways to improve it. What works for us might not work for you, but understanding your overall process and making it as smooth as possible for all parties involved goes a long way to making everyone’s job that much better.