AI Audio Research | Game Sound Generation Latency

I built an Unreal Engine 5 client that asks for sounds during play and a Python server that generates them with AudioGen, MMAudio or the ElevenLabs API, and timed each sound from the gameplay event to playback. Every model missed the latency budget on a first request; cached and pregenerated sounds met it.

2025–2026. Solo researcher and developer, supervised by Roel De Meulemeester. Unreal Engine 5.4, C++, Python. Thesis: Real-Time AI-Generated Sound Effects for Video Games, 68 pages.

From the forest path into the chapel

0:00Footsteps
An AnimNotify on the run animation asks for each step (“footstep crushing soft grass”), a physics event for the landing. Only the first step on a surface waits for the model.
0:12Forest ambience
Entering the forest’s trigger volume prepares its ambience. It crossfades in, so it can arrive seconds late.
0:20The chapel door
The opening and closing sounds are requested on the approach and are in the server log before the player reaches the door.
0:28Inside the chapel
The chapel has its own ambience, requested as “dark castle ambience”.

Measured latency

I timed every request in Unreal from the gameplay event to audible playback, with generation or network transfer and decoding included. Footsteps, impacts and UI sounds have a budget of 100 to 150 ms. Generated on the spot, a footstep arrives more than a second late.

End-to-end latency in Unreal, 5th to 95th percentile, and the latency of one AudioGen footstep by caching strategy
Method	Delay on a scale from 0 to 4 seconds	Measured
Procedural DSP		15–25 ms
MMAudio Small		1.4–1.9 s
AudioGen		2.1–2.8 s
MMAudio Large		2.6–3.4 s
ElevenLabs		1.8–3.5 s
One AudioGen footstep and the cache
First request, cache miss		2400 ms
Same step again, cache hit		45 ms
Predicted step, pregenerated		< 10 ms

Bars run from the 5th to the 95th percentile of clips of 0.5 to 2 seconds, five runs per method and prompt (table 5.1). Footstep rows: one 0.5-second AudioGen clip (table 5.3). Test machine: AMD Ryzen 9 7940HS, RTX 5070 Laptop GPU with 8 GB VRAM, 64 GB RAM, Windows 11, Unreal Engine 5.4.4, Python 3.11.7.

Prepared, cached or procedural

The demo in the Unreal editor: the player still on the path beside the chapel door, while the server window already lists request 6, wooden door creaking open — The player is still on the path, and the server log already lists request 6, “wooden door creaking open”.

Close-up of the server log in the same frame: requests 1 to 5 with their generation times, and request 6, wooden door creaking open, still without a time — The player is still on the path, and the server log already lists request 6, “wooden door creaking open”.

Generation works for sounds the game can ask for early, play again or wait for, such as ambience. My system works around the delay: it prepares sounds before the player reaches them, keeps every result, and plays a procedural sound when nothing else is ready.

Unreal sends each prompt through a C++ WebSocket client to the Python server. The PCM audio arrives on a network thread, and since Unreal creates a USoundWave only on the game thread, a task-graph dispatch builds it there.

The server stores each result under a SHA-256 hash of the prompt, model, length and sample rate, up to 500 MB in RAM and the rest on disk. Unreal varies each cached copy with a pitch shift within 2 semitones, a volume change within 3 dB, a low-pass for occlusion and the reverb of its zone.

A generation counts as failed when it comes back silent, misses its length by more than 20%, takes over 10 seconds or crashes the decoder. In a stress test of 500 generations per method, 24 (ElevenLabs) to 69 (MMAudio Large) failed and procedural DSP never did. After a failure the server goes down this list:

Retry once with a new seed
Try another model
Hand over to the procedural sound

The local models share the laptop’s 8 GB GPU with Unreal, which already holds 4 to 5 GB, so the server loads and unloads them on demand, and the game plays a cached or procedural sound while one loads.

A fourth model, TangoFlux, never ran: the stable PyTorch 2.4 had no prebuilt kernels for the laptop’s Blackwell GPU (sm_120).

Listening comparison

Four prompts, each generated by four methods
Prompt	Procedural DSPsynthesis	AudioGenlocal model	MMAudiolocal model	ElevenLabscloud API
Heavy footstep on wet stone
Machine gun burst
Cat meowing
Wooden door slam

For the thesis I rated 60 anonymised samples, 12 per method, loudness-matched to −18 LUFS, from 1 to 5 for how well each would work in a game. I was the only rater.

ElevenLabs averaged 4.3 with clear transients, but it sometimes answered a sound-effect prompt with music. MMAudio Large (3.9) and Small (3.5) had good textures and inconsistent lengths. AudioGen (3.6) was reliable but plain. Procedural DSP (2.4) sounded synthetic and “video gamey” but consistent, and it was the only method fast enough on a first request.

I also built a Python prototype, AI Audio Lab: choose an engine, type a prompt, read the generation time against real time, listen and rate the result from 1 to 5. The prototype logs each generation with its rating.