KEMBAR78
Add KHR_audio_emitter support to the GLTF module by aaronfranke · Pull Request #88204 · godotengine/godot · GitHub
Skip to content

Conversation

aaronfranke
Copy link
Member

@aaronfranke aaronfranke commented Feb 11, 2024

Implements this proposal godotengine/godot-proposals#8814

This PR adds support for audio import and export in the GLTF module using the not-yet-finalized KHR_audio_emitter GLTF extension (KhronosGroup/glTF#2137). This allows you to save audio inside of GLTF scenes, and load them back later. In the future this will also allow you to use GLTF as an interchange format between game engines.

You can try out a Three.js implementation here: https://omigroup.github.io/three-omi/

Some example use cases: a fountain that makes water noises, a gun that makes custom sounds when fired, the radio from Portal that plays music on a loop, or a tree that includes bird chirping noises or rustling leaves or something. For more details about the intended use cases, see the proposal.

Freely licensed example file: https://github.com/omigroup/gltf-extensions/tree/main/extensions/2.0/KHR_audio_emitter/examples/boom_box This file contains a boom box, a short looping music clip, OMI physics, and licensing information via KHR_xmp_json_ld. The model is CC0, created by The Khronos Group, and the music is CC-BY 3.0, created by Kevin MacLeod.

The code in this PR is ready for review, but note that the extension is not yet finalized. Usually my approach is for us to be pioneers with Godot and implement extensions that may not be finalized, like with OMI physics. However, for this extension, since it is using the KHR_ namespace, we must tread carefully and avoid shipping features in Khronos's namespace into production without Khronos's approval. Don't put words in their mouth, so to speak.

Production edit: closes godotengine/internal-team-priorities#44

@fire
Copy link
Member

fire commented Sep 1, 2024

We are waiting for the ratification of the various gltf extension standards.

@aaronfranke
Copy link
Member Author

Marking as draft to prevent accidental merging before ratification. However, the code is complete and ready for review.

@Calinou Calinou added this to the 4.x milestone Feb 13, 2025
@aaronfranke aaronfranke force-pushed the gltf-audio branch 2 times, most recently from 772bb1c to 8bfbb96 Compare March 9, 2025 22:29
@hybridherbst
Copy link

hybridherbst commented Apr 4, 2025

We're currently looking at adding KHR_audio_emitter support to UnityGLTF (for export and import), and I just tested some of our files on this Godot branch :)

Is it intentional that currently, embedded audio buffers don't seem to be supported? On playback of such a file, I get an error:

image

The file in question: 20250404-UnityGLTF-Audio.glb.zip

Godot seems to correctly see that there's an audio buffer embedded, its name and settings, but can't play it back. But our export implementation is still WIP – so it could also be that the file isn't valid in some way, I'm not sure if there's another player/implementation that supports the extension in its current form.

@aaronfranke
Copy link
Member Author

@hybridherbst Thank you for the test file. However, the problem is in the MP3 import code, not in the glTF importer.

Screenshot 2025-04-04 at 2 56 22 PM

As a test, I tried to save the data to a MP3 file and got this: test.mp3.zip. This correctly plays inside of VLC Media Player, but Godot is unable to import it:

Screenshot 2025-04-04 at 2 56 10 PM

The Godot MP3 import code is pretty trivial, it just calls into the MiniMP3 third-party library. The function returns an error code of 0 (so, no error), but the Hertz of the file is zero. I checked every function in MiniMP3 that contains hz = and none of the functions are even being called. I would open an issue on MiniMP3 but the library seems to be abandoned, maybe @godotengine/audio could chime in.

Anyway, it's not a bug in the glTF code. If Unity exports a glTF with an MP3 that Godot could normally read as a standalone file, then that same audio data should work inside of a glTF.

@aaronfranke aaronfranke changed the title Add audio support to the GLTF module Add KHR_audio_emitter support to the GLTF module Apr 4, 2025
@hybridherbst
Copy link

hybridherbst commented Apr 8, 2025

Thanks for looking at it! – turns out that the actual issue was a wrong mime type, the file specified mp3 but actually was a WAV. This made VLC and others that don't care so much play it back, but Godot (rightfully) rejected it. So the file you extracted would have needed to be renamed to .wav to work...

Here's an updated file with an mp3 and correct mime type:
20250408-UnityGLTF-Audio.glb.zip

@fire
Copy link
Member

fire commented Apr 8, 2025

I have critical concerns that WAV is in the spec and would cause portability issues with the Khronos specification.

@aaronfranke
Copy link
Member Author

@fire What issues? The corrected model from @hybridherbst loads fine.

@coderofsalvation
Copy link

coderofsalvation commented May 19, 2025

@fire is it possible you've misread the spec? it's both mp3 and wav.

"Unless specified by another extension, the only supported mimeTypes are audio/mpeg and audio/wav." <-- this is in the spec of this particular glTF extension not the glTF spec itself etc.

@aaronfranke aaronfranke force-pushed the gltf-audio branch 2 times, most recently from adce79c to 95fa474 Compare August 22, 2025 22:04
@lyuma
Copy link
Contributor

lyuma commented Aug 23, 2025

@coderofsalvation what you refer to as "the spec" is an unofficial proposed pull request by OMI (which many of us participate in) against a draft KHR spec. Unfortunately Khronos has been extremely slow formally ratifying or publishing this particular extension, but that means while very solid and implementable, the draft extension on a non-khronos repository cannot be treated as canon.

I believe @fire is referencing some contention related to certain formats from some of the Khronos meetings, not all of which is visible publicly. I attended one such public meeting over two years ago where the subject of audio formats came up, but I am not sure about the current state so this is speculation on my part.

anyway I just wanted to illustrate some of the context here. The KHR audio spec has been in work for over 4 years so there is quite a long history...

@coderofsalvation
Copy link

@lyuma thanks for providing context
@aaronfranke thanks for the new commit

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Work in progress

Development

Successfully merging this pull request may close these issues.

7 participants