Local music and audio generation
Generating music and audio from a text prompt, locally. What the models can do, which run on modest hardware, and why the licence is the first thing to check.
What you’ll learn
What local music and audio generation can do today, which models are practical on consumer hardware, and, importantly, how their licences differ.
Check the licence first
Audio generation is the one generative-media area where the licence, not the hardware, is often the deciding factor. The models sit at very different points: one of the best-known is non-commercial only, while others are freely usable. So before you invest time in a model, confirm you can actually use its output the way you intend.
What you can run
- ACE-Step is the practical default for most people. It generates full songs with duration control, runs in as little as 8GB of VRAM, and is Apache 2.0, so it can be used commercially. That combination is what sets it apart.
- MusicGen from Meta is capable and widely supported, and comes in several sizes so the smaller ones run on modest hardware. The catch is decisive: its weights are non-commercial (CC-BY-NC), so it is for personal and research use only, not commercial work.
- Stable Audio Open is best thought of as a sound effects and audio-texture tool rather than a music or vocal generator. It is free for smaller organisations, with an enterprise licence required above a revenue threshold.
How it works
Most of these models use diffusion, the same denoising technique as image and video models, applied to an audio representation; MusicGen instead generates audio tokens one step at a time. Either way, they turn a text description into a waveform, and some accept a reference melody to guide the result.
How to run it
Each model runs from its own repository or through a Hugging Face pipeline, and generation is light enough that the smaller ones work on ordinary consumer GPUs, or even a capable CPU for short clips.
What can go wrong
- Assuming you can use the output commercially. MusicGen’s non-commercial licence is the classic trap. If the work is commercial, reach for ACE-Step instead.
- Expecting polished vocals. Local audio models are strongest at instrumental music, sound effects, and textures; realistic singing is still hard.
- Very long pieces. Coherence tends to fade past a few minutes, so long tracks mean generating and arranging sections.
Next steps
For the other half of generative media, see local image generation and local video generation. For turning speech into text and back, see speech-to-text and text-to-speech.