Self-hosted voice for AI agents: Kokoro, Whisper and open source TTS

What is a self-hosted speech server?

It's a small web service on a machine you control that answers OpenAI's audio API: POST /v1/audio/speech turns text into audio and POST /v1/audio/transcriptions turns audio into text. Because the shape is shared, one client works with many servers and models.

Kokoro-FastAPI serves the Apache-licensed Kokoro-82M model and only speaks. Speaches does both, with faster-whisper for transcription and Kokoro or Piper for speech. mlx-audio runs Kokoro, Whisper and many newer models on Apple Silicon. Rakazo's default model names match mlx-audio's.

Kokoro TTS

Kokoro-82M is the open model most people start with. It has only 82 million parameters, is licensed Apache 2.0 for commercial use, and ranks close to much larger models on public listening tests. It ships 54 preset voices across American and British English, Spanish, French, Hindi, Italian, Japanese, Brazilian Portuguese and Mandarin; the English voices such as af_heart are the strongest. It can't clone a voice, only blend the presets.

It is fast enough for a CPU. Testers report about 6 times faster than real time on an M3 Pro's CPU, under a second to first audio on Apple Silicon, and 35 to 100 times real time on a mid-range NVIDIA GPU, using 2 to 4 GB of memory. On a Raspberry Pi 5 it runs at roughly real time. Kokoro sounds best on sentences of moderate length, which suits Rakazo, since replies are spoken a sentence at a time.

Best open source text to speech models in 2026

Kokoro is the safe default, but it isn't the only option. The table lists the open models with the most use, what their licenses allow, and whether you can put them behind the OpenAI-shaped API Rakazo talks to. Several of the highest-rated open-weight models are research or non-commercial only, which matters if your bots work for a business.

From each project's README or model card, checked November 2026. Check the license file yourself before commercial use.

Open source speech to text

For transcription, Whisper is still the default. whisper-large-v3-turbo (MIT, 809M parameters) is several times faster than large-v3 with similar accuracy in the main languages, and faster-whisper runs it with int8 in about 3 GB of GPU memory. On a CPU, a smaller Whisper model is the practical choice, and short hold-to-talk clips transcribe in a second or two. NVIDIA's Parakeet TDT 0.6B (CC-BY-4.0) is far faster and among the most accurate open models for English and 25 European languages, and runs through mlx-audio on a Mac.

Rakazo sends transcription to the same address as speech, so pick a server that does both: Speaches (faster-whisper with Kokoro or Piper) or mlx-audio on a Mac. Kokoro-FastAPI alone only speaks.

What self-hosted voice costs

The models are free; you pay for the machine. If Rakazo already runs on a Mac or a PC at home, Kokoro and a small Whisper model fit alongside it and the extra cost is electricity. A dedicated machine for voice ranges from a small rented CPU server to a GPU.

From the Hetzner, Apple and RunPod pricing pages, checked November 2026.

Self-hosted vs hosted voice APIs

The worked example is one bot that reads about 30 minutes of replies aloud each day and transcribes about 30 minutes of what you say, every day for a month: 15 hours of speech (about 810,000 characters) and 15 hours of transcription. Hosted, that ranges from about $14 on Google to about $49 on Cartesia a month. Self-hosted on hardware you already have, it costs close to nothing, and on a small rented server about $10. A GPU only pays off with many users: at $0.34 an hour, always on, it costs more than ElevenLabs until you generate about 113 hours of speech a month.

What you give up is polish. The best hosted voices still sound more natural than Kokoro, cloning is easier, and someone else keeps the server running.

Usage prices from each provider's pricing page, checked November 2026. Where speech is bought as a monthly plan, as with Cartesia, the cost shown is the fee for the smallest plan that covers it. Rakazo doesn't charge for voice: you connect your own key and the provider bills you.

Hosted servers that copy OpenAI's audio API

Self-hosted isn't limited to your own machine. It accepts any public HTTPS address with an API key, so a hosted service that serves OpenAI's audio endpoints can work there too: DeepInfra and Together AI host Kokoro for $0.99 and $4 per million characters, and Groq serves Whisper for $0.04 an hour. We haven't tested these with Rakazo, and two details matter. Rakazo sends speech and transcription to the same address, so the service needs to do both. And the voice list comes from the server; when a server doesn't publish one, Rakazo offers Kokoro's voice names, which only work if the service runs Kokoro.

Deepgram, Google, Azure and Amazon use their own API shapes, so Self-hosted can't reach them directly.

Prices in US dollars from each provider's pricing page, checked November 2026. An hour of speech is taken as about 54,000 characters (150 words a minute); real voices run a little faster or slower. Fish Audio bills per UTF-8 byte, which is one byte per character in English.

What you can do with self-hosted voice in Rakazo

Rakazo sends both speech and transcription to the one Server URL you enter. If that server does both, you get every voice feature in the voice and calls guide.

• Talk instead of typing. Hold the mic in the composer and your server transcribes what you say.
• Hear replies. Choose Speak this reply on a message, or turn on Read replies aloud in a bot's settings.
• Call a bot with Call at the top of the chat, or let a bot call you, for example when you ask it to call you at 5.
• Pick a voice from the server's own list. Kokoro voices show their language, and Rakazo passes that language to the server so a French voice reads with French pronunciation.
• Choose the models. Speech model and Transcription model take whatever names your server uses.

When to choose self-hosted voice

Choose it when you don't want a hosted vendor to process your voice, when you want voice with no vendor account or usage bill, or when you already run local models. It pairs naturally with a self-hosted Rakazo and a local model, so the whole assistant can work offline.

Hosted providers are less work and usually sound more polished. ElevenLabs and Fish Audio offer cloning, OpenAI reuses a key you may already have, and Cartesia is tuned for low latency. If Rakazo runs directly on a Mac, Server (Mac) speaks with nothing to install, but doesn't transcribe. See the comparison for all six.

Limits and network rules

• Quality and speed depend on your models and hardware. Whisper on a CPU can be slow on a call.
• A speech-only server such as Kokoro-FastAPI can't transcribe. Use a server that does both if you want to hold the mic or make calls.
• Addresses on your own network, localhost and host.docker.internal are refused unless the server sets MCP_ALLOW_PRIVATE_ENDPOINT=true on the API and worker. That setting applies to every user, so leave it off on a public install.
• A hostname that resolves to a private address must also be listed in RAKAZO_OPENAI_COMPAT_PRIVATE_HOSTS. Cloud metadata and link-local addresses are always blocked.
• A public address must use HTTPS if you add an API key. Setting RAKAZO_OPENAI_COMPAT_ALLOW_PUBLIC=0 blocks public servers, and hides the option if private ones aren't allowed either.
• Watch the license. Several popular open models, including F5-TTS, XTTS-v2 and Fish Audio's open weights, aren't free for commercial use.
• On the mobile app, calls can use your phone's own voices if you pick a Device voice in the bot's settings. Otherwise replies use this provider.

How to set it up

Whoever runs the Rakazo server allows the address once; then each person connects in the app. The self-hosted model settings cover the network options in full.

1. Start a speech server, for example the Kokoro-FastAPI Docker image, Speaches, or mlx_audio.server on a Mac.
2. On the Rakazo API and worker, set MCP_ALLOW_PRIVATE_ENDPOINT=true for a server on your network, and PHONE_CALLS_ENABLED=true to turn on hold-to-talk and calls. Restart them.
3. Open Settings → Voice, set Provider to Self-hosted and enter the Server URL, such as http://192.168.1.10:8880/v1. Rakazo adds /v1 if you leave it off. If Rakazo runs in Docker and the speech server is on the same machine, use host.docker.internal instead of localhost.
4. Add an API key only if your server needs one. Change Speech model and Transcription model if your server uses other names; Kokoro-FastAPI expects kokoro.
5. Choose Connect, pick a Voice and choose Hear a sample.