When Your Halloween Project Needs a Full Voice Pipeline
An NVIDIA engineer just built a talking garage door for Halloween. Not a simple motion-triggered speaker. A full conversational agent that listens to trick-or-treaters, responds in character as a grumpy old pirate ship, tracks conversation state, and—critically—softens when visitors remember their manners.
MonsterHouse runs entirely locally on a single workstation with dual RTX Blackwell GPUs. The system coordinates speech recognition, language modeling, zero-shot TTS, and real-time facial animation without cloud calls or noticeable lag. This is what happens when you give someone access to NVIDIA's Nemotron Voice Agent stack and a suspicious amount of free time in October.
The personality layer is where things get interesting. This isn't a chatbot reading off a script. It's a stateful character with a backstory (haunted pirate ship timbers, naturally), emotional progression, and a weakness for "please."
The Stack: ASR → Bolt E4B → Magpie TTS → Audio2Face-3D
The Nemotron Voice Agent coordinates everything. Here's what's running:
- ASR:
nemotron-speech-streaming-en-0.6b.q8_0.gguflistens to visitors - LLM: Nemotron 3.5 Bolt E4B 4GB generates dialogue
- TTS: Magpie TTS Zeroshot synthesizes speech with emotion tags
- Animation: Audio2Face-3D drives the wooden mouth in Unreal Engine 5
- Vision: Nemotron 3 Nano Omni describes costumes; YOLO and ByteTrack handle counting and tracking
The compute split is smart: an NVIDIA RTX PRO 6000 Blackwell Max-Q runs the voice pipeline, while an RTX PRO 4000 Blackwell handles Unreal Engine graphics and Audio2Face-3D. Keeping animation frame rate consistent regardless of inference load matters when you're trying to sell the illusion of a sentient building.
Bolt E4B: Small Language Models for Characters
The model choice here is the most technically interesting decision. After testing more than a dozen candidates, the builder settled on Bolt E4B—a 4-bit quantized small language model purpose-built for local conversational agents.
Why Bolt over larger alternatives like Qwen3 30B-A3B? Memory footprint and latency. The 4GB quantized release fits the compute budget while maintaining character consistency. The builder had early access through NVIDIA and found it solid enough to beat the frontrunners on this particular task.
But raw capability isn't enough. A character needs more than a prompt that says "be grumpy." It needs to follow a master conversation flow, respond to what visitors actually said, track encounter state across turns, and crucially—stay won over once a visitor earns its approval. Nobody wants to negotiate with the same garage door twice.
The LoRA Layer
To shape Bolt into this particular stubborn ship, the builder trained a LoRA adapter on over 5,000 multi-turn conversations. The dataset was created and judged by a local agent and a very large LLM, running overnight while the builder worked their day job.
Each LoRA took an hour or two to train. The builder benchmarked results for continuity, timing, tone, quality, repetition, and speed. The goal: give in eventually without turning into a cheerful concierge. A little resistance is part of the charm. An endless argument with a garage door is not.
The BF16 checkpoint is available for comparison, but the 4-bit version shipped for production.
Latency Hacks: Cached Audio and Streaming Text
A haunted house loses authority if it takes too long to complain. The builder reports median end-of-speech-to-audio latency of 1.70 seconds with Bolt Q4 and TTS, dropping to 0.72 seconds for cached responses.
The pipeline releases speakable text before an entire answer is finished. The house starts talking while the rest of the sentence is still generating. Prepared greetings and candy jokes cover familiar encounters—the text bank saves language-model work, and cached recordings skip synthesis entirely.
(Note: speech-end detection currently waits for 300ms of silence, which adds to perceived lag.)
Zero-Shot Voice Cloning with Seven Moods
Magpie TTS Zeroshot turns text into the Queen's English using short reference samples instead of trained voice models. The house has seven emotional registers—grumpy, menacing, amused, delighted, and others—each requiring a different 3-10 second voice sample.
The builder added a "Sample your Voice" block in the UI's tool tab to record lines in their best happy pirate or angry pirate voice. Bolt E4B decides what to say; Magpie delivers it with the proper delight or despair based on emotion tags dictated by the prompt template.
Audio2Face-3D expects clean neutral voice—not too deep, not too high. Since the character needed a very deep demonic voice with echo and chamber effects, the builder split the audio: Audio2Face-3D gets clean speech from the samples, and a second audio stream adds adjustable real-time vocal effects for the speakers.
Sync matters. Voice, animation, and effects have to land together or the illusion breaks. The builder reports a 100ms compensation for Audio2Face-3D buffering, set once and forgotten.
The Character Rig: A Grin Full of Splinters
The mouth came first. The builder imagined the wooden garage door boards splitting into jagged lips, with burning flames illuminating the driveway from beyond whenever the cursed mouth opened.
Building the custom animation rig in Autodesk Maya was one of the more labor-intensive parts of the project. The character needed to look ancient, expressive, and mildly offended by the existence of visitors. With no eyes, no eyebrows, and no blinking, it's a surprisingly economical monster.
The facial expression shapes export to Audio2Face-3D for animation. Virtual fire and effects live in Unreal Engine behind the door. The backstory—house built from sunken pirate ship timbers, haunted by its stormy past—gives the nautical complaints narrative weight. This is the ship, reincarnated as real estate and not entirely pleased about it.
The Control Desk for a Talking Building
Somewhere along the way, "make the house talk" required an interface full of controls. The builder can audition cached lines, adjust opening mood, follow conversations, change voice treatment, and tune mouth animation.
Tabs cover conversations, model selection, recordings, prompts, performance monitoring, and camera feeds. A soundboard and custom dialogue field let the operator put words in the monster's mouth. There are also model-loading controls and an Unreal Engine connection indicator, because sometimes the monster isn't sulking—it's just disconnected.
What This Demonstrates About Local Voice Agents
This project is a proof point for entirely local conversational AI. No cloud calls. No privacy concerns about recording trick-or-treaters. No dependency on network uptime during peak Halloween traffic.
The Nemotron Voice Agent framework handles coordination across the stack, making it feasible to swap models, test alternatives, and iterate on character performance without rewriting infrastructure. The builder tested over a dozen LLMs before settling on Bolt E4B, benchmarking each with trained LoRAs.
Small language models like Bolt E4B make character work practical on local hardware. You don't need 70B parameters to sustain a personality when you can fine-tune a 4GB model on curated dialogue. The tradeoff is upfront dataset work—5,000+ conversations—but that investment pays off in consistency and control.
Zero-shot TTS removes the voice training bottleneck. The builder recorded seven mood samples instead of spending weeks on voice data collection. Magpie handles the rest.
The Halloween Production
The performance needed a guide: greet visitors, brush them off with remarks like "I'm busy grumbling, go bother the neighbors!", but let persistence wear him down. A "please" could break through immediately. "Thank you" earned a warm farewell. Once won over, stay won over.
This is character state management in production. The house tracks encounter progress, adjusts tone based on visitor behavior, and remembers its relationship with each group. It's a conversational agent with a grudge, a soft spot, and a candy bowl.
And it runs on hardware you could fit in a garage. Which is good, because that's where it lives.