Image description
Hello, developers! 🚀 Today, we’re excited to introduce 3D GPT—a project that blends AI, voice interaction, and 3D visualization to create immersive conversational experiences. Whether you’re a hobbyist or a seasoned developer, this demo showcases how to bring characters to life in a browser with ease. Let’s dive into the details!

What is 3D GPT?

3D GPT is a technical demonstration of how AI-driven 3D characters can interact with users through natural voice conversations. It lets you:

  • Import VRM-formatted 3D models (a popular format for virtual characters).
  • Customize voice settings to match your character’s persona.
  • Generate dynamic responses with emotional expressions for lifelike interactions.

Core Features & Technologies

The magic of 3D GPT lies in its integration of cutting-edge tools:

  1. Voice Recognition

    Powered by the Web Speech API, the app converts your spoken words into text, enabling real-time interaction.

  2. AI-Powered Text Generation

    Responses are generated using Ollama API (running on CPU inference), which leverages the ChatGPT API under the hood. This ensures context-aware, emotionally rich dialogues.

  3. Text-to-Speech (TTS)

    The Conqua TTS API (via Audio API) transforms AI-generated text into natural-sounding speech, complete with tone adjustments.

  4. 3D Character Rendering

    Built with @pixiv/three-vrm, a library for rendering VRM models in Three.js, characters are displayed with smooth animations and expressions.

Getting Started: Run 3D GPT Locally

Ready to experiment? Here’s how to set up the project:

  1. Clone the Repository 
git clone git@github.com:izdrail/agent.izdrail.com
  1. Install Dependencies 
npm install
  1. Start the Development Server 
npm run dev
  1. Access the App Open your browser to http://localhost:1603 and start chatting with your 3D character!

Explore the Demo

Not ready to code? Try the live demo on Glitch:

👉 3D GPT Demo

APIs & Services to Explore

  • Ollama API: For AI text generation. Learn more about its capabilities and integration.
  • Conqua TTS: For high-quality text-to-speech. Check out the Audio API documentation for customization options.

Why This Matters

3D GPT isn’t just a demo—it’s a playground for innovation. Imagine:

  • Virtual assistants with personality.
  • Educational avatars for immersive learning.
  • Gaming characters that adapt to your emotions.

By combining open-source libraries and APIs, this project lowers the barrier to creating next-gen interactive experiences.

Join the Journey

We’d love for you to:

  • Fork the repo and tweak it to your needs.
  • Share your creations with the community.
  • Report issues or suggest features on GitHub.

Let’s push the boundaries of what’s possible with AI and 3D tech! 🌐✨

Happy coding!

The 3D GPT Team