This is a submission for the KendoReact Free Components Challenge.

What I Built

Dojo is an accessible collaborative WebRTC video chat with Generative AI.

To use it, share the url that's generated after visiting the demo link with the person you want to chat with

Image description

Project Overview

Dojo provides a seamless video chat experience with these features:

  • Real-time video and audio communication using WebRTC technology
  • Accessibility-focused design with keyboard navigation and screen reader support
  • AI-powered transcription to convert speech to text during conversations
  • Integrated AI chat assistant that can answer questions and provide information
  • Text messaging for when audio isn't practical
  • Sleek, Accessible UI built with KendoReact components

Demo

You can chat via the demo here: https://dojo-eight-beta.vercel.app/
Here is the code repository: https://github.com/jabartlett/dojo

Technical Implementation

WebRTC Infrastructure

The core of the application uses WebRTC for peer-to-peer communication. I implemented:

  • Socket.IO for signaling and connection establishment
  • ICE candidate negotiation for NAT traversal
  • Media stream handling for camera and microphone access
  • Data channels for text messaging and filter synchronization

I deliberately picked WebRTC because it's technically challenging, even with LLMs and cool stuff. You have to know Regex and stuff.

AI Integration

I integrated two key AI features:

  1. Speech-to-Text Transcription: Using Hugging Face's Whisper model to convert spoken audio to text, making conversations more accessible and searchable. So visually disabled users can use LLMs too, and use LLMs in tandem with people who have eyesight.

  2. AI Chat Assistant: Implemented using Phi-4, allowing users to ask questions and get intelligent responses during their conversations. Phi-4 is not too shabby, pretty good, really.

So I don't get some massive HuggingFace inference bill, I limited the AI responses to one each, but please contact me if disabled and I'll gladly remove the rate limiting for you!

Accessibility Features

Accessibility was a priority throughout development:

  • Full keyboard navigation support
  • ARIA attributes for screen reader compatibility
  • Black and white for easy contrast and responsive design
  • Transcription services for those with hearing impairments
  • Text alternatives for all audio communication

KendoReact Experience

The application leverages 10 free KendoReact components and they are shnazzy:

  1. KendoReact Buttons for intuitive call controls and actions
  2. KendoReact Animation for smooth transitions between states
  3. KendoReact Conversational UI for the chat interface
  4. KendoReact Dialogs for modal windows and notifications
  5. KendoReact Dropdowns for settings and filter selection
  6. KendoReact Indicators to show connection status and activity
  7. KendoReact Inputs for text entry and form controls
  8. KendoReact Notifications for system messages and alerts
  9. KendoReact ProgressBars for loading states and transcription progress
  10. KendoReact Tooltip for providing helpful context and instructions

Big props to Telerik for putting together super accessible components! Remarkably well done!

AIm to Impress

As I mentioned, this project (I didn't know what to call it, so it's called Dojo) integrates two powerful GenAI technologies to enhance communication:

  1. Speech-to-Text AI: I implemented Hugging Face's Whisper large-v3 model to provide real-time transcription of conversations. This not only makes the application more accessible but also creates a searchable record of discussions. The transcription service processes audio captured during calls and converts it to text with impressive accuracy, even handling different accents and background noise.

  2. Conversational AI Assistant: Using Microsoft's Phi-4 model through Hugging Face's API, I created an in-call AI assistant that users can interact with. This assistant can answer questions, provide information, and help with tasks during the conversation. The implementation includes context management to maintain conversation flow and rate limiting to manage API usage efficiently.

I <3 Comments!

Say hey and let's collab!

My Dev.to username is justin_bartlett_d31a8f55a