Over the past few weeks, I developed Luminary AI Kit — a full-stack, open-source AI chatbot starter built with Next.js, TypeScript, and the Vercel AI SDK. It supports streaming responses, tool calling, math rendering, vision, and much much more.

This project wasn't just about writing code — it was a deep dive into the nuances of building intelligent assistants. Here are five key lessons I learned along the way.

1. Rendering Math in AI Outputs: KaTeX vs. MathJax

Displaying mathematical equations in AI-generated content requires careful consideration. Initially, I explored MathJax, which renders equations as SVGs. While it offers broad LaTeX support, I found it less performant and harder to style consistently.

Switching to KaTeX was a game-changer. KaTeX renders math synchronously into HTML, allowing for easier styling and better performance. It also supports server-side rendering, ensuring consistent output across environments.

In Luminary AI Kit, I integrated KaTeX using better-react-mathjax, enabling seamless inline and block-level math rendering within the chat interface.

2. Navigating Context Windows and Model Limitations

Working with various language models revealed significant differences in context window sizes and capabilities. For instance, models like Gemma 2 9B have a context window of 8K tokens, which can be limiting when handling complex tool calls or maintaining extended conversations.

I learned that while smaller models are cost-effective, they often struggle with tasks requiring extensive context. For more robust performance — especially with tool calling — models like GPT-4 or Gemini 2.5 Pro are preferable due to their larger context windows and better handling of complex instructions.

3. Extending the Vercel AI SDK with Custom Tools

The Vercel AI SDK provides a solid foundation for building AI applications, but I needed additional functionality not available out-of-the-box. Specifically, I wanted to implement a custom web search tool, which is not available on models outside of the popular provider state - of the art options (OpenAI web search, Anthropic web search etc.).

By leveraging the SDK's support for custom providers and tool calling, I was able to create a bespoke solution. This involved defining a new tool schema and integrating it into the existing agent framework.

This experience underscored the flexibility of the Vercel AI SDK and the importance of understanding its extensibility features to tailor AI applications to specific needs.

4. Embracing Free Tiers for Development

Building AI applications can be resource-intensive, but I discovered that it's entirely feasible to develop and test robust applications within the constraints of free-tier offerings.

By carefully selecting services and optimizing usage, I managed to keep costs minimal without sacrificing functionality. This approach is particularly beneficial for small to medium projects, allowing for experimentation and iteration without significant financial investment.

5. The Limits of Freely Provided Models via Groq & OpenRouter

One of the key lessons from building Luminary AI Kit was understanding the trade-offs when using freely accessible models through platforms like Groq and OpenRouter. While these tools are fantastic for bootstrapping and experimentation, they come with practical limitations.

Many of the open-weight models offered — such as gemma-2-9b-it — struggle with context retention, instruction following, and coherence, even after significant tuning of system prompts. In my experience, these models often:

  • Ignore or distort system prompts even after refinement.
  • Fail to follow structured reasoning or tools logic, especially when invoking tool calls.
  • Regress when context starts to fill up, leading to hallucinations or empty responses.

While Groq’s blazing fast inference and OpenRouter’s vast model selection are tempting, their token-per-minute (TPM) limits and smaller context windows (e.g., 6k) make them unsuitable for more complex tasks like tool chaining, multi-turn conversations, or vision + text hybrid flows.

In contrast, closed-source models like GPT-4 or Gemini 1.5/2.5 Pro provide far more stable and coherent behavior across larger contexts — something that matters a lot when your app needs to track tools, documents, or math over long sessions.

Takeaway: These free models are great to learn with, but not always production-ready. If you're building something serious, budget for higher-end models or be ready to optimize around their quirks.

Final Thoughts

Building Luminary AI Kit was a journey of exploration and learning. It reinforced the importance of:

  • Choosing the right tools and libraries for specific tasks.
  • Understanding the limitations and strengths of different language models.
  • Leveraging extensible frameworks like the Vercel AI SDK.
  • Being resourceful and cost-conscious during development.

If you're interested in building AI-powered applications, I invite you to explore Luminary AI Kit. Feedback and contributions are welcome!