GitHub Copilot recently launched an agent mode. As a developer open to new things, I plan to try it out. The article is a working log of my experience.

For a trial project, I chose to implement a static blog generator that supports RAG, Chating and llms.txt generation. It is also from my own needs. Finally, I implemented a blog with the following features:

  • Static blog generation, of course, it has a tag system. But I decide not to implement comments feature.
  • Support for in-page AI dialogue, with RAG, syntax highlighting for code blocks and sanitization.
  • Support for llms.txt generation.

Screenshots

Blog Site

blog-site

Chat UI

chat-ui

RAG

rag-1

rag-2

llms.txt

  • llms.txt

llms.txt

  • llms-full.txt

llms-full.txt

Now, let's go!

Building Log

Preparation

Before starting, I highly recommend watching the video to know how to use it. Here is the video for you: VS Code Agent Mode Just Changed Everything

For how to install and configure, I would like to ignore it to save your time.

Action 1: Project Initialization

With so many excellent static blog generators available today, our project doesn't need to start from scratch at all.

Here, I chose Astro and used its create command to initialize a new project, remembering to select the blog template.

Experience 1: Vibe Coding doesn't mean abandoning previous experience; if there's a framework, go for it.

Action 2: Adding Tailwind CSS

For styling, Tailwind CSS is a must!

If you've watched the video above, you should know that in the current situation, the agent will generate incorrect code for you, including installing the wrong dependency packages and incorrect configuration code. Then, following the video, just have the agent #fetch the documentation first.

Similarly, it's recommended to follow the video and configure the corresponding copilot-instructions.md file, which is located in the .github directory. This is to set project-specific behavioral guidelines for the agent.

In addition, since the llms.txt is a new trend, many tools are starting to have their own llms.txt files. Therefore, it is strongly recommended to have the agent use it. Here, for simplicity, I directly created an llms-txt folder and put the various llms.txt and llms-full.txt files inside it.

Furthermore, you can also try Langchain's mcpdoc.

The video also suggests writing a PRD document, which is a product requirements document (actually just user stories). Personally, I don't think it's necessary because we all know that requirements often change, and initial requirements are mostly unreliable. Plus, the agent is an LLM with context size limitations, so it's more practical to put these in dialogues.

However, providing some background description is still meaningful and helps the agent understand the project's background and goals. In practice, this doesn't change much.

Here's a summary of other preparations mentioned in the video:

  • List the tech stack and best practices in copilot-instructions.md.
  • Best practices can be generated by AI.

Experience 2: After initializing the project, define agent behavioral guidelines and provide the necessary background.

Action 3: Adding Tags

This step went very smoothly, completed in just a few rounds of dialogue, including style adjustments, Content Schema definition, and more.

The reason for the smoothness is simple: it's a regular requirement.

Experience 3: Regular requirements, simple dialogue.

Action 4: Indexing Blog

The core of our requirements is a RAG system for a personal blog, so we need to index the contents. Here, the AI tried several times, but the results were not good, even after I explicitly told it to read llms.txt.

This requirement essentially has three parts:

  1. How to index.
  2. Indexing each post.
  3. Since it's a static site, indexing needs to happen at build time.

Only the last one was correct, the agent knew to call which function and to integrate it with Astro's build process.

The other two were less satisfactory:

  1. For the first task, it generated incomprehensible code. Even after explicitly specifying the technology choices, langchain and pgvector, the generated code was still unusable. This was somewhat expected, as the agent may not be able to keep up with these rapidly updating libraries.
  2. For the second task, indexing each post, it stubbornly stuck to a fixed mindset, always trying to parse markdown files using Astro's way. It didn't try other methods when it failed.

For this, manual intervention was necessary.

For the first task, it was simple. Because I'm building a RAG product, these AI-related functions can be copied from that product codebase with minor modifications.

For the second task, I stopped its attempts and suggested that since it was a build-time task, it might as well solve it by directly reading the files instead of trying different ways with Astro.

This time, it finally took a step in the right direction. The overall logic was fine, and it knew to call the indexing function that had just been completed. However, it still couldn't shake its tendency to over-engineer, and ran into trouble with the post metadata, even suggesting a library to handle it.

So, I stopped it again and told it to try regular expressions. This time it finally succeeded and passed my manual testing: during the build process, the posts were correctly indexed and stored in the database.

It sounds simple, but the whole process took quite a bit of back-and-forth time.

At this stage, we completed:

  1. Indexing each post.
  2. Incremental indexing, which is very important.
    • Because indexing is a time-consuming process.
    • By default, the index would be recreated every time the site is built, which is wasteful.
  3. Indexing is done at build time, perfectly integrated with Astro's build process.

Experience 4: Manual intervention; step in when necessary.

Action 5: Adding Floating Chat UI

This step was quite troublesome. The reasons are partly related to my lack of expertise in frontend development and Astro. But that's not the main reason; please see the details.

First, the UI implementation was changed several times (Dont' blame AI, that's my idea): react -> CopilotKit -> react.

  1. Initially, I asked the AI to implement a version using react.
    • Here, I encountered issues with Astro's ui island feature and React component rendering.
    • The AI tried several times, and also repeatedly attempted outdated Tailwind integration methods.
    • Its performance was similar to the stubbornness mentioned above, and again, none of them succeeded.
    • I also tried manually a few times, and all failed as well, after all, I'm not a frontend expert!
    • Later, I remembered Astro's ui island feature and hinted the AI to think in that direction.
    • This time it finally succeeded. After the AI adopted inline CSS, the rendering issue was resolved.
  2. Then, I considered using CopilotKit to lay the foundation for advanced chat UI solutions in the future.
    • After trying it, I found that CopilotKit doesn't support Astro well, and after reading its documentation, I realized it didn't fully match my needs, so I eventually gave up.
    • The AI didn't play a role here.

During this period, I also asked the AI to optimize the layout of the original Astro blog template and make other minor UI adjustments.

Experience 5: AI won't help you much if you are not an expert on what you're working on.

Action 6: Chatting with AI

With the util functions implemented before, this task was much smoother. With a single sentence, I had the Agent complete the integration. It accurately identified the function to call.

Similarly, with a single sentence, I had it add syntax highlighting to the output from the LLM.

After that, I optimized the function, switched to LangGraph ReactAgent, and adjusted the prompt to a RAG template. After completing these steps and replacing the function name, a simplest RAG version was done.

At this point, we already have a simplified RAG-enhanced static blog generator that supports hybrid search.

Experience 6: AI is a powerful tool if you know what you are doing.

Action 7: Streaming

As a common feature for chatting, streaming is a must-have feature. The implementation here was also quite interesting.

After I manually adjusted the util function to streaming mode (it was commented out before), the agent did implement streaming response return in the backend API endpoint.

However, the Agent encountered trouble when modifying the frontend. After prompting, it understood that it needed to use the AI SDK and also knew to use the AI SDK's React components. But the implementation was still not good. It didn't realize that it could directly use useChat to implement the UI logic and leverage LangChainAdapter to simplify the code.

The funniest thing was that it actually implemented the AI SDK's streaming protocol itself, and it did work 😄. But it consistently failed to refactor it into what I expected.

Finally, I manually replaced with LangChainAdapter, while the agent independently used useChat to complete the frontend's streaming logic.

Experience 7: If you're not picky, AI can generate bad but working code. However, making it generate code that conforms to current documentation and has a certain quality requires effort.

Action 8: Tailwind CSS Optimization and Other Manual Tasks

At this point, the main functionality was done. So, I decided to have it optimize the CSS of the entire project.

After several rounds of attempts, the AI consistently failed to generate code that met my expectations. Ultimately, I chose to seek help from the frontend colleagues in my team, who completed the CSS optimization.

Here, the project was also modified to support Vercel deployment, which was done manually. Because it was simple enough that doing it myself was faster than having the AI do it.

Action 9: llms.txt Generation

The AI also didn't play a role here because I thought there should be readily available answers for llms.txt online.

Sure enough, searching with the keywords "astro llms.txt" yielded a direct answer:
How to generate llms.txt and llms-full.txt with Astro.

After reading the original article, I found that it could be used directly in the project with minor adjustments. So, I didn't have the AI do it.

This is actually the same as what was mentioned earlier: having AI doesn't mean old experience is no longer useful, lol.

Action 10: Optimizing Prompt and Sanitization

I didn't have the AI modify the prompt because the enhancement here was just a small one: adding the original text citation in the answer, a piece of cake.

For sanitization, the AI did a good job. I basically didn't intervene much, and it quickly provided code that met expectations. I guess the reason is the same as before – there's already plenty of code available.

Others

The rest were just some small tasks: code organization, style organization, documentation organization. These were mostly done manually, without AI involvement.

Summary

The entire project took about 3 days, not consecutively. This was interspersed with other work meetings and trivial matters. It wasn't as amazing as the AI video bloggers online who claim to build and launch an application in xx hours and then sit back and collect money.

If I were to rate this experience, focusing on the AI part: 6.8 out of 10. Overall performance was acceptable:

  • Stable performance for regular requirements, good support for popular tools.
  • Average performance for non-standard requirements, but with some highlights, sometimes suggesting new ideas.
  • If the user lacks experience, the AI performs too politely to be as an expert during pairing.

In terms of time spent, it was actually lower than my expectations. I originally expected to complete it within 2 days. Because for experienced developers, such requirements would probably take around 3 days of work without AI.

However, considering the collaboration between a AI and a non-frontend-expert, the overall performance was passable.

Reflections

Regarding vibe coding itself, here are a few additional points:

  1. Don't get hung up on whether vibe coding improves programming efficiency. In the current stage, vibe coding is fully capable of building POCs and MVPs.
    • From a product perspective, it can be used to quickly validate product ideas and facilitate communication between all parties.
    • If it doesn't work, it can be quickly adjusted or abandoned.
    • At this stage, technical architecture and code quality are not the most important; the solutions provided by AI are sufficient to meet the technical needs of this stage.
  2. AI's capabilities are still rapidly evolving. There's no need to dismiss it based on its current performance or reject it out of an old-timer's conceit.
    • Once the technology matures, what you're currently proud of will be worthless. Think about it, would you compete with a car to see who runs faster?
    • Learning how to collaborate with AI as early as possible is the key!

That's all! If you are interested in the code, you can buy it on my Gumroad page.

Do you vibe coding today, ;)?