Introduction
In mid-April, I blogged about leveraging cloud and generative AI to maximize developer productivity. Among other areas, I highlighted the potential of AI in the PR Review/Gitflow process, suggesting tools like Evolua.io or CodeRabbit to offload routine code reviews and low-value fixes to AI assistants.
Today, I'm excited to explore Amazon's latest offering in this space: Amazon Q Developer in GitHub, which has just been released in preview. As an AWS Community Builder, I've had early access to this tool for over a week, and I wanted to put it through its paces with a real-world test:
Can Amazon Q Developer build a complete AWS SAM application from scratch, based solely on requirements provided in GitHub issues?
How Amazon Q Developer for GitHub Works
Before diving into my experiment, let's quickly review how Amazon Q Developer integrates with GitHub:
Issue-based development: Create a GitHub issue describing your feature requirements, add the "Amazon Q development agent" label, and Q Developer will generate a complete implementation and submit it as a pull request.
PR-based collaboration: Add the Q Developer label to an existing PR, and it will review your code, suggest improvements, or address requested changes.
In both cases, Amazon Q analyzes the context of your repository and comments, then either generates new code or modifies existing code. The AWS blog post explains the current capabilities and limitations of this preview version in detail.
My Experiment: Building an AWS SAM App from a Blank Repository
To test Amazon Q Developer's capabilities, I created a blank repository with just a README file. My goal was to see if Amazon Q could build a serverless photo application with AWS SAM, complete with GitHub Actions pipeline, solely based on requirements provided in issues.
Lesson #1: Crafting Effective Prompts is Critical
Following best practices for working with generative AI, I spent time crafting a detailed issue description that specified all the requirements for my AWS SAM application. This included:
- Application architecture details
- Required AWS resources
- Code structure
- Testing requirements
- GitHub Actions pipeline specifications
I submitted this as Issue #1, added the Amazon Q development agent label, and waited.
Lesson #2: Mind the Context Window
Within minutes, I received my first lesson in the limitations of Amazon Q Developer:
⚠️ The issue description is too long. I will only look at the first 1000 words.
Like most LLM-based tools, Amazon Q Developer has a context window limitation. My detailed prompt exceeded this limit, meaning it would only process part of my requirements.
Lesson #3: Concise Requirements Are More Effective
Taking this feedback, I used an LLM to help condense my requirements into a more focused version. I submitted this as Issue #2, focusing on just the essential specifications. While this issue acknowledged receipt of the task, it also encountered issues.
Lesson #4: Third Time's the Charm!
Not giving up, I created Issue #3 with an even more concise and structured requirements list. This time, success! Amazon Q Developer processed the issue and created Pull Request #4 with a complete implementation of the AWS SAM photo application.
Lesson #5: Amazon Q Developer Self-Checks Its Code
What happened next was less impressive than it should have been: Amazon Q Developer had to scan its own generated code for security vulnerabilities and other issues. Perhaps it should have simply written secure code from the start, rather than creating problems that required subsequent fixes.
However, it's valuable that Amazon Q Developer can perform an additional security scan on the changeset, providing an extra layer of protection to catch vulnerabilities before they're committed to the codebase.
Key Lessons from My Initial Testing
While my experiment hit some initial snags, the ultimate success revealed several valuable insights:
Master the art of concise prompting - Your issue descriptions must be detailed enough to guide Q Developer effectively while staying under 1000 words. This constraint forces you to prioritize the most critical requirements.
Start with focused, smaller tasks - Rather than requesting a complete application, begin with individual components or features to maximize success rate and learn the tool's capabilities.
Embrace an iterative workflow - Amazon Q Developer shines as a collaborative partner in an iterative development process, not as a one-shot solution. Be prepared to refine requirements and review generated code carefully.
Value the security scanning - The automatic security review of generated code is a standout feature that helps ensure you're not introducing vulnerabilities into your codebase.
Amazon Q Developer: Production-Ready or Preview Promise?
Despite being in preview, Amazon Q Developer shows remarkable potential for transforming development workflows. The GitHub integration feels natural, and the ability to generate code directly from issue descriptions could dramatically accelerate development cycles for teams willing to adapt their processes.
The automatic security scanning feature is particularly valuable, as it helps address one of the main concerns with AI-generated code: security and best practices. By actively scanning its own output, Amazon Q Developer demonstrates a commitment to both speed and quality.
That said, be prepared for the expected preview-stage limitations. Beyond the context window restriction, you might encounter occasional bugs or inconsistent results. The service may also experience capacity constraints since it's currently free to use—AWS might be throttling resources during peak times, leading to slower responses or temporary unavailability.
I Strongly Recommend Trying It Today
I encourage you to experiment with Amazon Q Developer in your GitHub repositories. It costs nothing during the preview period and requires no AWS account setup, making it the perfect time to explore its capabilities and limitations.
I'll be continuing my experiment by testing how well Amazon Q Developer handles iterations and revisions to the generated code. Watch for a follow-up post with more detailed results and practical tips for maximizing this tool's value for your team.
