I have been sharing about the problem I'm working to solve around AI Prompt optimisation online on IndieHacker, Product Hunt, X and discord/Slack groups.
First what am I building?
building Promptperf.dev this is a platform that will automate promt testing. Why is this important. As more and more AI models are available its hard to find out which model will provide the best performance and consistent output. My tool will allow users run automated tests against multiple models and configs and most importants on multiple runs. How is this useful? Take this as an example: user wants to test 4 AI models at temperatures 0.1, 0.4, 0.7 and 1.0 and has 3 prompts they want to test. To ensure consistency each config to be ran 10 times to ensure its not hallucinated. (4 Models x 4 temp x 3 prompts x 10 runs = 480 API Calls and manually entering results into some database/excel to compare the results)
Why now? AI model providers are starting to deprecate models eg. Gemini 1.0 is being deprecated which means all the apps running on Gemini 1.0 will now need to test a new model and just doing a model swap to the newer model doesnt work. So prompt testing will be required to find the next AI model that will easily be swapped in its place.
Let me share which platform has helped me get the most traction:
Day 1 first shared this on IndieHackers and X got 2 signups and about 30 visitors,
Next day I looked for online communities on discord and Slack: Most of these communities had showcase channel where I can promote the product but others had introductions which I got warnings from moderator when I shared the product Im building. I also replied to quite alot of X tweets. I created a list of potential AI influencers and started to reply to their tweets and where I could I plugged in PromptPerf.dev but I also updated my bio to ensure it directly users to the product from the profile. This provided another 2 signups, unsure from where but website visitors were up to 70 at this point
Day 3: Focused entirely on X and replied back to multiple accounts not focusing on the product but just replying anything AI related which I assume made me gain trust in the AI audience and got quite alot of follows which could have directed some visitors to the product. Here I got to 10 subscribers who had opted in.
Now I had created tags on the signup form for users to select if they are interested in helping with feedback or get early access. Heres the breakdown:
5 for early adopters, 4 for early adopter and provide feedback, 1 for just feedback.
Do you think the problem I'm working to solve is real?
