Show & Tell
we spent a month trying different AI chatbot solutions (intercom, drift, custom gpt) and nothing worked well for our use case. the canned responses were too rigid and the LLM-based ones hallucinated our product features
ended up building one with tinyagent and heres what made it work:
- fed it our actual knowledge base docs (about 50 articles)
- gave it access to look up order status via an API tool
- set a system prompt that tells it to say "let me connect you with a human" if its not confident
- connected it to our support ticket table so conversations get logged
its handling about 40% of our support volume now and the handoff to humans is actually clean. the key was giving it real tools (order lookup, account status) not just text to parrot back
happy to share more details if anyones interested
ok THIS is what i needed to understand about agents. the tools thing is key. its not just a chatbot its actually doing stuff (looking up orders etc). now i get it
really nice writeup. couple questions:
- how do you handle when the agent gives a wrong answer? is there a feedback loop?
- whats the average response time compared to a human?
- how much does this cost to run monthly with the LLM calls?
good questions:
- wrong answers: we log all conversations to a table. someone on our team reviews the agent conversations weekly and if we spot a bad answer we add it to the knowledge base as a correction. over time it gets better
- response time: instant. like 2-3 seconds including the order lookup. humans average 15 min during business hours and longer after
- cost: hard to give exact numbers but roughly 30-40 dollars a month in LLM credits for about 800 conversations. way cheaper than another support hire
we built something similar for our SaaS. one thing that helped with hallucination is being very explicit in the system prompt about what the agent CANNOT do. like "you cannot process refunds, issue credits, or change account settings. if asked, say you will create a ticket for the team"
also restricting the knowledge base to only your own docs (not general web knowledge) keeps it grounded
Community