GenAI Pipeline · 50reviews evaluated · 6 prompt versions tested
ChicStyle, a women's fashion retailer, needed a way to triage large spikes of customer reviews during peak sales periods without manually reading every one.
accuracy predicting recommend / not-recommend, zero labeled training data
of reviews are 4-5 stars — the positivity bias every prompt had to account for
Three prompting strategies, each with a basic (V1) and enhanced (V2) version, scored by an LLM-as-judge on sentiment accuracy, category correctness, and message quality.
Adding business context and a fixed category list gave a small, genuine improvement. The best-scoring version overall.
Adding explicit rules for a 'Mixed' sentiment label backfired — the model over-applied it to 60% of reviews.
Inherited the same 'Mixed' rule and the same over-classification problem, worsened by step-by-step reasoning that amplified it.
“Simpler prompts outperformed more complex ones — the most counterintuitive finding of the project. Zero-Shot V2 beat both Few-Shot V2 and Chain-of-Thought V2.”
“Overly prescriptive rule definitions can constrain the model into rigid classifications that reduce nuance rather than improve it.”
The configuration recommended for production, based on the results above.