What building RetainIQ taught me

RetainIQ won the Innovation category at Amazon AIdeaS 2026. I already wrote the architecture breakdown: the stack, the scoring model, what runs on Amazon Bedrock. That post answers what RetainIQ is. This one is about what building it actually taught me, and none of the three lessons were about the model.

Lesson 1: the hard part is deciding what not to surface

Amazon Nova Pro is good at extracting knowledge entities from raw signals: who owns what, which systems depend on which, where the single points of failure sit. That part worked early and it worked well. The first version of the product alerted on everything it found. Every topic with any concentration risk got a flag. It felt thorough. It was actually noise, and noise trains people to stop looking. The rewrite only fires an intervention when three conditions are true at once: the concentration is high enough, the topic is genuinely business-critical, and recent activity says the risk is live right now. Almost every change between v1 and v2 was subtractive. I spent more time deciding what to cut than what to add, and that is the part that made the product usable instead of just impressive in a demo. If you are building anything on top of an LLM, the model is rarely the bottleneck. The judgment about what the model's output is allowed to reach a human is.

Lesson 2: privacy is architecture, not a slide

Early feedback on the first version, when it was still called OMDA, included a comment that stuck with me: I had talked about privacy without building it. That was fair. "We take data privacy seriously" was a line in my pitch deck, not a property of the system. So it became a first-class part of the architecture instead of a claim about it. PII redaction runs before any signal touches a model, and it is a visible admin toggle, not a background promise. The option to delete raw content once analysis is done is the same: visible, per-customer, on by default in the settings a user actually sees. The lesson generalizes past this one product. If privacy only shows up in your marketing copy, it is not a feature yet. It becomes one the moment a user can see it, control it, and verify it did what it said.

Lesson 3: critical judges beat friendly feedback

The version that made the semifinals of AIdeaS was rejected with blunt, specific feedback. It stung, and it was the most useful product review I have had. Two comments in particular rebuilt the product: that it sounded like another knowledge-management tool, and that privacy was a claim, not a feature. My first instinct with both was to explain why the judges were wrong. I did not act on either comment until I stopped defending the pitch and actually treated the criticism as data. Once I did, the rewrite followed directly: reposition around measuring knowledge health instead of storing knowledge, and make privacy something the product does instead of something the deck says. Fifty strangers giving a pitch a hard, honest look taught me more in one review cycle than six months of supportive feedback from people who wanted me to succeed. Encouragement feels better. Criticism moves the product further. I would rather have the second one earlier next time.

Where this leaves the product

RetainIQ is live at retainiq.cloud, and the full architecture is in the previous post if you want the how. This post is the why behind the decisions that shaped it: cut ruthlessly, make privacy visible, and treat harsh feedback as the fastest path to a better product. None of that is specific to AI products. It just happened to be the thing that decided whether an AI product actually held up outside a demo.