Welcome back to our Kotae redesign series. This is a three-part series that outlines how we went about putting together a better, more user-friendly product that ultimately serves our clients and customers by focusing on their pain points and needs.

In Part 1, we explored how our user research uncovered major usability issues, from difficulties in onboarding to friction in core tasks like creating Custom Responses. Part 2 then went into how we came up with design solutions: a cleaner dashboard, a guided setup, and an improved task flow to help users get value from Kotae faster.

In Part 3, we’ll be covering how we validated our new design to ensure that the new version is truly an improvement over the old, and that the user experience lives up to the standards we aspire to. We put this new version to the test through a second round of usability testing, measuring it against our earlier benchmark, and evaluating whether our changes actually improved the experience.

Our Process

To understand whether the new design actually improved the user experience, we conducted a focused usability study. We asked a group of internal participants to role-play as different types of Kotae users, roles that were based on our three personas. We had them role-play as new users to see whether or not they could intuitively navigate our interface and complete their assigned tasks with minimal difficulty. Each participant interacted with a fully functional prototype in a controlled test environment. This ensured consistency across sessions while still allowing us to observe natural behavior.

The two primary tasks mirrored the real-world goals we had redesigned around. First, users were asked to go through the onboarding process, starting from the marketing landing page and progressing all the way through setup. Then, they were asked to create a Custom Response using the new dashboard tools. These two flows had previously been identified as high-friction areas, and were the same two flows we tested in our benchmark study, so they served as ideal test cases for measuring whether our interventions had the intended effect.

Alongside behavioral observations, we recorded task outcomes and timing, and asked participants to rate their experience using the System Usability Scale (SUS) — a standardized and widely used measure of perceived usability. We also collected open-ended feedback at each stage to capture thoughts, hesitations, and suggestions in the users’ own words.

Comparing Against the Benchmark

The three variables from our initial usability test were tested again to measure the impact of the redesign: task completion rate, time to complete each task, and SUS score. Together, these provided a mix of outcome-focused and perception-based data, allowing us to evaluate not only whether users were able to complete their goals, but how efficient and satisfying the process felt along the way.

Task completion rate was our basic indicator of effectiveness — a simple measure of whether participants could achieve what they set out to do without getting stuck or giving up. Time-to-complete gave us a concrete way to assess efficiency, especially compared to the longer task durations we saw in earlier rounds of testing. The SUS score added a subjective layer: after finishing both tasks, participants answered a standardized set of questions about how intuitive, easy, and enjoyable the system felt overall.

We also noted which parts of the UI users gravitated toward naturally, which elements they hesitated on, and which flows required clarification. This mix of qualitative and quantitative insight gave us a well-rounded view of how the redesign held up, as well as what still needed refinement.

Measuring the Results: Before and After

The redesigned experience delivered measurable improvements across key metrics.

‍

‍

In addition to the numbers we measured, users offered qualitative feedback that reinforced the value of the redesign. All reported better impressions of dashboard navigation and onboarding intuitiveness than the first round of testers did. Creating a Custom Response, which had previously caused confusion, now felt “straightforward” and “well-guided.”

What We’re Improving Next

The validation confirmed that we’re heading in the right direction. Task flows are faster, satisfaction is higher, and users are less likely to get stuck. Still, the study revealed several areas where we’ve made further improvements:

• Clarity around system status — A few users weren’t sure if the bot would continue actively training if they were to navigate to another page. We are now adding clearer feedback indicators.
  
  
• Terminology refinement — Some Japanese users experienced uncertainty about the meaning of certain features. We’re addressing these issues by improving the UX writing in all languages we offer.
  
  
• Minor UX issues — We also found a few smaller points of friction that we’re addressing. Things like default field behaviors and other interaction details.

Image: Annotated UI highlighting system feedback and terminology flags

Looking Ahead

The redesigned experience is already live, and we’re continuing to monitor real user behavior to inform the next round of refinements. We’re also constantly speaking with our users, and regularly share real user quotes with our product, design and development teams, as users engage with the updated platform.

The Kotae redesign was a multi-stage effort grounded in user research, from the initial discovery of friction points, to the design responses we introduced, to the validation work covered in this final post. At each step, we were guided by real data and direct user feedback rather than assumptions or intuition. The end result wasn’t just a cleaner interface, but a product that better reflects how people actually work with it.

One of the biggest lessons from this process is that teams can never fully predict how users will behave just by “putting themselves in the user’s shoes.” Even the most experienced designers and developers bring their own biases. And those blind spots can only be uncovered through structured user research. We saw firsthand how a feature we thought was clear still caused confusion, or how a redesigned flow worked only when the user followed it exactly. Those insights came not from guesswork, but from observation.

Research-driven design not only helps identify what’s broken, but it also sharpens product thinking, builds team alignment, and increases confidence in the launched product. It’s the difference between a product that simply looks good and one that looks good while also being user-friendly to its intended audience.

At Tokyo Techies, user research and usability testing are embedded into every design engagement we take on. Whether you’re building a new product or rethinking an existing one, we can help you uncover the real needs of your users, and come up with validated design solutions.

Thanks for following along with the Kotae redesign series! Check back regularly for future design case studies, or read more below.

‍