Design Evaluation: What to do when your product might not what your user need

We all know the drill, to build good products, iterations of research, design, development are needed. However, in the real world, that might not be the case. Sometimes, products are built based on team capability, instead of user need.

When the question is not about did you design the product right, but whether you design the right product instead; design evaluation might be the right tools to help you answer the question.

In early 2020, I revamped a product that can generate patterns using machine learning. The technology was ground-breaking, but no one used the product. More recently, I evaluated a new tool that aims to automate campaign websites for marketing teams. The problem is, when the tool that is supposed to help the team launches, no one wants to use it. The product teams go extra miles to demo it, but nothing changed.

Both scenarios happen in two different environments, but they both hold the same premises and similar background. Both created without proper research and limited understanding of the user. The first product aims to create a tenun pattern for traditional tenun weaver in rural areas with limited internet. While the technology was ground-breaking, it takes hours of stable internet connection to generate only one pattern. In short, the app won’t work on site.

While for the other product, after a series of evaluation, we found that to increase marketing campaign success, what needed is not an automation tool, but a support consultation related to SEO, customer experience and data tracking instead.

When faced with a product that might not be what your user needs; or might not solve your user’s problem, design evaluation helps designers identify what needs to be changed or improved.

Pre and Post Evaluation

There are so many methods to conduct a design evaluation, from heuristic evaluation to usability testing. However, if you need to identify user motivation while assessing the product, pre and post-evaluation methods come in handy. Pre-evaluation conducted to users who haven’t tried the product yet, best to determine the expectation of the products. Meanwhile, post evaluation aims user who already use the product, best to measure performance and satisfaction goals.

Setting up the goal

Before conducting an evaluation, you should always begin with the high-level question: what are we aiming to learn through this evaluation? Once the objective is set, you can start creating the goals.

There are two main goals of design evaluation, user goal and the evaluation goal. User goals are the goals that need to be achieved by the user during design evaluation, i.e. finishing certain tasks. Meanwhile, the evaluation goal is what you aim for the product based on the evaluation result, for example, product improvement or a go/no go decision.

Setting up the task

When conducting pre and post evaluation, you will gather information related to user motivation while assessing the product. So, you will gather general information such as the current process as well as pain points, and also product feedback. Since you will have two types of user, you might need two scenarios as well. While post-evaluation might give elaborated feedbacks related to the product, pre-evaluation feedbacks might not be as elaborated since the user has not used the product yet. However, you might identify user expectations and determine if the design meets the expectation through pre-evaluation. You will also need an additional step of product walk-through for pre-evaluation.

Setting up the scenarios

Since you will test the product to two types of user, you need to create two sets of scenarios. However, to improve the feedback and increase time efficiency, you can also provide surveys/questionnaires for post evaluation regarding their experience while using the product so that you can create follow up questions based on their responses.

Set up the tone

Since the goal is to gather feedback and expectation instead of assessing how users interact with the product, we can treat the user as a peer. That way, the evaluation can be more flexible. And if you are about to do design evaluation without prior experience to UX research, putting a user as a peer will help you to immediately point out flaws in the research design so that you can reshape your strategic approach.

Important note

Since you might end up revamping the product, or else, sunsetted the product altogether, it is important to involve the product owner or stakeholders early on. Make sure that you provide ample time as well when conducting the evaluation. If you have to evaluate under a short period, sending a questionnaire beforehand will provide context for the user, thus saving you time.

Takeaway

When faced with a product that might not solve users problem, design evaluation helps designers identify what needs to be changed or improved. There are so many methods to conduct a design evaluation. However, if you need to identify user motivation while assessing the product, pre-evaluation and post-evaluation methods come in handy.

It will help you determine the expectation of the products while measuring performance and satisfaction goals. Make sure to provide ample time when conducting the evaluation, and involve the stakeholder early on to minimise rejection.


Design Evaluation: What to do when your product might not what your user need was originally published in UX Planet on Medium, where people are continuing the conversation by highlighting and responding to this story.

Leave a Reply

Your email address will not be published. Required fields are marked *