AI Chat Experiences – Empowering students with high quality responses by refining AI classifier precision

📊  Real-time data
✏️  User testing
🧠  AI
🌎  Web

MY ROLE

As a Senior Product Designer at Chegg, an educational platform renowned for providing step-by-step solutions that help students learn and solve their homework questions, I led a strategic initiative to enhance our AI classifier's accuracy. I facilitated cross-functional workshops, aligning diverse teams on the importance of user-centric design while translating business objectives into actionable user needs. By conducting thorough user testing, I identified key preferences and successfully balanced them with business requirements. My efforts culminated in a 1.36% improvement in AI accuracy and a 6% boost in user engagement.

TEAM

1 Product Manager, 1 Front-End Engineer, 4 Analytics, and 2 ML Engineers

PROJECT

Implemented in the Chegg website for both desktop and mobile web

TIMELINE

Apr 2024 – Jun 2024

Key point to watch

You can see my initiative in solving problems with a minor design change that stakeholders thought required engineering efforts

Problem Statement

How might we better understand whether a student is asking a new question or follow-ups to provide high quality responses?

Success Metrics – AI classifier’s accuracy
Chegg is an educational platform where students can get both expert-verified solutions and instant solutions generated by our AI. To deliver the best quality answers, it’s important to understand students’ intent on their question: whether they’re asking a new homework question or follow-up questions.

Approach

Utilized user testing and led workshops to define user benefits instead of solely focusing on business goals.

As the lead product designer on this project, my biggest goal is to improve intent classifier, which is our business goals, without harming user experience. To do so, led brainstorming sessions with stakeholders to get diverse perspectives and verify multiple solutions through user research focused on its usability.

Achievement

Enhanced the AI accuracy by 1.36%Increased user engagement with the button by 6%

It achieved statistically significant improvement in our success metrics (AI classifier accuracy) by changing the visual treatment of the ‘New Question’ button on the left navigation. This nudged users to start a new conversation for new questions, reducing misclassification and providing explicit signals to train our classifier model and lower error rates.

Background

Chegg offers an educational AI chat – students can ask homework questions, get solutions, and ask follow-up questions.

For our AI to provide the best response, it's crucial to understand and classify if students are asking a new question or a follow-up.

For example, when a student asked a homework question, but we misclassified as a follow-up, then it will not go through the search and automatically directed to the AI to generate the answer. This means we cannot provide the expert-verified solution through search, which is one of our biggest value. On the other hand, when a studnet asked a follow-up, but we misclassified as a homework, then we’re not sending the whole context, so, it will provide the low quality response.

Problem

But, our AI classifier – which classify students’ intent on question – was not accurate enough.

Workshop

To solve technical limitations through design, I first needed to fully understand the problem, so I led a workshop with ML engineers. As a result, I’ve got more user-centered goal.

Design Explorations

Based on this goal and ideation from the workshop, I explored six concepts and compared trade-offs, and prioritized top three ideas.

Select intent before submitting question
Ask feedback on AI classifier
Click CTA for ask a follow-up
Make 'New question' CTA prominent
Provide a reminder to start a new conversation
PROS
CONS
OPTION A – Select intent before submitting question
·
Use this explicit signal to train the AI classification model
·
Reduce the immediate costs and erros of the AI classifier
·
It adds one more step for users to ask questions
·
If user selects the wrong one, it will harm the AI classification model
OPTION B – Ask feedback on AI classifier
·
Use this explicit signal to train the AI classification model
·
Cannot reduce the immediate costs and errors of the AI classifier
·
Place the burden on users
OPTION C – Click CTA for asking a follow-up question
·
Use this explicit signal to train the AI classification model
·
Reduce the immediate costs and erros of the AI classifier
·
It adds one more step for users to ask questions
OPTION D – Make CTA more prominent
·
Use this explicit signal to train the AI classification model
·
Reduce the immediate costs and erros of the AI classifier
·
Help users effectively manage their chat history
·
Easily discover the button to start a new question
·
It might distract users from the main content
OPTION E – Provide a reminder to start a new conversation
·
Use this explicit signal to train the AI classification model
·
Help users effectively manager their chat history
·
Cannot reduce the immediate costs and errors of the AI classifier

User Testing

I conducted a concept testing with 18 users to understand students’ reaction towards 3 concepts. And I selected the most promising design (D) based on the 4 decision matrix I defined.

Final Design

Improved presence of the 'New question' button to nudge users naturally, eliminating the need for classification, while training the classifier

Impact

Enhanced the AI accuracy by 1.36%Increased user engagement with the button by 6%

Next Project →