Design Sprint — Define Phase
Combining Large-Scale Attitudinal and Behavioral Data to Complement, Existing User Experience Research Methods
Using Google’s HEART Framework to Identify Goals, Signals, and Metrics
The focus of this article is applying the HEART framework and the Goals-Signals-Metrics process not to convince you of it’s validity. However, I encourage you to read the original paper titled: Measuring the User Experience on a Large Scale: User-Centered Metrics for Web Applications
For those that want TLDR version of the paper I’ll summarize it below.
Most organizations rely heavily on PULSE metrics — an acronym that stands for Page Views, Uptime, Latency, Seven-Day Active Users, and Earnings — to determine a product’s health. Although these metrics are viable they are indirect metrics of user experience and lead to ambiguous interpretation of causality. Therefore they should be triangulated with other sources such as usability studies, diary studies, field studies, task completion rate, time-on-task etc. However, these techniques to garner behavioral data are limited to small amount of participants in controlled environments.
Many organizations attempt to address limitations in PULSE data, and behavioral data with large scale attitudinal data such as surveys and A/B tests and fill in the gaps with web analytics software like Google Analytics. Still it can be difficult to utilize this combination effectively. The sheer amount of data is likely overwhelming your team and it is hard to find insight in the vast amount of information available.
To address the aforementioned shortcomings Kerry Rodden, Hilary Hutchinson, and Xin Fu spent several years at Google working on the problem to create large-scale user-centered product metrics. Their efforts let them to the development of the HEART framework and the Goals-Signals-Metrics process. They applied the framework and process to over 20 different products within Google both consumer and business focused to “fine tune” it and concluded that it is “extremely helpful for focusing discussions with teams.”
HEART Metrics
HEART was designed to address the shortcomings in PULSE across measuring user experience quality and providing actionable data. Like PULSE it is also an acronym and stands for Happiness, Engagement, Adoption, Retention, and Task success.
Happiness measures the subjective aspects of the user experience including visual appeal, satisfaction, perceived ease of use and likelihood to recommend.
Engagement is the user’s level of involvement as measured by behavioral proxies such as depth of an interaction over some period of time.
Adoption metrics track how many new users start using a product during a given time period.
Retention track how many of the users from a given time period are still present in some later time period.
Task Success encompasses several traditional behavioral metrics of user experience, such as efficiency (e.g. time to complete a task), effectiveness (e.g. percent of tasks completed), and error rate.
Goals — Signals — Metrics
Is a simple process that steps teams through articulating the goals of a product or feature, then identify signals that indicate success, and finally builds specific metrics to track on a dashboard.
- Goals state the outcome the user is trying to achieve
- Signals are the manifestation of goals in term of user behavior and or attitudes
- Metrics translate signals into measurements that are suitable for tracking something over time on a dashboard
Real World Application
After completing the Understand phase of your Design Sprint including framing the problem, iterating on How Might We statements, completing a competitive analysis with Rose, Thorn, it is time to define your products success metrics using the HEART framework and the Goals, Signals, Metrics process.
To get started have the Design Sprint team create table with each category of the HEART framework in the first column, and the Goals, Signals, and Metrics for each category in the subsequent columns. Next, add a Goal, Signal, and Metric for each category of the HEART framework. An example for a photo editing application is below:
+--------------+---------------------------+-------------------+--------------------------------------------------------------------------+
| - | Goal | Signal | Metric |
+--------------+---------------------------+-------------------+--------------------------------------------------------------------------+
| Happiness | Share with friends | Refer a friend | Referral sign up rate |
| Engagement | Save photos | Upload photo | AVG # of photos per day per user |
| Adoption | Register for a free trial | Create an Account | Count of Free Trials |
| Retention | Use application | Activate account | % of Free Trials that convert to paid |
| Task Success | Improve photo quality | Apply filter | AVG # of uploaded photos with filter/AVG # of uploaded photos w/o filter |
+--------------+---------------------------+-------------------+--------------------------------------------------------------------------+Helpful Tips
- While writing goals don’t worry about the feasibility of finding relevant signals or metrics. Focus on the user experience and the tasks the users need to accomplish to see value in your product or feature
- Choose signals that are sensitive and specific to goals not for unrelated reasons
- Raw counts go up as user base grows ensure you normalized data. Ratios, percentages, or averages will be more useful than counts for metrics
After the team has aligned on the table’s contents save the document as an artifact for future use.
If you are excited about how you will use this framework please clap this article. To clap for a story or list, click the clap button on the post page. Pro-tip: You can clap up to 50 times per post by holding down the clap button.
