Recruiting participants for remote user tests

I recently attended my first remote design sprint at Automattic. It ran over a two week period and  was attended by fellow designers, developers, data scientists, happiness engineers, and marketers. Together we created two prototypes and tested them out with real customers. I found the experience to be both rewarding and inspiring. We came up with a lot of great ideas and validated them too. It was also really nice collaborating with a number of people across the company that I don’t usually get to work with.

One of my major contributions to the sprint was recruiting real customers for the prototype test. The job was a whole lot easier thanks to an internal post written by my designer colleague Mike Shelton which also took part in the design sprint. He shared his experience recruiting customers from a previous test and also included the templates he created for people like myself to use. With that post, I had everything I needed once we determined exactly which type of customers we wanted to test.

Finding participants

Before the sprint even started, I had been running tests to see how many people I could recruit with a poll on our signup form at WordPress.com. Fortunately my tests worked out well so I knew I could count on running another one to collect participants for our test. I created a new poll using HotJar and added a little prompt at the end of the poll that asked participants to sign up for other customer research initiatives. This is what it looked like:

poll.png

Screening participants

Since we wanted to talk to a specific group of people, the link at the end of the poll lead to a Google form which asked visitors to complete a series of questions. We added questions that would help us qualify the people signing up for this test and other tests in the future. Since this was a remote prototype test, one of the key questions was if people were comfortable talking over a video conference.

screener-shot.png
View all the questions

Sending out invitations

I wanted the email to look like it was coming from a real person so I sent it with Inbox using my full name and company email address a week before we have our test date scheduled. It worked because some people replied back which was great — I was able to do some additional research for another project. I also made sure to add the qualified email addresses to the bcc field so that the recipients couldn’t see any other people being emailed.

These were the key points I included in the email:

  • Introduce myself and tell people where I got their email
  • Explain what we’re testing along with all the logistics
  • Identified what the recipient gets for participating — we decided to offer a $50 Amazon gift card because we felt like we were asking a lot for someone to take an hour out of their busy schedules
  • How to sign up for the test.

email.png

Coordinating schedules

Getting people across multiple time zones to agree on a date and time over email can be confusing and time consuming. There’s lots of back and forth involved and it’s easy to make mistakes.

That’s why we use Calendly to manage our calendar availability and bookings. It’s easy to setup, offers a number of customizable options, and displays times relative to the timezone of the viewer . You can also automate and personalize event reminders so that you have one less thing to worry about.

calendly.png

Sending a reminder

People’s lives are busy so it can be understandable if someone forgets something they committed to a week ago. Sending a reminder a couple hours before the test is a good way to avoid that and to add any last bits of logistical information required for the test. In my case, I made sure to include:

  • the time,
  • a link to download the software used for video conferencing — we use Zoom.us for it’s video quality, number of participants, screen sharing, and recording capabilities, and
  • a link to join your video conference.

Here’s what my email looked like:

reminder.png

Following up after the test

Lastly, we ended things off with one last email thanking our participants for their time. We included instructions for them to claim their $50 Amazon gift card and also took the opportunity to collect some final feedback about the test so we could make improvements for future participants.

follow-up.png

I hope you enjoyed reading this post and remember it next time you’re planning on recruiting participants for your own user test or interview. If you do, let me know how it works out for you by reaching out on Twitter.

This post also appeared in Automattic’s product design blog.

Creating empathy for your customers

In my last post, I wrote how we use interviews to learn about our customers. They allow us to ask them questions and in their responses we find insights and get inspired. In this post, I’d like to share some other ways we build empathy for the people using our products.

Support Rotation 💬

It starts from our first day on the job. Everyone that joins Automattic spends three weeks doing what’s called a support rotation. We learn about our products by reading support documentation and running through exercises. Once we’ve completed our training, we get to business and start offering support to our customers with our Happiness team. We start by answering inquiries by email and then upgrade to doing live chat.

After we complete our rotation we are released into our new roles with a wealth of knowledge about our company and our customers. That’s not the end though, once a year we return back to the Happiness for a week to offer more customer support. It’s a great way for us to keep in touch with the challenges our customers are facing.

Business concierge 💼

When people sign up for the WordPress.com Business plan they get access to a one on one setup appointment from a member of our Happiness team. The sessions are conducted over live chat or screen share and usually last around 30 minutes. The goal for these sessions is for the user to walk away feeling comfortable and confident building their site on WordPress.com.

In addition to helping our customers, we use these sessions to learn. Anyone can join in real time or listen to a recording at a time that suits them. There is no limit to how many sessions you can attend. After each session, the recording and notes are tagged and posted. Our support team uses these posts to generate monthly reports that give us great insights into who our customers are and the challenges they face. For someone like me that works on our signup and onboarding experiences the content in these reports is invaluable.

User research campfires 🔥

Every month members from across Automattic gather to share their stories about the research they’re conducting. We do presentations, have discussions, and talk about our processes. Through these sessions we hear about our customers and what we’re doing to overcome their challenges. It’s a great place to learn from each other and grow our skills a team. The meetups are recorded for people that can’t attend and serve as a reference to new people joining our team.

Web chefs 🍜

We are often encouraged to build sites for people we know or encounter. There is no better way to experience your product than using it in real world situations. With limited time, resources, and budgets you get a first hand encounter of the challenges people face using your product.

Out of all the activities I’ve written in this post, this is my favorite way of getting close to our customers and building empathy for them. The insights I’ve gained have helped me make better decisions in my design process and allow me to offer great feedback to my peers.

I’m always interested to learn how people conduct their research. What have you found to be an effective way of learning from your customers? Let me know in the comments below.

This post also appeared in Automattic’s product design blog.

Designing with words

Reading Storytelling for User Experience by Whitney Quesenbery and Kevin Brooks got me thinking about different ways of exploring ideas in my design process. It inspired me to try something new on a recent project. I found that both the end result and the journey were very rewarding.

Starting with words

For most projects I start with sketches, flow diagrams, or wireframes. The end result is always some sort of visual output. I have my graphic design background to thank for that. With this project, I changed things up a little and described my user journey in writing rather than illustrating it. The end result was like a story about our customer’s journey through our signup process.

I broke each step into a series of bullets that were short and concise. It was important for me to make them easy to read. My self imposed constraint forced me to keep things at a high level. This made it better for collecting feedback and moving onto the next stage of the project because people didn’t get caught up in small details.

Words are cheap

I don’t even need to think when I type because of how long I’ve been using a computer. It’s like I look at a screen and words appear right out my brain. Working on this project in this way made me realize that I can type quicker than I can sketch. I was able to cover lots of ground in less time and generated more ideas than I normally would.

I worked in waves by documenting my thoughts and then editing them. The ideas got better with each edit and sometimes they’d even spark new ones. I had to be diligent to capture them before they escaped. My keyboard skills came in hand as I copy and pasted new phrases, finished them off, and then got back to the original thought. After a while, the content began to mature. and I started to think about the implementation.

Structure brings Freedom

I remember my visual brain going wild while writing these flows. The descriptions painted such vivid pictures. I couldn’t wait to get to the visual part of the project. Once I did, I found it really easy to get going. Working within the constraints of the written text was very fun for me. With the high level details all figured out, I was able to focus on the details and get creative.

Conclusion

With this experience behind me, I can easily say I’d do it again. Heck, I’d like to see if there are other ways I can incorporate words into my design process. I’m going to continue exploring and writing about it here so check back soon. If you have any suggestions let me know, I’d love to hear about them.

Observing your customers in the wild

I’ve been trying a new ethnographic research technique called Sales Safari. It was developed by cross functional designers Amy Hoy and Alex Hillman. It unfortunately isn’t very well documented so I decided to write about it after reading how to do it in a book called  “Designing Products People Love” by Scott Hurff.

The goal of this approach is to create great products by studying what your customers are saying. Insights come from analyzing large quantities of data collected from online public spaces like forums and social networks. I thought it might work well to craft a great signup experience for our customers on the new project I’m working on.

How I ran my Sales Safari

I started by identifying the places where our customers hang out. Facebook, Twitter, LinkedIn, Google Plus, Reddit and the WP.com forums where a good places to start. Later, I also reviewed some of our customer’s sites, survey responses, and support requests. I closely read through the conversations while visiting each location. Phrases that stuck out to me were copied and pasted in a spreadsheet. I created a single keyword to summarize the phrase so I could categorize them later. Here’s how Hillman describes what you should be looking for:

“You start collecting jargon, some of their specific detailed language and words they use to describe the problem […] Elements and contributions to their worldview, their deep-seated beliefs that are unshakable. Then also the things that they talk about, they recommend. The things that they buy.”

As I progressed, I felt like this was a never ending process. There is so much content out there. Each conversation lead to another and sometimes even to a whole new “watering hole” — a term Amy uses to describe places where people go to share their views. I eventually decided I had enough data to take a peek. As I started grouping the phrases by keyword, the patterns started to become clear right away.

What I learned

After going through this process I definitely feel like I know more about our customers than when we started. This kind of research takes a lot of time and isn’t ever really over. I intend to continue doing it on a regular basis to stay in touch with what our customers are saying and feeling. This new understanding of our customers has both empowered and inspired me to explore different ways we can show people the value of our product early on so we they can stick around with us for longer.

Share your thoughts

Have you tried this approach before? Do you have other ways you like to learn from your customers? Reach out on Twitter and let me know.

Product and marketing: two sides of the same coin

Colin Bentley and Brian Donohue are product managers at Intercom — a product first company. They were interviewed on the company’s podcast about their experiences since they joined the company of 30. Today, there are 300 people working at Intercom and their product services over 100,000 active customers.

As a product first company it was normal for them to think if you built a great product, everything else would follow. People were cynical when Intercom started making their first marketing hires.

Over time, the product folks began to see they weren’t very different than the marketers. They realized they both focus on the needs of their customers. While the product managers build features, the marketers position them to their customer’s so they’re desirable and useful. As an example, Twitter’s tweet button could have been easily labeled submit. Instead, a lot of work went into positioning it as a Tweet button so that Tweeting could become a thing.

Today, Colin and Brian explain how important it is for their team to collaborate. It’s essential for them to fuse both the marketing and product perspectives. This allows them not only to build the most important things for their customers but also gives their product a unique place in the market.

Listen to the full podcast on Intercom’s site.

Apprehensive feet waiting to take their first steps

How to get started on your next design project

Getting started on a new project can be intimidating. Thankfully as designers we have these tools to help us get the ol’ juices flowing.

User flows

Experience your product as the people using it would. Take screenshots, write notes, and press every single button you find. Document your journey in steps as you go and take notes.

Stakeholder interviews

Learn from your peers and customers. Find anyone that’s worked on your project before you, speak with your peers, and interview the decision makers on the project. Most importantly, get in touch with the people using your final product.

Competitor analysis

See how your product measures up and find some inspiration along the way. Start by identifying your direct competitors. Then expand search to similar products. Finally look for companies in completely different fields. Make a list of the defining features of your product. As you review each competitor: document the features they have, take notes, and add new features to your list that you find interesting.

Sales safari

Gain a unique perspective about your customer. Find people using your product without influencing them in any way. The best way to do this is to look up things they’ve said in the past by checking your forums, customer support requests, old surveys, and social media accounts. Look out for how they speak, what they think about your product, and what challenges they’re running into.

Inspiration boards

Capture your favourite pieces to inspire you throughout your project. Visit your favourite inspiration sites or use screenshots from your competitor analysis. Save images and arrange them in a way that makes sense to you. Make it really visual and physical. Print it out if you have to and stick it somewhere where you can see it from time to time.

Data analysis

Find out how your product is performing. Get familiar with tools like Google Analytics, mySQL data bases, and what ever reporting software your company uses. Understand your business performance metrics so you can figure out how you can make a real difference both for your customers and your business.

Communication

Keep everyone up to date with your progress. Share your findings as frequently as you can to spur discussion and gather ideas. It easier to get buy in when people are involved in the process. Get people excited by posting your diagrams, notes, and sketches for everyone to see.

Conclusion

What does your design process look like? Is there anything on my list that’s missing? Want to learn more about one of these in more depth. Reach out on Twitter and let me know.

The key to success for your product

I just listened to Scott Bellsky explain the importance of a product’s first mile on Intercom’s podcast. The first mile comes right after someone signs up for your product. It is often referred to as an onboarding experience. This is where you orient your users within your product and give them a sense of what they’re going to do next. Careful attention is paid to the type of copy you write, how your tour looks, and what defaults you set for your users.

According to Scott, the key is to have someone do a few key things that shows them the value of your product over the long term.

Here are some of the notes I took from the podcast:

  • The first mile is usually the last mile a team focuses on. Teams often spend lots of time building their products and only think about the onboarding experience as an after thought or when they see their users aren’t sticking around.
  • Scott believes that in the first 15-30 seconds of every new customers experience, they are lazy, vein, and selfish. Lazy because they don’t want to read or do any work. Vein because they want to know how the product will make them better. Selfish because they want the product to make them look good. These are important characteristics to keep in mind when crafting this experience.
  • He suggests that maybe we shouldn’t tell them or show them, but rather we should just do it for them with the defaults completed.
  • It’s important to remember we can’t believe people will have a relationship with us if they can’t get through those first interactions. They can’t buy into our vision or to recommend us to other people. It just needs to serve them now!
  • The first mile is successful when users reach the “zone”. When they experience that sense of accomplishment and see how the product will serve them in the future.
  • Messaging apps like Telegram and Slack have really smooth onboarding. They even think about the small things like the magic link to log you in because when users come back the second time, they don’t usually remember their passwords.
  • The first mile isn’t something that can be done once and forgotten about. It’s important to continually optimize the first mile by understanding your users and being empathetic to their needs and circumstances.

You can listen to the full podcast on Intercom’s blog or your can learn more about the first mile by reading Scott’s blog post on Medium.