Build, measure, learn for the win

One of the first projects I worked on at Automattic was a small A/B test for the first screen of the WordPress.com sign-up process. As the first screen in our funnel, it served a critical role, but it hadn’t been tested in quite some time. I tweaked the visual design and completely changed the copy for my first attempt at improving conversions on that screen. Setting it up and tracking the results felt really easy compared anything I had done before. I felt empowered and simultaneously discovered how projects work at Automattic.

That first test was a success and was implemented as the default for all our visitors. It remained that way for a little under a year as our priorities shifted to higher impact areas of our funnel. Little bits of feedback trickled in over time that kept me thinking about many possible improvements but it wasn’t until recently that I decided to take action.

The problem

One of the most common complaints I heard about our screen was that the terminology was confusing. We asked people what type of site they wanted and presented them with four options: a blog, a website, a portfolio, and an online store. These were some of the questions that kept coming up: what’s the difference between a website and a blog? can I change my mind later? two of these options sound good, what if I want both?

Our mistake was assuming people understood what to us seemed like simple concepts but for them were actually foreign. This made a lot of sense when I look back at all the polls I ran and conversations I had with our customers over the last year. I learned that our customers are not like us and that the majority of people signing up at WordPress.com had little to no experience building sites or using WordPress.

screen-shot-2018-01-02-at-10-38-08-am.jpg
Graphic courtesy John Maeda and Rebrand Cities.

A new hope

Armed with this awareness about our customers, I reworked this screen to simulate a first meeting between a design agency and their client. The screen would ask them a series of questions to understand their needs, and then our tools would build a site that addressed their goals. My biggest concerns with this approach were 1) not scaring people with too many questions so early in the signup flow, and 2) knowing which questions to ask.

I quickly developed high level strategies to tackle both these problems. Designing with a mobile context in mind would ensure that I made it quick and easy for people to answer our questions and using a tool like Hotjar, I could quickly run a series of polls to validate and modify my questions.

Learning our customer’s vocabulary

One of my questions could be answered in a number of ways and therefore required an open text field to collect people’s responses. Of course, typing on a phone sucks, so I wanted to make the field as painless as possible. An autocomplete feature made the most sense as it offered people suggestions while they typed their answer. We could not only save people time from writing out the full word but we would also get cleaner data too. To pull this off, I needed a collection of commonly used terms to power the suggestions.

I created a new poll that included the question as it would appear in my design and quickly noticed the answers weren’t what I expected. Thanks to HotJar I was able to turn off the poll, rephrase the question, and launch the poll again within a matter of minutes — all without writing a single line of code. This time around I was happy with the data coming in and let the poll run until we had enough responses. After a couple days we had a lot of raw data to comb through so I used a spreadsheet and a PHP script to clean it up and help us make sense of it all.

php2.jpg

Build, measure, learn

Another question asked people to pick their answer(s) from a handful of choices. I drafted a list of options based on data we had collected in the past and ran them in a poll along with an “other” field that allowed people to type their answer if couldn’t find what they were looking for in my list.

The first iteration of this poll was dominated by responses entered in the “other” field. Some of them were variations of what I had in my list but there were also lots of common responses that were not represented in my list at all. I used this data to update my list and then published another version of the poll. There were some decent improvements in the results but it was still not quite there yet so I rinsed and repeated. This time I got it right because the majority of the responses were selected from my list rather than entered into the “other” field. In addition to that, most of the people that did fill out the “other” field either flipped me the bird or just said hi.

Progress on multiple fronts

I wasted no time while these polls were running and started building the form right after the design was sketched out. The code was built in small chunks which made it easy to review and kept the momentum of the project going with visible progress. As the data came in, I added it to my work in progress until I had enough of a prototype to run some usability tests. The tests revealed some really minor tweaks that needed to happen but mostly confirmed that we were on the right path. Then, shortly after I completed my last poll, I had everything I needed to run my A/B test at scale and see how this new design stacked up against the old one.

The results

It didn’t take long for us to reach statistical significance but I still let the test run for at least seven days to account for any seasonality during the week. The results indicated that this design profoundly influenced our business metrics. Like it says in NEA’s Future of Design report:

Ultimately everything a design team does has consequences (both good and bad) for a business. The more you know about the business and its goals/intentions, the better a design team is positioned to deliver good experiences and results for the business.

— Tim Riley: Senior Director of Digital Experience at Warby Parker

This post also appeared on Automattic’s Design blog.

Crossing the digital divide

Taxiing home after my first Automattic Grand Meetup in Whistler, I was happy to get to know my driver Mohamed. He has been running his family business for over 20 years and also drives a taxi to help make ends meet. Like me, he has three kids and we spoke about them at length. We also spoke about work and got excited when he heard that I’m designer at WordPress.com. He pulled out his phone right away and told me that he downloaded the app with the intention of starting a new site but didn’t have the time to get to it.

We’re often encouraged to work with people that use our products because it helps us see our product through their eyes. This seemed like the perfect opportunity so I offered to build Mohamed his site on WordPress.com. He was really grateful and humbled me with his response. We exchanged contact info and agreed to meet after I recovered from my trip.

Going into this I thought I was just going to build a site but while prepping for our first meeting I realized it could be a lot more. I had been working on a new post-signup checklist to help new site owners get their site ready to launch. Most of the details were figured out except the checklist tasks which I was still in the process of finalizing. My epiphany led to me drafting up an agenda which basically included latest version of my checklist — what a great way to test it out I thought.

We met at Starbucks, my usual out-of-office meeting place, and he brought a friend along. Mohamed confessed he wasn’t very good with computers and needed someone to help him out with the “technical stuff”. His friend was also a small business owner and managed multiple businesses so he was very interested to learn more about WordPress. After some personal introductions, I asked Mohamed to tell me more about his business and what he was hoping his website would do for him. Within a couple of minutes, I had enough information to start working through my checklist.

Through multiple experiences of my own and hearing about others, I would have to say generating content is the hardest part of building a site. We can help you create a site in seconds and give it a professional look, but without the content, it isn’t very useful. I tried to be resourceful and started by asking if they had any existing social accounts or websites. My thinking was that I could maybe get a logo, a colour pallet, some images perhaps, and if I am really lucky some writing too — but no, they didn’t have any accounts. This reminded me of a poll we once ran where almost 60% of 4400 people signing up said they didn’t have an existing online presence. It was also something really important to keep in mind so we can think of other ways to help people generate content for their sites.

As I went down my list, I would explain why the task was important and how I was executing it. They couldn’t believe how quick it was coming together and they loved the design even though they didn’t pick one. In our time together, we built out the entire architecture for their site, personalized it, set up multiple ways for people to contact them, and added some copy to get them going. The site wasn’t 100% complete but Mohamed was ecstatic that his business was no longer limited to his geographical location. He could now do business with people all over the world. I left them with some homework for next time we meet where we hope to wrap things up.

I walked away from this experience feeling really energized and excited about what I had learned. For one, it was great to see the impact our work has on people’s lives. I also learned a lot about our product and the people using it. My biggest take away was witnessing someone that hadn’t fully crossed the digital divide and how that affects his business. I noticed what a big role Mohamed’s mobile devices has on his work and ability to provide for his family. It’s so clear to me why a mobile-first, or even mobile-only, policy is so important when I see people like Mohamed turning to their phones over desktop computers to connect with the world around them. I highly recommend that people working on products seek out opportunities like these.

This post also appeared in Automattic’s product design blog.

Recruiting participants for remote user tests

I recently attended my first remote design sprint at Automattic. It ran over a two week period and  was attended by fellow designers, developers, data scientists, happiness engineers, and marketers. Together we created two prototypes and tested them out with real customers. I found the experience to be both rewarding and inspiring. We came up with a lot of great ideas and validated them too. It was also really nice collaborating with a number of people across the company that I don’t usually get to work with.

One of my major contributions to the sprint was recruiting real customers for the prototype test. The job was a whole lot easier thanks to an internal post written by my designer colleague Mike Shelton which also took part in the design sprint. He shared his experience recruiting customers from a previous test and also included the templates he created for people like myself to use. With that post, I had everything I needed once we determined exactly which type of customers we wanted to test.

Finding participants

Before the sprint even started, I had been running tests to see how many people I could recruit with a poll on our signup form at WordPress.com. Fortunately my tests worked out well so I knew I could count on running another one to collect participants for our test. I created a new poll using HotJar and added a little prompt at the end of the poll that asked participants to sign up for other customer research initiatives. This is what it looked like:

poll.png

Screening participants

Since we wanted to talk to a specific group of people, the link at the end of the poll lead to a Google form which asked visitors to complete a series of questions. We added questions that would help us qualify the people signing up for this test and other tests in the future. Since this was a remote prototype test, one of the key questions was if people were comfortable talking over a video conference.

screener-shot.png
View all the questions

Sending out invitations

I wanted the email to look like it was coming from a real person so I sent it with Inbox using my full name and company email address a week before we have our test date scheduled. It worked because some people replied back which was great — I was able to do some additional research for another project. I also made sure to add the qualified email addresses to the bcc field so that the recipients couldn’t see any other people being emailed.

These were the key points I included in the email:

  • Introduce myself and tell people where I got their email
  • Explain what we’re testing along with all the logistics
  • Identified what the recipient gets for participating — we decided to offer a $50 Amazon gift card because we felt like we were asking a lot for someone to take an hour out of their busy schedules
  • How to sign up for the test.

email.png

Coordinating schedules

Getting people across multiple time zones to agree on a date and time over email can be confusing and time consuming. There’s lots of back and forth involved and it’s easy to make mistakes.

That’s why we use Calendly to manage our calendar availability and bookings. It’s easy to setup, offers a number of customizable options, and displays times relative to the timezone of the viewer . You can also automate and personalize event reminders so that you have one less thing to worry about.

calendly.png

Sending a reminder

People’s lives are busy so it can be understandable if someone forgets something they committed to a week ago. Sending a reminder a couple hours before the test is a good way to avoid that and to add any last bits of logistical information required for the test. In my case, I made sure to include:

  • the time,
  • a link to download the software used for video conferencing — we use Zoom.us for it’s video quality, number of participants, screen sharing, and recording capabilities, and
  • a link to join your video conference.

Here’s what my email looked like:

reminder.png

Following up after the test

Lastly, we ended things off with one last email thanking our participants for their time. We included instructions for them to claim their $50 Amazon gift card and also took the opportunity to collect some final feedback about the test so we could make improvements for future participants.

follow-up.png

I hope you enjoyed reading this post and remember it next time you’re planning on recruiting participants for your own user test or interview. If you do, let me know how it works out for you by reaching out on Twitter.

This post also appeared in Automattic’s product design blog.

Creating empathy for your customers

In my last post, I wrote how we use interviews to learn about our customers. They allow us to ask them questions and in their responses we find insights and get inspired. In this post, I’d like to share some other ways we build empathy for the people using our products.

Support Rotation 💬

It starts from our first day on the job. Everyone that joins Automattic spends three weeks doing what’s called a support rotation. We learn about our products by reading support documentation and running through exercises. Once we’ve completed our training, we get to business and start offering support to our customers with our Happiness team. We start by answering inquiries by email and then upgrade to doing live chat.

After we complete our rotation we are released into our new roles with a wealth of knowledge about our company and our customers. That’s not the end though, once a year we return back to the Happiness for a week to offer more customer support. It’s a great way for us to keep in touch with the challenges our customers are facing.

Business concierge 💼

When people sign up for the WordPress.com Business plan they get access to a one on one setup appointment from a member of our Happiness team. The sessions are conducted over live chat or screen share and usually last around 30 minutes. The goal for these sessions is for the user to walk away feeling comfortable and confident building their site on WordPress.com.

In addition to helping our customers, we use these sessions to learn. Anyone can join in real time or listen to a recording at a time that suits them. There is no limit to how many sessions you can attend. After each session, the recording and notes are tagged and posted. Our support team uses these posts to generate monthly reports that give us great insights into who our customers are and the challenges they face. For someone like me that works on our signup and onboarding experiences the content in these reports is invaluable.

User research campfires 🔥

Every month members from across Automattic gather to share their stories about the research they’re conducting. We do presentations, have discussions, and talk about our processes. Through these sessions we hear about our customers and what we’re doing to overcome their challenges. It’s a great place to learn from each other and grow our skills a team. The meetups are recorded for people that can’t attend and serve as a reference to new people joining our team.

Web chefs 🍜

We are often encouraged to build sites for people we know or encounter. There is no better way to experience your product than using it in real world situations. With limited time, resources, and budgets you get a first hand encounter of the challenges people face using your product.

Out of all the activities I’ve written in this post, this is my favorite way of getting close to our customers and building empathy for them. The insights I’ve gained have helped me make better decisions in my design process and allow me to offer great feedback to my peers.

I’m always interested to learn how people conduct their research. What have you found to be an effective way of learning from your customers? Let me know in the comments below.

This post also appeared in Automattic’s product design blog.

Designing with words

Reading Storytelling for User Experience by Whitney Quesenbery and Kevin Brooks got me thinking about different ways of exploring ideas in my design process. It inspired me to try something new on a recent project. I found that both the end result and the journey were very rewarding.

Starting with words

For most projects I start with sketches, flow diagrams, or wireframes. The end result is always some sort of visual output. I have my graphic design background to thank for that. With this project, I changed things up a little and described my user journey in writing rather than illustrating it. The end result was like a story about our customer’s journey through our signup process.

I broke each step into a series of bullets that were short and concise. It was important for me to make them easy to read. My self imposed constraint forced me to keep things at a high level. This made it better for collecting feedback and moving onto the next stage of the project because people didn’t get caught up in small details.

Words are cheap

I don’t even need to think when I type because of how long I’ve been using a computer. It’s like I look at a screen and words appear right out my brain. Working on this project in this way made me realize that I can type quicker than I can sketch. I was able to cover lots of ground in less time and generated more ideas than I normally would.

I worked in waves by documenting my thoughts and then editing them. The ideas got better with each edit and sometimes they’d even spark new ones. I had to be diligent to capture them before they escaped. My keyboard skills came in hand as I copy and pasted new phrases, finished them off, and then got back to the original thought. After a while, the content began to mature. and I started to think about the implementation.

Structure brings Freedom

I remember my visual brain going wild while writing these flows. The descriptions painted such vivid pictures. I couldn’t wait to get to the visual part of the project. Once I did, I found it really easy to get going. Working within the constraints of the written text was very fun for me. With the high level details all figured out, I was able to focus on the details and get creative.

Conclusion

With this experience behind me, I can easily say I’d do it again. Heck, I’d like to see if there are other ways I can incorporate words into my design process. I’m going to continue exploring and writing about it here so check back soon. If you have any suggestions let me know, I’d love to hear about them.

Observing your customers in the wild

I’ve been trying a new ethnographic research technique called Sales Safari. It was developed by cross functional designers Amy Hoy and Alex Hillman. It unfortunately isn’t very well documented so I decided to write about it after reading how to do it in a book called  “Designing Products People Love” by Scott Hurff.

The goal of this approach is to create great products by studying what your customers are saying. Insights come from analyzing large quantities of data collected from online public spaces like forums and social networks. I thought it might work well to craft a great signup experience for our customers on the new project I’m working on.

How I ran my Sales Safari

I started by identifying the places where our customers hang out. Facebook, Twitter, LinkedIn, Google Plus, Reddit and the WP.com forums where a good places to start. Later, I also reviewed some of our customer’s sites, survey responses, and support requests. I closely read through the conversations while visiting each location. Phrases that stuck out to me were copied and pasted in a spreadsheet. I created a single keyword to summarize the phrase so I could categorize them later. Here’s how Hillman describes what you should be looking for:

“You start collecting jargon, some of their specific detailed language and words they use to describe the problem […] Elements and contributions to their worldview, their deep-seated beliefs that are unshakable. Then also the things that they talk about, they recommend. The things that they buy.”

As I progressed, I felt like this was a never ending process. There is so much content out there. Each conversation lead to another and sometimes even to a whole new “watering hole” — a term Amy uses to describe places where people go to share their views. I eventually decided I had enough data to take a peek. As I started grouping the phrases by keyword, the patterns started to become clear right away.

What I learned

After going through this process I definitely feel like I know more about our customers than when we started. This kind of research takes a lot of time and isn’t ever really over. I intend to continue doing it on a regular basis to stay in touch with what our customers are saying and feeling. This new understanding of our customers has both empowered and inspired me to explore different ways we can show people the value of our product early on so we they can stick around with us for longer.

Share your thoughts

Have you tried this approach before? Do you have other ways you like to learn from your customers? Reach out on Twitter and let me know.

Product and marketing: two sides of the same coin

Colin Bentley and Brian Donohue are product managers at Intercom — a product first company. They were interviewed on the company’s podcast about their experiences since they joined the company of 30. Today, there are 300 people working at Intercom and their product services over 100,000 active customers.

As a product first company it was normal for them to think if you built a great product, everything else would follow. People were cynical when Intercom started making their first marketing hires.

Over time, the product folks began to see they weren’t very different than the marketers. They realized they both focus on the needs of their customers. While the product managers build features, the marketers position them to their customer’s so they’re desirable and useful. As an example, Twitter’s tweet button could have been easily labeled submit. Instead, a lot of work went into positioning it as a Tweet button so that Tweeting could become a thing.

Today, Colin and Brian explain how important it is for their team to collaborate. It’s essential for them to fuse both the marketing and product perspectives. This allows them not only to build the most important things for their customers but also gives their product a unique place in the market.

Listen to the full podcast on Intercom’s site.

How to get started on your next design project

Getting started on a new project can be intimidating. Thankfully as designers we have these tools to help us get the ol’ juices flowing.

User flows

Experience your product as the people using it would. Take screenshots, write notes, and press every single button you find. Document your journey in steps as you go and take notes.

Stakeholder interviews

Learn from your peers and customers. Find anyone that’s worked on your project before you, speak with your peers, and interview the decision makers on the project. Most importantly, get in touch with the people using your final product.

Competitor analysis

See how your product measures up and find some inspiration along the way. Start by identifying your direct competitors. Then expand search to similar products. Finally look for companies in completely different fields. Make a list of the defining features of your product. As you review each competitor: document the features they have, take notes, and add new features to your list that you find interesting.

Sales safari

Gain a unique perspective about your customer. Find people using your product without influencing them in any way. The best way to do this is to look up things they’ve said in the past by checking your forums, customer support requests, old surveys, and social media accounts. Look out for how they speak, what they think about your product, and what challenges they’re running into.

Inspiration boards

Capture your favourite pieces to inspire you throughout your project. Visit your favourite inspiration sites or use screenshots from your competitor analysis. Save images and arrange them in a way that makes sense to you. Make it really visual and physical. Print it out if you have to and stick it somewhere where you can see it from time to time.

Data analysis

Find out how your product is performing. Get familiar with tools like Google Analytics, mySQL data bases, and what ever reporting software your company uses. Understand your business performance metrics so you can figure out how you can make a real difference both for your customers and your business.

Communication

Keep everyone up to date with your progress. Share your findings as frequently as you can to spur discussion and gather ideas. It easier to get buy in when people are involved in the process. Get people excited by posting your diagrams, notes, and sketches for everyone to see.

Conclusion

What does your design process look like? Is there anything on my list that’s missing? Want to learn more about one of these in more depth. Reach out on Twitter and let me know.

The first mile: the key to long term success for your product

I just listened to Scott Bellsky explain the importance of a product’s first mile on Intercom’s podcast. The first mile comes right after someone signs up for your product. It is often referred to as an onboarding experience. This is where you orient your users within your product and give them a sense of what they’re going to do next. Careful attention is paid to the type of copy you write, how your tour looks, and what defaults you set for your users.

According to Scott, the key is to have someone do a few key things that shows them the value of your product over the long term.

Here are some of the notes I took from the podcast:

  • The first mile is usually the last mile a team focuses on. Teams often spend lots of time building their products and only think about the onboarding experience as an after thought or when they see their users aren’t sticking around.
  • Scott believes that in the first 15-30 seconds of every new customers experience, they are lazy, vein, and selfish. Lazy because they don’t want to read or do any work. Vein because they want to know how the product will make them better. Selfish because they want the product to make them look good. These are important characteristics to keep in mind when crafting this experience.
  • He suggests that maybe we shouldn’t tell them or show them, but rather we should just do it for them with the defaults completed.
  • It’s important to remember we can’t believe people will have a relationship with us if they can’t get through those first interactions. They can’t buy into our vision or to recommend us to other people. It just needs to serve them now!
  • The first mile is successful when users reach the “zone”. When they experience that sense of accomplishment and see how the product will serve them in the future.
  • Messaging apps like Telegram and Slack have really smooth onboarding. They even think about the small things like the magic link to log you in because when users come back the second time, they don’t usually remember their passwords.
  • The first mile isn’t something that can be done once and forgotten about. It’s important to continually optimize the first mile by understanding your users and being empathetic to their needs and circumstances.

You can listen to the full podcast on Intercom’s blog or your can learn more about the first mile by reading Scott’s blog post on Medium.