Build, measure, learn for the win

One of the first projects I worked on at Automattic was a small A/B test for the first screen of the WordPress.com sign-up process. As the first screen in our funnel, it served a critical role, but it hadn’t been tested in quite some time. I tweaked the visual design and completely changed the copy for my first attempt at improving conversions on that screen. Setting it up and tracking the results felt really easy compared anything I had done before. I felt empowered and simultaneously discovered how projects work at Automattic.

That first test was a success and was implemented as the default for all our visitors. It remained that way for a little under a year as our priorities shifted to higher impact areas of our funnel. Little bits of feedback trickled in over time that kept me thinking about many possible improvements but it wasn’t until recently that I decided to take action.

The problem

One of the most common complaints I heard about our screen was that the terminology was confusing. We asked people what type of site they wanted and presented them with four options: a blog, a website, a portfolio, and an online store. These were some of the questions that kept coming up: what’s the difference between a website and a blog? can I change my mind later? two of these options sound good, what if I want both?

Our mistake was assuming people understood what to us seemed like simple concepts but for them were actually foreign. This made a lot of sense when I look back at all the polls I ran and conversations I had with our customers over the last year. I learned that our customers are not like us and that the majority of people signing up at WordPress.com had little to no experience building sites or using WordPress.

screen-shot-2018-01-02-at-10-38-08-am.jpg
Graphic courtesy John Maeda and Rebrand Cities.

A new hope

Armed with this awareness about our customers, I reworked this screen to simulate a first meeting between a design agency and their client. The screen would ask them a series of questions to understand their needs, and then our tools would build a site that addressed their goals. My biggest concerns with this approach were 1) not scaring people with too many questions so early in the signup flow, and 2) knowing which questions to ask.

I quickly developed high level strategies to tackle both these problems. Designing with a mobile context in mind would ensure that I made it quick and easy for people to answer our questions and using a tool like Hotjar, I could quickly run a series of polls to validate and modify my questions.

Learning our customer’s vocabulary

One of my questions could be answered in a number of ways and therefore required an open text field to collect people’s responses. Of course, typing on a phone sucks, so I wanted to make the field as painless as possible. An autocomplete feature made the most sense as it offered people suggestions while they typed their answer. We could not only save people time from writing out the full word but we would also get cleaner data too. To pull this off, I needed a collection of commonly used terms to power the suggestions.

I created a new poll that included the question as it would appear in my design and quickly noticed the answers weren’t what I expected. Thanks to HotJar I was able to turn off the poll, rephrase the question, and launch the poll again within a matter of minutes — all without writing a single line of code. This time around I was happy with the data coming in and let the poll run until we had enough responses. After a couple days we had a lot of raw data to comb through so I used a spreadsheet and a PHP script to clean it up and help us make sense of it all.

php2.jpg

Build, measure, learn

Another question asked people to pick their answer(s) from a handful of choices. I drafted a list of options based on data we had collected in the past and ran them in a poll along with an “other” field that allowed people to type their answer if couldn’t find what they were looking for in my list.

The first iteration of this poll was dominated by responses entered in the “other” field. Some of them were variations of what I had in my list but there were also lots of common responses that were not represented in my list at all. I used this data to update my list and then published another version of the poll. There were some decent improvements in the results but it was still not quite there yet so I rinsed and repeated. This time I got it right because the majority of the responses were selected from my list rather than entered into the “other” field. In addition to that, most of the people that did fill out the “other” field either flipped me the bird or just said hi.

Progress on multiple fronts

I wasted no time while these polls were running and started building the form right after the design was sketched out. The code was built in small chunks which made it easy to review and kept the momentum of the project going with visible progress. As the data came in, I added it to my work in progress until I had enough of a prototype to run some usability tests. The tests revealed some really minor tweaks that needed to happen but mostly confirmed that we were on the right path. Then, shortly after I completed my last poll, I had everything I needed to run my A/B test at scale and see how this new design stacked up against the old one.

The results

It didn’t take long for us to reach statistical significance but I still let the test run for at least seven days to account for any seasonality during the week. The results indicated that this design profoundly influenced our business metrics. Like it says in NEA’s Future of Design report:

Ultimately everything a design team does has consequences (both good and bad) for a business. The more you know about the business and its goals/intentions, the better a design team is positioned to deliver good experiences and results for the business.

— Tim Riley: Senior Director of Digital Experience at Warby Parker

This post also appeared on Automattic’s Design blog.

Driving more traffic to my blog

I have been running my blog for a little over a year now and after getting over the fear of writing content, the next biggest challenge I ran into was driving traffic to it. It seems I did something right with one of my posts that brings in one or two organic visitors a day but I’m not entirely sure what I did to achieve that. For the rest of my content, I noticed the amount of traffic coming in is equal to the amount of effort I put into promoting it. I wanted to see if there was a more sustainable way of growing traffic for my blog so I reached out to some experts in the field to see what they had to say.

Luca Sartoni

Luca is a Growth Engineer at Automattic, he’s my go-to reference for everything a/b test related and as of now also SEO/SEM. When I asked Luca what I should do to get more traffic to my site he responded with the advice he gives everyone that asks the same question:

It’s all about consistency, differentiation of channels, and concertation.

Consistency: A piece of content should be consistently promoted across all your channels. Images and headlines are important for grabbing attention so make sure to use something that works well everywhere keeping in mind that images still need to be formatted appropriately for each channel. Luca suggests that you can’t go wrong with a title under 120 characters.

Differentiation of channels: For maximum impact, a piece of content should be distributed on all channels. Every channel has a different cost and reach so your overall distribution will be impacted by the amount of money or time you spend on it.

Luca finds that paid channels always out perform organic ones. Although organic channels are free, they end up being more expensive when you look at the time and resources you put into getting it right. It’s is hard to measure your efforts and can take a long time to see the results come through. Alternatively, with paid channels you can buy traffic and optimize your output in real time.

Concertation: Channels are not only interdependent on each other but also complimentary. It’s important to keep in mind how they all play off each other. Social media channels like Facebook also operate as search engines. For this reason, your efforts for picking the best keywords are not only important for search engines but also on social media channels. Another good example is how traffic coming to your site through an organic search can drive traffic to your social channels which would increase your ability to engage with these people.

Luca stresses that focusing on a single channel won’t get you the results you need. You shouldn’t be thinking of search engine optimization OR search engine advertising, it should be search engine optimization AND search engine advertising.

Simon Heaton

Simon is a Growth Marketing Manager at Shopify, we worked together for a couple years on the same team and saw some incredible growth on the Shopify Partners Blog thanks to Simon’s efforts. He believes that search engine optimization is a key contributor to sustainable growth. It can be hard to compete on keywords depending on your subject matter so Simon suggests going with a long-tail approach where you optimize for a bunch of small, more relevant keywords where the aggregate volume has potential to be larger than those whales.

According to Wordstream.com long-tail keywords are:

Long-tail keywords are longer and more specific keyword phrases that visitors are more likely to use when they’re closer to a point-of-purchase or when they’re using voice search. They’re a little bit counter-intuitive, at first, but they can be hugely valuable if you know how to use them.

Take this example: if you’re a company that sells classic furniture, the chances are that your pages are never going to appear near the top of an organic search for “furniture” because there’s too much competition (this is particularly true if you’re a smaller company or a startup). But if you specialize in, say, contemporary art-deco furniture, then keywords like “contemporary Art Deco-influenced semi-circle lounge” are going to reliably find those consumers looking for exactly that product.

Managing long-tail keywords is simply a matter of establishing better lines of communication between your business and the customers who are already out there, actively shopping for what you provide.

Obviously, you’re going to draw less traffic with a long-tail keyword than you would with a more common one, but the traffic you do draw will be better: more focused, more committed, and more desirous of your services.

Alex Hosselet

Alex is Digital & Marketing Strategist for United Way Ottawa, we worked together a number of years where Alex was a marketing manager on a product I designed for. I always admired Alex’s passion for connecting with people and measuring results.

Alex’s advice was to make sure you have a good combination of content strategy and search engine optimization. Content strategy is all about making the right content that attracts people. That involves taking the time to understand your audience and developing a plan with clear goals that can be measured. Search engine optimization is a combination of writing, research, and coding best practices.

Takeaways

My biggest take away is there’s no silver bullet to getting more traffic — it’s a lot of hard work. If you want people to visit your site, you need to pay with your time or money. Based on these conversations, here are some of the tactics I’ll employ to increase traffic to my blog:

  • Share content on as many channels as possible,
  • Try paid advertising,
  • Optimize posts for long-tail keywords and SEO best practices, and
  • Improve visibility of my social media channels on my website.

Building better app layouts

I’ve been writing CSS for over ten years and I’m glad to finally see there’s a way to create a fixed navigation that doesn’t feel like a hack. Among other issues, normally you would need to add ghost padding or margins to account for the fixed positioning. Thanks to the adoption of Flexbox that should be a thing of the past.

I’m working on a design for an App-ified PWA version of my blog and wanted to try something different with the navigation. My design has a navigation bar fixed to the bottom of small screens for easy and comfortable access while maintaining a conventional navigation at the top of the screen for large ones. I thought that would be perfect for Flexbox and there’s no better way to see how it feels than to mock it up.

With my trusted CSS Tricks guide at hand I made my way to Codepen and created a prototype to try it out. I used Flexbox to create two panes that take up all the space on the screen. One with a fixed height for the navigation and another big stretchy one for the content. It was easy for me to change the position of the navigation on different screen sizes thanks to Flexbox’s ability to change the direction columns flow.

The code

This is the code I used to make it work:

HTML

<div class="navigation">
  ... navigation goes here
</div>

<div class="content">
  ... content goes here
</div>

CSS

html {
  height: 100%
}

body {
  flex-grow: 1;
  height: 100%;
  display: flex;
  flex-direction: column-reverse;
}

@media (max-width: 850px) {
  body {
    flex-direction: column;
  }
}

.content {
  flex-grow: 1;
  overflow-y: auto;
}

The prototype

Head on over to Codepen, check it out for yourself, and let me know what you think in the comments below.
View the prototype

Reflections on my time off from work

After nine long months, my wife and I finally had our third baby Alessio. It was an emotional, exciting, and exhilarating experience to be part of the birth in our home. Thankfully, I was able to take some time off from work so we could get adjusted to the new norm.

Over the years, I have come to realize that it’s best to completely disconnect from work while I’m on vacation or away from the office — there’s nothing worse than reading that one email, or Slack message, that comes in and occupies your thoughts for the rest of your time off. This time around things were a little different leading up to my time off so I had to reconsider my position. I was right in the middle of a project that had a lot of momentum and wasn’t sure when the baby would come. I’ve been told to think of a baby’s due date as more of a due month. Anyways, I decided I’d check in from time to time to stay on top of things and make sure I didn’t fall behind.

I’m happy to report that nothing came up that totally distracted me from my time at home with the family. My colleagues were very respectful of me being away and almost completely reduced their interactions with me. A couple things of interest popped up from time to time but I felt bad chiming in because of the expectation it could set with other colleagues — I don’t want anyone to think they need to check-in and comment on my stuff while they’re away because I did.  I thought it would be better to leave those items unaddressed until I got back.

When I got back from work things were just as they were when I left. I still had a lot of catching up to do and realized the efforts I took to stay up to date didn’t really make much of a difference. With that said, checking in from time to time didn’t really take away from my leave in any impactful way but it was a distraction nevertheless. Faced with the choice again, I feel confident in my belief that it’s best to disconnect completely when I’m away from work.

Crossing the digital divide

Taxiing home after my first Automattic Grand Meetup in Whistler, I was happy to get to know my driver Mohamed. He has been running his family business for over 20 years and also drives a taxi to help make ends meet. Like me, he has three kids and we spoke about them at length. We also spoke about work and got excited when he heard that I’m designer at WordPress.com. He pulled out his phone right away and told me that he downloaded the app with the intention of starting a new site but didn’t have the time to get to it.

We’re often encouraged to work with people that use our products because it helps us see our product through their eyes. This seemed like the perfect opportunity so I offered to build Mohamed his site on WordPress.com. He was really grateful and humbled me with his response. We exchanged contact info and agreed to meet after I recovered from my trip.

Going into this I thought I was just going to build a site but while prepping for our first meeting I realized it could be a lot more. I had been working on a new post-signup checklist to help new site owners get their site ready to launch. Most of the details were figured out except the checklist tasks which I was still in the process of finalizing. My epiphany led to me drafting up an agenda which basically included latest version of my checklist — what a great way to test it out I thought.

We met at Starbucks, my usual out-of-office meeting place, and he brought a friend along. Mohamed confessed he wasn’t very good with computers and needed someone to help him out with the “technical stuff”. His friend was also a small business owner and managed multiple businesses so he was very interested to learn more about WordPress. After some personal introductions, I asked Mohamed to tell me more about his business and what he was hoping his website would do for him. Within a couple of minutes, I had enough information to start working through my checklist.

Through multiple experiences of my own and hearing about others, I would have to say generating content is the hardest part of building a site. We can help you create a site in seconds and give it a professional look, but without the content, it isn’t very useful. I tried to be resourceful and started by asking if they had any existing social accounts or websites. My thinking was that I could maybe get a logo, a colour pallet, some images perhaps, and if I am really lucky some writing too — but no, they didn’t have any accounts. This reminded me of a poll we once ran where almost 60% of 4400 people signing up said they didn’t have an existing online presence. It was also something really important to keep in mind so we can think of other ways to help people generate content for their sites.

As I went down my list, I would explain why the task was important and how I was executing it. They couldn’t believe how quick it was coming together and they loved the design even though they didn’t pick one. In our time together, we built out the entire architecture for their site, personalized it, set up multiple ways for people to contact them, and added some copy to get them going. The site wasn’t 100% complete but Mohamed was ecstatic that his business was no longer limited to his geographical location. He could now do business with people all over the world. I left them with some homework for next time we meet where we hope to wrap things up.

I walked away from this experience feeling really energized and excited about what I had learned. For one, it was great to see the impact our work has on people’s lives. I also learned a lot about our product and the people using it. My biggest take away was witnessing someone that hadn’t fully crossed the digital divide and how that affects his business. I noticed what a big role Mohamed’s mobile devices has on his work and ability to provide for his family. It’s so clear to me why a mobile-first, or even mobile-only, policy is so important when I see people like Mohamed turning to their phones over desktop computers to connect with the world around them. I highly recommend that people working on products seek out opportunities like these.

This post also appeared in Automattic’s product design blog.

Recruiting participants for remote user tests

I recently attended my first remote design sprint at Automattic. It ran over a two week period and  was attended by fellow designers, developers, data scientists, happiness engineers, and marketers. Together we created two prototypes and tested them out with real customers. I found the experience to be both rewarding and inspiring. We came up with a lot of great ideas and validated them too. It was also really nice collaborating with a number of people across the company that I don’t usually get to work with.

One of my major contributions to the sprint was recruiting real customers for the prototype test. The job was a whole lot easier thanks to an internal post written by my designer colleague Mike Shelton which also took part in the design sprint. He shared his experience recruiting customers from a previous test and also included the templates he created for people like myself to use. With that post, I had everything I needed once we determined exactly which type of customers we wanted to test.

Finding participants

Before the sprint even started, I had been running tests to see how many people I could recruit with a poll on our signup form at WordPress.com. Fortunately my tests worked out well so I knew I could count on running another one to collect participants for our test. I created a new poll using HotJar and added a little prompt at the end of the poll that asked participants to sign up for other customer research initiatives. This is what it looked like:

poll.png

Screening participants

Since we wanted to talk to a specific group of people, the link at the end of the poll lead to a Google form which asked visitors to complete a series of questions. We added questions that would help us qualify the people signing up for this test and other tests in the future. Since this was a remote prototype test, one of the key questions was if people were comfortable talking over a video conference.

screener-shot.png
View all the questions

Sending out invitations

I wanted the email to look like it was coming from a real person so I sent it with Inbox using my full name and company email address a week before we have our test date scheduled. It worked because some people replied back which was great — I was able to do some additional research for another project. I also made sure to add the qualified email addresses to the bcc field so that the recipients couldn’t see any other people being emailed.

These were the key points I included in the email:

  • Introduce myself and tell people where I got their email
  • Explain what we’re testing along with all the logistics
  • Identified what the recipient gets for participating — we decided to offer a $50 Amazon gift card because we felt like we were asking a lot for someone to take an hour out of their busy schedules
  • How to sign up for the test.

email.png

Coordinating schedules

Getting people across multiple time zones to agree on a date and time over email can be confusing and time consuming. There’s lots of back and forth involved and it’s easy to make mistakes.

That’s why we use Calendly to manage our calendar availability and bookings. It’s easy to setup, offers a number of customizable options, and displays times relative to the timezone of the viewer . You can also automate and personalize event reminders so that you have one less thing to worry about.

calendly.png

Sending a reminder

People’s lives are busy so it can be understandable if someone forgets something they committed to a week ago. Sending a reminder a couple hours before the test is a good way to avoid that and to add any last bits of logistical information required for the test. In my case, I made sure to include:

  • the time,
  • a link to download the software used for video conferencing — we use Zoom.us for it’s video quality, number of participants, screen sharing, and recording capabilities, and
  • a link to join your video conference.

Here’s what my email looked like:

reminder.png

Following up after the test

Lastly, we ended things off with one last email thanking our participants for their time. We included instructions for them to claim their $50 Amazon gift card and also took the opportunity to collect some final feedback about the test so we could make improvements for future participants.

follow-up.png

I hope you enjoyed reading this post and remember it next time you’re planning on recruiting participants for your own user test or interview. If you do, let me know how it works out for you by reaching out on Twitter.

This post also appeared in Automattic’s product design blog.

Creating empathy for your customers

In my last post, I wrote how we use interviews to learn about our customers. They allow us to ask them questions and in their responses we find insights and get inspired. In this post, I’d like to share some other ways we build empathy for the people using our products.

Support Rotation 💬

It starts from our first day on the job. Everyone that joins Automattic spends three weeks doing what’s called a support rotation. We learn about our products by reading support documentation and running through exercises. Once we’ve completed our training, we get to business and start offering support to our customers with our Happiness team. We start by answering inquiries by email and then upgrade to doing live chat.

After we complete our rotation we are released into our new roles with a wealth of knowledge about our company and our customers. That’s not the end though, once a year we return back to the Happiness for a week to offer more customer support. It’s a great way for us to keep in touch with the challenges our customers are facing.

Business concierge 💼

When people sign up for the WordPress.com Business plan they get access to a one on one setup appointment from a member of our Happiness team. The sessions are conducted over live chat or screen share and usually last around 30 minutes. The goal for these sessions is for the user to walk away feeling comfortable and confident building their site on WordPress.com.

In addition to helping our customers, we use these sessions to learn. Anyone can join in real time or listen to a recording at a time that suits them. There is no limit to how many sessions you can attend. After each session, the recording and notes are tagged and posted. Our support team uses these posts to generate monthly reports that give us great insights into who our customers are and the challenges they face. For someone like me that works on our signup and onboarding experiences the content in these reports is invaluable.

User research campfires 🔥

Every month members from across Automattic gather to share their stories about the research they’re conducting. We do presentations, have discussions, and talk about our processes. Through these sessions we hear about our customers and what we’re doing to overcome their challenges. It’s a great place to learn from each other and grow our skills a team. The meetups are recorded for people that can’t attend and serve as a reference to new people joining our team.

Web chefs 🍜

We are often encouraged to build sites for people we know or encounter. There is no better way to experience your product than using it in real world situations. With limited time, resources, and budgets you get a first hand encounter of the challenges people face using your product.

Out of all the activities I’ve written in this post, this is my favorite way of getting close to our customers and building empathy for them. The insights I’ve gained have helped me make better decisions in my design process and allow me to offer great feedback to my peers.

I’m always interested to learn how people conduct their research. What have you found to be an effective way of learning from your customers? Let me know in the comments below.

This post also appeared in Automattic’s product design blog.

Designing with words

Reading Storytelling for User Experience by Whitney Quesenbery and Kevin Brooks got me thinking about different ways of exploring ideas in my design process. It inspired me to try something new on a recent project. I found that both the end result and the journey were very rewarding.

Starting with words

For most projects I start with sketches, flow diagrams, or wireframes. The end result is always some sort of visual output. I have my graphic design background to thank for that. With this project, I changed things up a little and described my user journey in writing rather than illustrating it. The end result was like a story about our customer’s journey through our signup process.

I broke each step into a series of bullets that were short and concise. It was important for me to make them easy to read. My self imposed constraint forced me to keep things at a high level. This made it better for collecting feedback and moving onto the next stage of the project because people didn’t get caught up in small details.

Words are cheap

I don’t even need to think when I type because of how long I’ve been using a computer. It’s like I look at a screen and words appear right out my brain. Working on this project in this way made me realize that I can type quicker than I can sketch. I was able to cover lots of ground in less time and generated more ideas than I normally would.

I worked in waves by documenting my thoughts and then editing them. The ideas got better with each edit and sometimes they’d even spark new ones. I had to be diligent to capture them before they escaped. My keyboard skills came in hand as I copy and pasted new phrases, finished them off, and then got back to the original thought. After a while, the content began to mature. and I started to think about the implementation.

Structure brings Freedom

I remember my visual brain going wild while writing these flows. The descriptions painted such vivid pictures. I couldn’t wait to get to the visual part of the project. Once I did, I found it really easy to get going. Working within the constraints of the written text was very fun for me. With the high level details all figured out, I was able to focus on the details and get creative.

Conclusion

With this experience behind me, I can easily say I’d do it again. Heck, I’d like to see if there are other ways I can incorporate words into my design process. I’m going to continue exploring and writing about it here so check back soon. If you have any suggestions let me know, I’d love to hear about them.

Observing your customers in the wild

I’ve been trying a new ethnographic research technique called Sales Safari. It was developed by cross functional designers Amy Hoy and Alex Hillman. It unfortunately isn’t very well documented so I decided to write about it after reading how to do it in a book called  “Designing Products People Love” by Scott Hurff.

The goal of this approach is to create great products by studying what your customers are saying. Insights come from analyzing large quantities of data collected from online public spaces like forums and social networks. I thought it might work well to craft a great signup experience for our customers on the new project I’m working on.

How I ran my Sales Safari

I started by identifying the places where our customers hang out. Facebook, Twitter, LinkedIn, Google Plus, Reddit and the WP.com forums where a good places to start. Later, I also reviewed some of our customer’s sites, survey responses, and support requests. I closely read through the conversations while visiting each location. Phrases that stuck out to me were copied and pasted in a spreadsheet. I created a single keyword to summarize the phrase so I could categorize them later. Here’s how Hillman describes what you should be looking for:

“You start collecting jargon, some of their specific detailed language and words they use to describe the problem […] Elements and contributions to their worldview, their deep-seated beliefs that are unshakable. Then also the things that they talk about, they recommend. The things that they buy.”

As I progressed, I felt like this was a never ending process. There is so much content out there. Each conversation lead to another and sometimes even to a whole new “watering hole” — a term Amy uses to describe places where people go to share their views. I eventually decided I had enough data to take a peek. As I started grouping the phrases by keyword, the patterns started to become clear right away.

What I learned

After going through this process I definitely feel like I know more about our customers than when we started. This kind of research takes a lot of time and isn’t ever really over. I intend to continue doing it on a regular basis to stay in touch with what our customers are saying and feeling. This new understanding of our customers has both empowered and inspired me to explore different ways we can show people the value of our product early on so we they can stick around with us for longer.

Share your thoughts

Have you tried this approach before? Do you have other ways you like to learn from your customers? Reach out on Twitter and let me know.