Synthesizing usability test results

This week we got another usability study in the books by testing the most recent iteration of our signup and onboarding flow. Thanks to some help from our design operations team I was able to pull it off without a hitch. All I had to do was fill in a request, slightly massage a template script, and then show up for the sessions. Our recruitment coordinator, the amazing Lilibet Greig, did the rest by finding participants, screening them, and then following up with them to make sure they showed up for the study.

Running the test

Over the course of a week, I worked with two other designers to interview six participants and watched them build a website with WordPress.com. Each key segment of our customer base was represented and their experience using site builders ranged from novice to professional.

Starting from our marketing home page, they were asked to verbalize their thoughts as they made their way through the process. While one of us spoke with and guided the participants, the others documented their observations and quotes as individual cards on our virtual Mural board. The board was divided into sections representing each screen in our flow and we used different colour “stickies” for each participant.

Synthesizing the results

Once we completed the sessions, I made sure to go through all the recordings, including the ones I moderated, and added my notes to the board. It might seem redundant for multiple people to take notes but I think it’s important to bring multiple perspectives from different view points because we all perceive things differently.

We gathered a great deal of raw data from this study and I won’t lie when I say it was kind of intimidating to think how long it would take to comb through it all but I eventually got through it thanks to the colour coding and organization of the board. I read every single note and grouped similar items to identify patterns which lead to insights that I used to put a report together along with my recommendations.

Creating a report

The final report turned out to be fairly long due to the amount of ground we covered in the study. I therefore summarized the top findings and recommendations on the front page to make it easy to digest. I made sure to include direct quotes and video snippets though out the rest of the report to back up the insights because nothing beats seeing things with your own eyes.

Now that we have our list of recommendations, I’m going to work with my product and engineering partners to prioritize them using the ICE method.

The outcome

Doing usability studies, and customer interviews, are always exciting for me because not only can you uncover problems with your design but you can also be inspired with new ideas. This study was no different — we uncovered some issues in our flow and by means of observing people use our product, I come up with some new directions to explore.

Growing my skills with side projects

I recently started working on a new side project and noticed it’s starting to paying off both personally and professionally. It got me thinking about all the projects I have done before it and one of the common threads that stuck out was that they each helped me feel happier and healthier. They created a space for me to break away from work, practice my craft, and experience new things.

Now, as a father of three kids with a very demanding job, I will be the first to admit that it’s hard to make time for this kind of thing but just like just like exercising, I believe that taking on projects outside of work is something I need to prioritize to be on the top of my professional and personal game. As I reflected back, I thought I’d share some of the projects I have worked on and how they’ve paid off:

Hundred days of sketching

A little over a month ago I committed to a hundred days of sketching as part of the hundred day creative challenge. At the end of most days, I open my sketch book and spend some time running through a number of exercises until I’m happy with the end result or my page is full. I almost always start with the date and then reflect back on my day while sketching all kinds of things. After I’m done, I post the sketch up on a blog I created just for this very purpose so I can look back over time and also hold myself accountable.

The mental pause from work, and everyday life, is very relaxing. In some ways it’s like therapy as I process my thoughts and express them visually. I have also found myself rediscovering old personal interests like nature, patterns, numbers, and letters. The every day practice of exploring and being creative feels like it’s rubbing off on my work because it feels easier for me to come up with more comps when I’m evaluating solutions to a problem.

I have begun looking forward to this time everyday and look forward to continuing with this practice hopefully beyond the hundred day challenge.

Design herald

A couple years ago, I wanted to have a go at creating my own app and looked at my personal interests for inspiration. Since I like staying up to date with the news, and also get lots of enjoyment from apps like Yahoo and Google News, I thought why not create a news app for designers. I came up with an idea for an app called Design Herald that would aggregate and curate the top UX design news from across the web.

It was a lot of fun coming up with the name, doing some branding work, and designing the interface. From a development perspective, I decided to go with React Native because it’s a JavaScript language, which I know how to write, and it compiles to multiple platforms like iOS and Android — plus it was what everyone was talking about at the time. I dove right in to the code and learned as I went. I built a backend tool with Drupal that collected all the news and aggregated it into a single source that could be consumed by the app.

After a couple weeks, I had a fully functioning app working on my local machine. That’s when I ran into problems because I couldn’t find any documentation for actually publishing the app to the various App Stores. I was a little disappointed but not overly worried because I learned a new programming language and had lots of fun in the process. A month or two later, I started a new job at Automattic and was happy to find out that they used React as their code base for WordPress.com. Thanks to my side project I was able to jump right into the code and add extra value by executing my own a/b tests and project work.

Design herald — take two

Although my initial attempt at launching Design Herald didn’t work, the idea didn’t die there. Almost two years later, an opportunity to revive my idea presented itself when I was promoted to Design Director of the UX practice at Automattic. I was assigned seven direct reports and one of my responsibilities, as their director, was to level up their skills in the UX discipline. Sharing the latest design news and articles seemed like an obvious first step in doing that but regularly posting links into our Slack group got annoying very quickly — that’s when I remembered Design Herald.

I dug up my old app and spent a bunch of time trying to get it running again but that wasn’t going anywhere so I thought I would approach it from a different angle — this time I had the idea of combining WordPress and IFTTT. With IFTTT, I was able to “watch” any site’s RSS feed and then, if the feed updated, I could post the article to my new WordPress site. Compared to the app that took me a couple weeks to develop, this new approach only took me a couple days to get up and running. All I needed to do was create a WordPress.com site, gather up my sources, and setup all my “Applets” in IFTTT.

One of the first things I noticed after setting it up was there was a lot of sponsored content coming through. An update to my IFTTT settings allowed me to set the posts as drafts rather than publishing then publicly right away and that was all fixed up. This new work flow also meant that I needed to spend a little extra time to review the posts but that actually turned out to be a good thing for me because now I’m staying up to date with all the latest UX news.

Personal site

When I first read about progressive web apps my imagination went wild thinking of all the different things you could do with them. Unfortunately the role I was working in at the time was unrelated and didn’t afford me the opportunity to explore those possibilities. I therefore decided to pursue this interest on my own time and created a side project to dig in and learn more.

My idea was to rethink my personal website as a progressive web app. At the time, I had been blogging with WordPress.com for a couple months and the site I had was more than enough but when I looked at what some of the competitors we’re doing, there were parts of it that were outdated. The new format of a progressive web app gave me the opportunity to incorporate some more modern blogging features along with some of the benefits that progressive web apps bring.

Once I completed my new design, I took the opportunity to practice my React skills and started coding. Looking back, I can easily say this was the most ambitious coding project I ever worked on and in the end it payed off because not only did I get a new site but I also learned a lot about developing with React. Thanks to all those extra hours of practice, I feel more confident taking on development projects in my day to day work and have also become a regular contributor to our development efforts.

Hundred days of patterns

The first time I heard about the hundred day creative project was back in 2014. Cassie Slack, a designer colleague at Shopify, introduced me to the idea while presenting her work to the rest of the design team. Her work inspired a number of us to try the project or for ourselves. Some people created visual art like portraits and illustrations, some wrote stories and limericks, and others played music — I chose to design patterns.

My first patterns were replicas and then I moved on to making my own. Early on in the project, I noticed that I liked working with colour too so that introduced a whole other dimension for me to experiment with. I tried all kinds of combinations and even went monochromatic for a while. The constrains I placed on myself lead to some interesting outcomes.

I noticed the creative practice rubbing off on my day to day work as a marketing designer. It started getting easier for me to come up with lots of different ideas for our campaigns and I also experimented with colours I would never dream of using before. The overall experience left quite an impression on me because I often tell people about it and I also just decided to pickup another hundred day challenge again.

Wrapping up

If you made it all the way down here, thank you for reading. I hope you can see how beneficial these projects have been for me. If you’re working on a project yourself or have any ideas for a new one, I’d love to hear about them — just reach out to me over LinkedIn.

Managing a project with a remote team

One of the things I like most about working at a distributed company is that I get to work with people from all over the world. Being exposed to other cultures and ways of life has helped me grow as a person. This perk can however come with its challenges as I learned after leading a number of projects with colleagues spread across multiple time zones.

Coordinating meetings, staying in sync, and maintaining momentum can be tricky at times but it’s not impossible thanks to a number of tools and processes which I’ll cover in this post. I’ll start with the brief because it’s a great starting point for every project.

The brief

A brief is a high level document that tells you everything you need to know about a project. The idea is to keep it short so that anyone can easily consume it and get up to speed with what you’re building, why you’re building it, and the current status of the project. It’s the kind of thing you want to share early in a project to collect feedback and align stakeholders on what you’re trying to do.

Everyone approaches their briefs a little differently because of the nature of their project, company, or stakeholders. I like to treat them as living documents and start with a set of questions that evolve over time as we get closer to delivery. These are the questions I used for my last project:

  • What problem are you trying to solve?
  • What’s your bet?
  • What’s the desired outcome?
  • How will you measure success?
  • Who are the stakeholders?
  • Frequently asked questions

Thanks to its collaborative features, Google Docs is the perfect tool for working on a brief. It makes it really easy for anyone to jump in, leave comments, or contribute at any time.

Kick off meeting

Once you have your brief in a good place, and you know what problem your trying to solve, then it’s a good time to meet with your team and kick things off. The goal of this meeting should be to determine clear roles for everyone on the team but it’s also a great time to review the brief, discuss timelines, and agree on how you will work as a team.

At Automattic, we do the majority of our calls over Zoom video chat and use a tool called Doodle to help schedule a time that works for everyone. There are times we use Slack Video Conferencing or Google Hangouts but prefer Zoom for groups of people because it has the best quality. For these types of calls, we record the meeting, take notes, and post them in a place where people, that couldn’t make the call, can comment on them at a later date.

Slack

It’s important for distributed teams to have an open line of communication so they can share their progress, stay up to date with what others are doing, and ask questions. I like to use a dedicated project channel on Slack to create a virtual home for the team to collaborate and have discussions.

We value transparency at Automattic so these are usually public channels that anyone at the company can join to keep track of progress, provide feedback, and ask questions. As good as Slack can be for teams to communicate in real time, it can also be tough for folks in different timezones to participate in conversations because things can easily get lost in the back scroll. For that reason, it’s good to consider additional commutation channels that are better at preserving decisions and facilitating asynchronous discussions.

P2

At Automattic, we use an internal network of blogs, called P2, to document decisions and communicate asynchronously. There’s a social element to the blogs so that you can @mention other folks to grab their attention or cross post your message to another blog if you’d like to share something with an entire group of people following that blog. Being able to comment on each other’s posts is where the real magic happens. At first, it can be tricky to know where you post something, or what you post on P2 compared to sharing it in Slack, but over time it starts to feel very natural.

In the context of our projects, we have a dedicated P2 to memorialize decisions made in Slack, share updates across the company, and getting feedback from people outside the project team.

Project board

Having a board where you can see every piece of work that needs to happen is a beautiful thing. When it’s done right, people know exactly what to do and do it. Momentum builds and projects get delivered. This is a habit traditionally practiced by developers but I have found lots of success incorporating the work of other team members, like designers, marketers, and writers, into the board as well.

Our project boards generally follow a common pattern with a Backlog column for new tasks, an Up Next column for prioritized tasks that should be worked on next, a Doing column for tasks being worked on, a Requires Review column for tasks that require feedback, and lastly a Done column for completed tasks. Up until very recently, our team was using Trello to manage our work but we just made the switch over to Github Projects. Both tools have their ups and downs but the move was ultimately made so we can reduce duplicated efforts with a tool that’s already part of the developers’ workflow.

Now, it would be nice if these boards managed themselves but unfortunately they don’t. In order to keep up our up to date, a group of folks from the team meet weekly to review our progress, groom our backlog, prioritize our work for the next week, and then publish a report to keep our stakeholders up to date. As the week goes on, team members pick up cards on the board, assign themselves to them, do the work, and share their progress when necessary.

Status updates

For status updates we use Geekbot for sending an automated Slack prompt that asks people a series of questions. We started out with infrequent weekly prompts early in the project and then switched to daily prompts as we drew closer to our deadline. We crafted the questions in the prompt so we could learn how people were feeling about the general status of the project, what they had completed since the last update, what they would work on next, and if they had any roadblocks. The updates were helpful for everyone to see what they were all working on and to help each other out when roadblocks popped up.

Feedback

Feedback comes in many forms and is invaluable for baking quality into your product. On the design side, we have regular design critiques where we bring designers and developers, from across the company, together to review work and offer feedback. Work is also posted in multiple formats for people to review and share their thoughts on their own time.

On the development side, we do code reviews to not only make sure we’re writing good code but also to test the work that’s being implement. Both designers and developers participate in the review process so we can work collaboratively to highlight any issues and also overcome any unexpected issues.

Collecting internal feedback is great but can still leave you open to making mistakes. This is why it’s so important to include your customers in your product development cycle. We do this by running usability studies at various stages of our projects. At the beginning, they’re helpful for understanding your customers expectations, problems, and view points. Then as the product evolves, it’s helpful seeing customers using what you created first hand to uncover any problems they’re running into. Usually five to eight sessions are enough to give you what you need.

Retrospectives

Even though there is lots of communication throughout a project, I believe that it’s important to take deliberate time to look back and see how things are going. I say deliberate because there have been so many times where things felt like they were going well until we took the time to dig in and all of a sudden we found all kinds of issues. Those were hard discussions for a number of reasons but we all felt better for having them and improved as a team.

There are a number of retrospective activities out there but we keep going back to the tried and true “Start, Stop, Continue” because it just works. We use Funretro.io as a digital board for people to add their “stickies” on their own time and then we meet as a group to discuss what everyone added. As people read through their stickies, we ask them to group them with similar topics on the board. Lastly, we vote on the topics we would like to cover and then pick the top 3 we want to action on before our next retro.

So how different is it working with a remote team

The majority of things I shared in this post aren’t unique to remote teams. As a matter of fact, I have been working with most of these methods, and tools, longer than I have been working at a remote company. There might be some minor differences in how you do things but at the end of the day, whether you work in an office or with a remote team, the key to any successful project is good communication.

Building my first progressive web app

Over the Christmas holidays I came across some articles about Progressive Web Apps (PWAs) and then a couple months later, l had a brand new site. I wasn’t planning on building one at the time but the articles had such an effect on me that I felt compelled to get to work. I learned that a PWA is a websites that has the same capabilities like a mobile native app with things like push notifications and device hardware access. It reminded me of an idea I had heard about making sites that weren’t sites but apps and made me wonder what that might like look on my own site. Then, one thing lead to another and I was sketching ideas for a new personal site.

From a programming perspective, I decided to go with React so I could practice my coding skills and also because I really enjoyed working with it in the past. This was one of the most ambitious coding projects that I had ever conceived for myself so I thought I’d also leverage a framework to do some of the heavy lifting for me. After doing my research, a couple promising options presented themselves and decided to move forward with React PWA — a boilerplate for creating PWAs with React. It offered a number of prepackaged features like offline support, code splitting, server side rendering, and React Router which would all let me get right to work.

First signs of trouble

At first, things were going really well and I felt like I was making great progress building out my local version but then things get really hairy when I was done and went to deploy it to my hosting provider. Since this was the first React App I had ever built from scratch, I wasn’t aware of any of the hosting requirements for it to work. At that time, I thought React was a JavaScript language and that it would work on any old web host but I very quickly found out I was wrong. Even though I had it setup to run locally, I wasn’t aware you needed Node.js server to run your application and so I had to find a host that could support that. Thankfully I found Netlify which offers some really amazing services like free hosting for personal sites, free HTTPs certificates, and my favourite, continuous deployments with GitHub.

With the host figured out, I was ready to deploy the first version of my site and start the celebrations but then I ran into another issue. Something was configured different on my local machine compared to my host — React PWA boilerplate didn’t like and so all I got were errors. I banged my head against the wall trying to figure it out until I finally gave in and hired someone on Fiverr to figure it out. The process was super simple and within a day of posting my job request, I had my issue resolved and my site running only to find out that I had another problem with how something was rendering thanks to the framework I was using. At this point, I had put in a significant amount of my free time into trying to figure out the hosting issue and so running into this took the wind right out of my sails. I decided to put things on pause for a bit and hoped that maybe something would come to me in the future.

Try and try again

A couple days passed and I was still feeling a little deflated from things not working out but was determined to move forward so I picked a new framework and tried again. I had been reading about Gatsby and thought that, that would serve my needs just fine. Having learned my lesson from before, the first thing I did was deploy a copy of Gatsby up to my hosting environment to see if it worked. With things looking good, I proceeded to take pieces of code, from my old code base, and transfer them over to my new site while continually deploying to my server. I started with the easy stuff and was excited to be making progress again but then ran into another problem when I tried to connect my WordPress blog to it. I kept getting one of those weird developer errors that didn’t make any sense and was dead in the water for a second time. I remember being disappointed and once again thought that some time away would hopefully reveal a path forward.

Third time’s a charm

After about a week, I had pretty much given up on making Gatsby work but felt compelled to get this project out the door because I was really proud of what I had created and wanted people to see it. This time I said forget the frameworks and started with a blank slate. Even though I was starting from scratch I had a working version that I had built locally with React PWA so I copied it over piece by piece and deployed it continuously to make sure everything was ok. Slow and steady, I finally got it all up and running on Netlify and connected my domain to it. I must have visited it a million times in the first week marvelling at what I had accomplished, noticing every little detail that I needed to fix, and shipped updates immediately.

Onwards and upwards

While I was busy building, and then fixing, my site I spent very little time writing. All that work really made me appreciate the WordPress theme I used before because it meant I spent zero time working on making it work and all my time writing. That being said, I have thoroughly enjoyed working on this site and continue to dabble on it as a test bed for trying new things like working with CSS Grid, animations, and service workers. I have made a bunch of updates since I launching the site but still have so much I want to do with it because my full vision has not been realized yet — little by little, I’ll get there!

From zero to one

After its first year in business, the Shopify Plus team learned a lot about their customers and who they were. It was clear their website didn’t accurately represent them anymore and so they decided it was time for a redesign. I was picked to lead the project and saw it from concept through to delivery — even coding parts of it.

We collected both qualitative and quantitative data to measure the effectiveness of the new design after it launched. It was successful not only based on how our customers perceived us but also by the number, and quality, of leads we were collecting. When it was all said and done, I wrote an article for Invision’s blog and shared the following design tools and process I used for the redesign:

Project brief

The very first thing I did when I started this project was to write up a project brief so I could keep everyone on the same page. I made sure to update it as we went along so that it always reflected the latest state of the project. It served as a great tool to communicate the goals of the project and onboard new team members onto the project.

These were some questions I included:

  • What are we trying to achieve?
  • How will we know the project is a success?
  • What do we need to do?
  • Why do we need to do it?
  • What are the must-haves?
  • Who are we doing this for?
  • How will they know about it?
  • Who’s on the project team?
  • What’s our deadline?

Stakeholder interviews

At the very beginning of the project, I kicked things off by running 1-on-1 interviews with members of our executive team and some of our top customers to learn from them what Shopify Plus is all about. I recorded each interview so I could pay full attention and took notes later when I listened the recordings over again. The notes allowed me to capture the key points which highlighted patterns and themes which we used to guide the direction of the new design.

I remember how inspired I felt after my first round of executive interviews — it was like I knew everything there was to know about the product. After that, writing vision and mission statements, brand guidelines, project briefs, and the websites copy came easy. As we progressed through the project, we would continue to check in with different customers and test the design to make improvements.

Competitive analysis

Early in the project, I reviewed the the competitive landscape to see how other people were speaking to our customers. I started with a list of direct competitors, then moved on to businesses that serve our target audience outside the ecommerce field.

The analysis highlighted common patterns appearing on multiple sites which echoed what our customers were looking for in our interviews with them. One of the common patterns we noticed was how busy, and confusing, a lot of these enterprise sites were. We honed in on that fact and used clarity and simplicity as a key principle to set ourselves apart.

Information architecture

Early in my web design career, I learned to draw information architecture (IA) diagrams as a way to document an existing system or express a new one. Early in the process, I drew a diagram of our old site that showed the hierarchy and relationships of screens.

Through that process, I evaluated the content and the traffic to the screens to find lots of orphans or deeply linked pages. I used the information I gathered to create a new IA and suggested changes. The diagram was a great tool that helped us have important discussions around defining the scope of our project releases and acted like a checklist as we wrote, designed, and coded all the pages.

information-architecture
Our information architecture diagram

Wireframes

I used wireframes throughout the redesign to help think through problems, and get approval for ideas. Some were sketched on napkins and shared face-to-face, and others were drafted and shared online.

An unexpected bonus for me was discovering how much easier it was to write copy directly in the wireframe rather than in a document. Seeing the copy visually was helpful for me to create a through line for each screen and across the pages for a nice coherent narrative.

wireframes (1)
High-fidelity wireframe used for the redesign

Inspiration boards

My inspiration boards allowed me to share my vision of what our new brand might look like and helped me get early buy in from the team.

I developed my concepts in Adobe Illustrator by combining words, images, and colour swatches into something that looked like an ad. The same elements appeared in all the boards I created, but their placement and appearance changed from board to board.

mood-board
The final inspiration board we picked for the site
other-mood-boards
Some of the other inspiration boards we created to explore alternative directions.

Prototypes

Back in my consulting days, I found people didn’t quite pay attention to wireframes or web designs if they were printed on paper. Showing them a design on a screen was better but the thing I have found that worked the best, to captured people’s attention, was sharing something interactive.

I started developing interactive mockups early in the wireframing process. We shared them with our customers and the team throughout the project until most of the site was coded. We had people speak out their thoughts as they browsed through the prototype. Their feedback highlighted problem areas and validated design choices which helped us shape the end result into something successful for both us and our customers.

invision-mock
InVision project page for Shopify Plus

End result

Up and to the right chart

Up and to the right

In early 2016, I was presented with an opportunity to run the operations for Shopify’s 50 person Partners team — made up of five sub teams. As a designer the role was very different than what I was used to and it came with a lot of interesting challenges. The responsibilities of this role included product management, regular reporting, and organizational management. One of my favourite challenges to work on in my time there was giving people the information they needed to make the best decisions at the right time.

Doing my research

Thanks to my design background, I had a number of tools to lean on which helped me guide the team to achieve successful results for four straight quarters. Kicking things off, I approached this new role like I would any other project and put on my researcher hat. While I made sure business went on as usual, I also spent a good part of my early days in the role analyzing tons of data, interviewing all kinds of stakeholders, and running a number of facilitated sessions.

Creating a vision

Working closely with each team, we created a shared vision of what we wanted to achieve. From there, I helped them connect their goals to our business metrics and then used the data designer in me to create dashboards for reporting back the progress. The work didn’t end there though, even though we had all these fancy dashboards and beautifully articulated goals things didn’t feel like they progressed as we initially had planned.

Testing and refining

The problem I noticed after our first quarterly check-in was that people were working on completely different things than we agreed on at the beginning of the quarter. That’s when I learned one of the most valuable lessons while in this role: if you don’t check your goals regularly then it’s easy for other priorities to creep in and distract the team from achieving their goals.

After that initial meeting, I set up regular check-ins with the team leads to review our progress, discuss roadblocks, and also collect feedback on our process so we could continue to refine it. These efforts very quickly paid off as we were able to move more nimbly in prioritizing new efforts and making adjustments to our plans where necessary.

Build, measure, learn

One of the first projects I worked on at Automattic was a small A/B test for the first screen of the WordPress.com sign-up process. As the first screen in our funnel, it served a critical role, but it hadn’t been tested in quite some time. I tweaked the visual design and completely changed the copy for my first attempt at improving conversions on that screen. Setting it up and tracking the results felt really easy compared anything I had done before. I felt empowered and simultaneously discovered how projects work at Automattic.

That first test was a success and was implemented as the default for all our visitors. It remained that way for a little under a year as our priorities shifted to higher impact areas of our funnel. Little bits of feedback trickled in over time that kept me thinking about many possible improvements but it wasn’t until recently that I decided to take action.

The problem

One of the most common complaints I heard about our screen was that the terminology was confusing. We asked people what type of site they wanted and presented them with four options: a blog, a website, a portfolio, and an online store. These were some of the questions that kept coming up: what’s the difference between a website and a blog? can I change my mind later? two of these options sound good, what if I want both?

Our mistake was assuming people understood what to us seemed like simple concepts but for them were actually foreign. This made a lot of sense when I look back at all the polls I ran and conversations I had with our customers over the last year. I learned that our customers are not like us and that the majority of people signing up at WordPress.com had little to no experience building sites or using WordPress.

screen-shot-2018-01-02-at-10-38-08-am.jpg
Graphic courtesy John Maeda and Rebrand Cities.

A new hope

Armed with this awareness about our customers, I reworked this screen to simulate a first meeting between a design agency and their client. The screen would ask them a series of questions to understand their needs, and then our tools would build a site that addressed their goals.

My biggest concerns with this approach were 1) not scaring people with too many questions so early in the signup flow, and 2) knowing which questions to ask. I quickly developed high level strategies to tackle both these problems. Designing with a mobile context in mind would ensure that I made it quick and easy for people to answer our questions and using a tool like Hotjar, I could quickly run a series of polls to validate and modify my questions.

Learning our customer’s vocabulary

One of my questions could be answered in a number of ways and therefore required an open text field to collect people’s responses. Of course, typing on a phone sucks, so I wanted to make the field as painless as possible. An autocomplete feature made the most sense as it offered people suggestions while they typed their answer. We could not only save people time from writing out the full word but we would also get cleaner data too. To pull this off, I needed a collection of commonly used terms to power the suggestions.

I created a new poll that included the question as it would appear in my design and quickly noticed the answers weren’t what I expected. Thanks to HotJar I was able to turn off the poll, rephrase the question, and launch the poll again within a matter of minutes — all without writing a single line of code. This time around I was happy with the data coming in and let the poll run until we had enough responses. After a couple days we had a lot of raw data to comb through so I used a spreadsheet and a PHP script to clean it up and help us make sense of it all.

php2.jpg

Build, measure, learn

Another question asked people to pick their answer(s) from a handful of choices. I drafted a list of options based on data we had collected in the past and ran them in a poll along with an “other” field that allowed people to type their answer if couldn’t find what they were looking for in my list.

The first iteration of this poll was dominated by responses entered in the “other” field. Some of them were variations of what I had in my list but there were also lots of common responses that were not represented in my list at all. I used this data to update my list and then published another version of the poll. There were some decent improvements in the results but it was still not quite there yet so I rinsed and repeated. This time I got it right because the majority of the responses were selected from my list rather than entered into the “other” field. In addition to that, most of the people that did fill out the “other” field either flipped me the bird or just said hi.

Progress on multiple fronts

I wasted no time while these polls were running and started building the form right after the design was sketched out. The code was built in small chunks which made it easy to review and kept the momentum of the project going with visible progress. As the data came in, I added it to my work in progress until I had enough of a prototype to run some usability tests. The tests revealed some really minor tweaks that needed to happen but mostly confirmed that we were on the right path. Then, shortly after I completed my last poll, I had everything I needed to run my A/B test at scale and see how this new design stacked up against the old one.

The results

It didn’t take long for us to reach statistical significance but I still let the test run for at least seven days to account for any seasonality during the week. The results indicated that this design profoundly influenced our business metrics. Like it says in NEA’s Future of Design report:

Ultimately everything a design team does has consequences (both good and bad) for a business. The more you know about the business and its goals/intentions, the better a design team is positioned to deliver good experiences and results for the business.

— Tim Riley: Senior Director of Digital Experience at Warby Parker

This post also appeared on Automattic’s Design blog.

Crossing the digital divide

Taxiing home after my first Automattic Grand Meetup in Whistler, I was happy to get to know my driver Mohamed. He has been running his family business for over 20 years and also drives a taxi to help make ends meet. Like me, he has three kids and we spoke about them at length. We also spoke about work and got excited when he heard that I’m designer at WordPress.com. He pulled out his phone right away and told me that he downloaded the app with the intention of starting a new site but didn’t have the time to get to it.

We’re often encouraged to work with people that use our products because it helps us see our product through their eyes. This seemed like the perfect opportunity so I offered to build Mohamed his site on WordPress.com. He was really grateful and humbled me with his response. We exchanged contact info and agreed to meet after I recovered from my trip.

Going into this I thought I was just going to build a site but while prepping for our first meeting I realized it could be a lot more. I had been working on a new post-signup checklist to help new site owners get their site ready to launch. Most of the details were figured out except the checklist tasks which I was still in the process of finalizing. My epiphany led to me drafting up an agenda which basically included latest version of my checklist — what a great way to test it out I thought.

We met at Starbucks, my usual out-of-office meeting place, and he brought a friend along. Mohamed confessed he wasn’t very good with computers and needed someone to help him out with the “technical stuff”. His friend was also a small business owner and managed multiple businesses so he was very interested to learn more about WordPress. After some personal introductions, I asked Mohamed to tell me more about his business and what he was hoping his website would do for him. Within a couple of minutes, I had enough information to start working through my checklist.

Through multiple experiences of my own and hearing about others, I would have to say generating content is the hardest part of building a site. We can help you create a site in seconds and give it a professional look, but without the content, it isn’t very useful. I tried to be resourceful and started by asking if they had any existing social accounts or websites. My thinking was that I could maybe get a logo, a colour pallet, some images perhaps, and if I am really lucky some writing too — but no, they didn’t have any accounts. This reminded me of a poll we once ran where almost 60% of 4400 people signing up said they didn’t have an existing online presence. It was also something really important to keep in mind so we can think of other ways to help people generate content for their sites.

As I went down my list, I would explain why the task was important and how I was executing it. They couldn’t believe how quick it was coming together and they loved the design even though they didn’t pick one. In our time together, we built out the entire architecture for their site, personalized it, set up multiple ways for people to contact them, and added some copy to get them going. The site wasn’t 100% complete but Mohamed was ecstatic that his business was no longer limited to his geographical location. He could now do business with people all over the world. I left them with some homework for next time we meet where we hope to wrap things up.

I walked away from this experience feeling really energized and excited about what I had learned. For one, it was great to see the impact our work has on people’s lives. I also learned a lot about our product and the people using it. My biggest take away was witnessing someone that hadn’t fully crossed the digital divide and how that affects his business. I noticed what a big role Mohamed’s mobile devices has on his work and ability to provide for his family. It’s so clear to me why a mobile-first, or even mobile-only, policy is so important when I see people like Mohamed turning to their phones over desktop computers to connect with the world around them. I highly recommend that people working on products seek out opportunities like these.

This post also appeared in Automattic’s product design blog.

Recruiting participants for remote user tests

I recently attended my first remote design sprint at Automattic. It ran over a two week period and  was attended by fellow designers, developers, data scientists, happiness engineers, and marketers. Together we created two prototypes and tested them out with real customers. I found the experience to be both rewarding and inspiring. We came up with a lot of great ideas and validated them too. It was also really nice collaborating with a number of people across the company that I don’t usually get to work with.

One of my major contributions to the sprint was recruiting real customers for the prototype test. The job was a whole lot easier thanks to an internal post written by my designer colleague Mike Shelton which also took part in the design sprint. He shared his experience recruiting customers from a previous test and also included the templates he created for people like myself to use. With that post, I had everything I needed once we determined exactly which type of customers we wanted to test.

Finding participants

Before the sprint even started, I had been running tests to see how many people I could recruit with a poll on our signup form at WordPress.com. Fortunately my tests worked out well so I knew I could count on running another one to collect participants for our test. I created a new poll using HotJar and added a little prompt at the end of the poll that asked participants to sign up for other customer research initiatives. This is what it looked like:

poll.png

Screening participants

Since we wanted to talk to a specific group of people, the link at the end of the poll lead to a Google form which asked visitors to complete a series of questions. We added questions that would help us qualify the people signing up for this test and other tests in the future. Since this was a remote prototype test, one of the key questions was if people were comfortable talking over a video conference.

screener-shot.png

View all the questions

Sending out invitations

I wanted the email to look like it was coming from a real person so I sent it with Inbox using my full name and company email address a week before we have our test date scheduled. It worked because some people replied back which was great — I was able to do some additional research for another project. I also made sure to add the qualified email addresses to the bcc field so that the recipients couldn’t see any other people being emailed.

These were the key points I included in the email:

  • Introduce myself and tell people where I got their email
  • Explain what we’re testing along with all the logistics
  • Identified what the recipient gets for participating — we decided to offer a $50 Amazon gift card because we felt like we were asking a lot for someone to take an hour out of their busy schedules
  • How to sign up for the test.
email.png

Coordinating schedules

Getting people across multiple time zones to agree on a date and time over email can be confusing and time consuming. There’s lots of back and forth involved and it’s easy to make mistakes.

That’s why we use Calendly to manage our calendar availability and bookings. It’s easy to setup, offers a number of customizable options, and displays times relative to the timezone of the viewer . You can also automate and personalize event reminders so that you have one less thing to worry about.

calendly.png

Sending a reminder

People’s lives are busy so it can be understandable if someone forgets something they committed to a week ago. Sending a reminder a couple hours before the test is a good way to avoid that and to add any last bits of logistical information required for the test. In my case, I made sure to include:

  • the time,
  • a link to download the software used for video conferencing — we use Zoom.us for it’s video quality, number of participants, screen sharing, and recording capabilities, and
  • a link to join your video conference.

Here’s what my email looked like:

reminder.png

Following up after the test

Lastly, we ended things off with one last email thanking our participants for their time. We included instructions for them to claim their $50 Amazon gift card and also took the opportunity to collect some final feedback about the test so we could make improvements for future participants.

follow-up.png

I hope you enjoyed reading this post and remember it next time you’re planning on recruiting participants for your own user test or interview. If you do, let me know how it works out for you by reaching out on Twitter.

This post also appeared in Automattic’s product design blog.