TORCHx’s mobile CRM was a bloated, fully-featured responsive website that was difficult to use and didn’t support the needs of real estate agents who are on-the-go. We set out to improve it so busy agents could quickly switch contexts and respond to customers when they’re running between showings or at an open house.
This project lasted about 8 months, since it was running alongside 3 other initiatives and we had to hit the pause button a couple of times. I worked closely with a Product Manager and a Lead Software Engineer throughout the entire process.
In early 2017, our engineering team was fully devoted to resolving our product's technical & infrastructure problems, which was the #1 priority for the business. This led us to explore other ways we could provide value to our customers, outside of those constraints and our 2-week release cycle.
Our product's mobile capabilities were of increasing concern, since that was coming up more often on sales & customer support calls, and competitors were releasing new mobile apps. I had also been itching to improve our mobile website's interface for a really long time: despite real estate agents' need for on-the-go tools, our product only saw 10% mobile traffic, likely due to its cluttered interface and poor usability:
We wanted to make sure we weren't acting purely on competitive pressure, so we conducted research with our customers to understand if there was deeper value to improving our mobile website. We spoke with 10 real estate agents over the course of a week - here's what we learned:
They spend at least 50% of their time away from their office and laptop
They want to review client information before responding to messages so they're well informed of their needs
They're using other mobile apps and responsive sites to supplement the gaps in our mobile website
They encountered numerous usability problems
They weren’t tracking conversations that happen on-the-go because manual entry was inconvenient in our mobile site
They often forgot their password, and we didn't store their logged-in credentials for more than a few days due to outdated security reasons
Other than solving usability issues, we felt that an improved mobile website could really help our customers manage their client relationships on-the-go. This aligned with the top unmet need from our quantitative survey, "minimize the number of leads that fall through the cracks," so we felt confident in moving forward with this project. Additionally, one of our lead engineers, who had experience developing mobile applications and websites, had some time that he could dedicate towards building a lean, functional prototype we could use to test hypotheses.
Now the question was: what do we release?
The first step was to put something in the hands of our customers and learn from it. After conducting research, the next step was to generate ideas, align on what we wanted to test, and come up with a more focused hypothesis. I facilitated a design studio with participants from Product Management, Design, Product Marketing and Engineering to ensure everyone was bought-in, engaged, and involved in the product development process.
We generated ideas that fell into different buckets:
Notification-driven experiences that drive users to important communications or tasks
Location-based experiences that let users "snooze" tasks & messages until they return to the office, or that support on-the-go home searches when agents are out showing homes to homebuyers
Using gestures (e.g. swiping) for quick actions, like categorizing a lead, passing on a lead, or contacting a lead through a specific medium (...yes, this was the token Tinder-inspired idea)
Leveraging OS functionality like Spotlight (iOS) to help agents find leads indexed from our CRM to their phone, or rich caller ID (Android) to provide agents with detailed info on leads when they call them
We voted and aligned on testing a notification-driven experience that would alert agents of new leads and incoming messages. The team went through a few story-mapping exercises to find the "thinnest" slice for our first test. That way we could more quickly build a testable prototype, and really focus the test on a risky hypothesis.
This helped us define our risky hypothesis:
With this in mind, I could move forward with design iterations. We decided to build a new version of the mobile website instead of a mobile app for a few reasons:
Building a mobile website was cheaper and more time-effective from an engineering perspective
We weren't testing the need for a mobile app; rather, we were testing the need for an optimized mobile experience focused on messaging features
Releasing the testable prototype would be much easier through our own infrastructure, versus relying the Android or iOS app stores for approval and distribution
After a few usability tests with customers and design reviews with the product team, we had run through several design iterations of the optimized mobile web experience:
This really helped us hone in on what features we needed to build into the new experience so we could properly test our hypothesis. The final process flow would notify users of a new message via SMS, automatically log them into the mobile CRM with the tap of a link, and drop them directly into the conversation with their lead where they can easily respond.
Additionally, users would have access to a full list of conversations, and a details page with information about the person they're messaging. Thus, the testable experience really boils down to three screens, plus the initial SMS notifications that triggers it:
At this point I was working closely with our lead engineer to implement these designs, make small tweaks to the experience, and resolve gaps that we hadn't fully accounted for (e.g. what happens if a message fails to send?). I also created a research & testing plan with input from Product, Engineering, and Data Analytics teammates so we could track progress during the test. We decided to start with an alpha test so we could iron out any final technical or UI bugs before rolling out the experience to a larger population for a beta test.
We recruited 5 clients who were open to testing the updated experience. We "turned on" the new experience for these 5 clients, and waited. We monitored usage data and text messages sent to these clients through our platform. We weren't seeing any usage or text messages come through after a week, so I decided to reach out to these clients.
As it turns out, in that 1-week timeframe they hadn't received text messages from potential homebuyers. They had been texting leads that they were already nurturing for some time, but those conversations were taking place in their native SMS applications, not through our platform. The test relied on our clients getting new leads, and we couldn't guarantee that would happen within a reasonable timeframe for such a small population of users (5 clients). We waited another week, and still nothing happened.
At this point we had to decide whether we'd continue waiting, or expand the test to more clients who were receiving text messages from potential homebuyers more regularly, or hit the pause button. We ultimately decided to hit the pause button because our lead engineer, who was the only technical resource on this project, had to switch gears and dedicatea 100% of his time to technical & infrastructure problems.
There were a couple of key takeaways at the conclusion of this project:
Test design is crucial to successfully testing a hypothesis. Sounds quite obvious, but we didn't foresee how heavily our test relied on external variables (like clients receiving a text message), and thus we didn't bake in enough time for the test to run properly.
Getting quickly to a testable prototype helps mitigate risk and development costs. While the time we spent ideating, story-mapping, and iterating on design concepts was valuable, we should have built a testable prototype sooner, so we could more quickly learn if we had to shift expectations around how our test would work.
Spinning up a lean test with just one resource is very risky. The intent was pure and good: despite limited engineering resources, we wanted to find ways to continuously provide value to our customers. While we did learn about our customers throughout the discovery process, the risk came back to haunt us as our only resource got pulled into a more time-sensitive, business-critical project.
I was disheartened, but this pushed us to keep looking for ways to deliver value to our customers through experiences that didn't rely on the limited engineering resources we had at hand. This was the silver lining, since it led us to redesign our onboarding process which ultimately had an incredibly positive impact on our customers and the business. You can learn more about that project here.