Feb 23, 2018
You’ve already come to the conclusion that experimentation is the best way to create the optimal user experience. You know that a better user experience can improve engagement, retention and revenue and thus helps you grow your product and organizational KPIs faster. This article will answer the question of whether you should build or buy software.
Experimentation testing gives you cold, hard data to answer any kind of question about how to improve your product or service. By presenting your customers with different versions of your product, you can measure the cause-and-effect relationship between any changes and the impact on KPIs.
While A/B testing is simple in concept, it is hard to execute well, especially given the fragmented customer journey across digital and physical channels today. Now that you’ve made the right decision that experimentation is a priority for your business, you need to choose a framework that will enable you to rapidly run experiments and accurately analyze them.
Should you build or should you buy?
Here are the key considerations that you should keep in mind while answering this question.
Your team is really good at building whatever product they are building (probably better than our engineers at that!). The basis of all SaaS products is letting the experts build what they build best. In a world as complicated as product development, we believe it’s important to have a system built by a team of experts (who have scaled a product for the likes of Viber, Vevo, Strava, and Wall Street Journal).
One big problem with building in-house software is that it’s often not “democratic,” or accessible by the people who need to use it to achieve their KPIs. There are multiple teams that have an ongoing interest in using experimentation to meet their goals (product, marketing, engineering, and data to name a few), and the usage of the tool will vary amongst these different stakeholders. Satisfying the requirements of different users can be very difficult, something that requires a deep understanding of each of these personas and their goals.
The cost of building your own platform isn’t just limited to the upfront development; you also have to consider the ongoing maintenance. Studies show that 75% of the total cost of software can be attributed to maintenance. Around a quarter of that goes to fixing bugs once the software is deployed, another quarter goes to adapting the code to a changing business environment, and the remaining half goes to enhancing the feature set for future iterations. You incur all of those future costs when you ship a “light-weight” custom A/B testing framework, because the implicit expectation is that you will fix, adapt and evolve the framework as the business grows.
If you do have the time and resources to build a successful experimentation framework, it’s important to think high-level and long-term about the requirements that you will have down road as you develop a robust culture of experimentation. We have found that in-house solutions are often built for a specific use case and are ultimately limiting as the experimentation roadmap develops. Without years of knowledge built up through practice and observation, it’s hard to come up with an end product that will fit nearly all possible edge cases.
When buying an existing solution, integration and implementation is seamless and typically takes less than a few weeks. Building an experimentation solution, however, can take months– even years. And while you’re building in-house, your competitors are already experimenting, iterating and growing.
For more detail on the topic, download our free whitepaper.
Mobile product management is taking the PM world by storm. As more companies make the transition to mobile, PMs everywhere are expanding their knowledge to encompass a new set of tools that can be applied directly to mobile products. Although...Read More
We’re letting the app stores define the way we build and release apps.It’s a mistake that every team has made at some time and still one that’s plaguing the majority of mobile teams today. And we know exactly who’s to...Read More
I’ve been a personal fan of Etsy for a long time (half the art in my house was found while lying in bed with the Etsy app on my iPad). I’ve always been impressed with how they’ve managed to develop...Read More