In analytics, you have to crawl before you run. Event analytic tools provide instant context, but to enable flexible and deep analysis data teams will eventually want a horizontal BI platform.
Data is an important tool in the product manager’s toolbox, and it’s often said the best product decisions come from a mix of data and product intuition. However, product intuition actually is formed by looking at data – but that includes all kinds of data, not just the quantitative metrics we’ve become laser focused on. In addition to pure product metrics, there are lots of data points that sharpen our intuition over time, such as hearing customer feedback, reading support tickets, looking at sales trends, etc. The best product managers are the ones who have a grasp not only on what drives success metrics for their product, but also understand the business and customer journey end-to-end.
When it comes to getting access to and understanding data, there are typically two options that arise in product orgs: a generalized business intelligence (BI) platform meant for serving all types of users across the company, or a verticalized event analytics tool meant for specifically analyzing product data.
When looking into which path to take, there are three considerations: speed of analysis, analysis depth and breadth, and consistency and trust. Let’s dive into each of those areas.
Speed of analysis
Winner: Event analytics hands down. With simple defaults, event analytics gets you up and running immediately doing standard product analyses.
When making product decisions, oftentimes speed is the most important thing. If it’s going to take a week to get an analysis back, we’re most likely going to make the decision without it (or risk blocking the team and progress!).
Event analytics platforms typically have an edge here for two reasons: They are quick to set up and implement, and can be managed by the product and engineering team without involving other groups (thus avoiding some of those cross-functional prioritization conversations) The user interfaces are specialized for product analyses ( event/user counts, cohorts, funnels, etc). This makes it relatively fast for users to learn and use without requiring knowledge of things like SQL.
On the other hand, BI platforms can take time to implement, may require data teams to get involved to perform data ingest and transformation tasks, and they have a more generalized interface that can be trickier to learn. But, read further to see why a BI platform can be a great choice for more complete analytics.
Depth and breadth
Winner: BI. With horizontal platforms, you can answer questions far beyond just events. How does marketing influence product behavior that influences support tickets that influences purchases.
One of the most important attributes of product analysis is the ability to go both deep and wide.
With both event analytics and BI platforms, it’s relatively easy to get high level metrics – e.g. user signups, WAU, product usage, etc. But nuanced analysis requires you to break out of the black box. Insight comes from diving in.
For example, let’s say user signups look great – it’s up and to the right! However, conversion to paid users is low. Why?
With an event analytics tool you may be able to dive in a little – build an event funnel and see if there’s a step where users are dropping off. From there, the decision may be to spend a few sprints optimizing that part of the workflow., after doing that, however, conversion is still not improving. What gives?
With a BI platform, you have access to all your data, so you’re able to understand the problem from all angles. Let’s say you identify the group of users who are dropping off and are able to pull back other sources of data related to them – such as support tickets. Not only do you have the number of support tickets filed, but you can actually drill into the ticket text and read what’s going on. You learn that the issue isn’t a confusing product workflow, but there’s actually a key feature missing that’s preventing users from upgrading to the paid version. Now armed with that information, you spend the team’s time building out that feature vs optimizing the upgrade flow.
This is where being able to go both wide and deep is extremely impactful and can change the outcome of product decisions. When you are constrained to one source of data, every decision will start to look like optimizations for that one data stream vs seeing the bigger picture.
Consistency and trust
Winner: BI. BI can enable an organization to define canonical metrics and establish a single-source of truth.
Lastly, consistency is an often overlooked but important attribute of your analytics platform.
I’ll share an example from my own experience. At a previous company, we were building out a product led growth motion for a new product line. There was one key metric that was critical to the success of the whole operation and was reported up to leadership and the board. That metric was PQAs, or product qualified accounts. Each department was responsible for some part of improving this metric – marketing needed to generate enough user signups, product was responsible for improving activation and retention, and sales needed to convert PQAs to paying customers.
The challenge was getting to a consistent definition that each team could track and measure. We first built the metric out in the event analytics tool, but that didn’t include enough attributes that the sales team needed. It got exported to a spreadsheet and combined with salesforce data. Now something didn’t add up between what marketing was reporting and sales was reporting. We had to bring over a data analyst just to help us untangle the mess! And worst of all, leadership was getting 3 different answers from 3 different departments in the many different tools they use.
Often it seems faster to build analyses in departmental tools, but in the long run it can lead to confusion at best and loss of trust in the data at worst. And as product managers, our job is ensuring the success of the product – and that can mean aligning a lot of cross-functional teams to make an initiative successful rather than purely optimizing product metrics.
Investing in a BI platform that helps define and manage consistent metrics can prevent a lot of headaches down the road.
What to choose
For product teams, there are certainly tradeoffs between BI and event analytics platforms. Do you go for the speed of an event analytics product or the breadth and scalability of a BI platform?
When you are just getting started, event analytics tools provide immediate time to value. As your organization or analysis needs to get more sophisticated, you’ll need a more flexible tool.
At Omni, we’re focused on creating a new BI platform that lets you get started and answer questions quickly without the typical overhead of legacy BI products, while still scaling with you and providing consistency as your data and business needs get more complex.