Data analysis project
Omlet
Data analysis project
Use agile analysis to identify key issues never found
Contents：
＊The numbers showed in the following content are remade.
Background：
To achieve the annual goal, the company wants to increase the retention rate in a limited time. Before the project, I have done several user research and data analyses that strongly support the project.
Tools are used in the project：
1. Kibana
2. R
3. Zeppelin book
4.Google sheet
Define the key metrics：
Retention, retention, retention
After meeting with the boss, retention will be the priority because of the annual goal. However, we wanted to focus on low-hanging fruit to grow the product in a limited time.
To identify a clear direction for the project, I check the retention, and the result shows that we lost the most users in the early period.
Time
There is something we never tried before.
Before the project, we knew what users did on the first day to lead to better retention and why they came back.
But we didn't know why users left after the first day. So I chose to focus on looking into the reason for losing users in Day1.
Step1：Data exploration & Hypothesis
Data exploration
I reference the research result that if users had completed the main feature, e.g., start a stream, join a game, etc., their retention is better than those who didn't complete anything.
I broke down the behavior of users who didn't complete any main feature on the first day. After that, the result showed that 80% users failed in authorizing the permission, which is the necessary process to start the tools for users to complete the main feature.
Hypothesis
According to the result of the exploration, I assumed that the main reason for losing users is that users encountered difficulty when trying to authorize, which stopped them from using Omlet again.
H1：The failure of authorizing decreases the early retention.
Step2：Use small sample to verify the hypothesis
Experiment design
Because of the limitation of Kibana, I couldn't download the entire data. So, I chose to experiment with a small sample but control the confidence interval near 1%. I separated the sample into three gropus：
Group A：Users had completed any main feature on Day0.
Group B：User didn't complete any main feature but did authorize the permission on day0.
Group C：User didn't complete any main feature and authorize the permission on day0.
After that, I picked two metrics to evaluate the performance of retention：
- Each group's day1 - day7retention
- How many days users return in the first 7 days.
The result of the experiment - Day1 - Day 7 retention
I calculated the data with R and visualized it with the Google sheet. There is no surprise that Group A got the best performance than Group B & C. Even though Group B's and C's users didn't complete the main feature, there is a difference between them. The result confirmed that failure of authorizing caused lower retention.
The result of the experiment - How many days users return in the first 7 days.
I used the t-test to compare the number of days the user returned in the first 7 days.
Surprisingly, I got a significant result：
Group A > Group B：P value < 0.00
Group A > Group C：P value < 0.00
Group B > Group C：P value < 0.00
According to the experiment result, I confirmed the hypothesis is correct.
The next step is to expand the verification.
Step3：Expand the confirmation and point the problem
Expand the confirmation
I worked with a data engineer to get the last six months data as observations. After comparing Day1 and Day7 retention of the three groups, The data confirmed that this problem existed in the platform for a long time.
.png)
In addition, a high portion of users who didn't complete the main feature on day0 didn't authorize the permission.
Authorization is the difficulty that users encountered in the process of doing the main feature.
Point out the problem
To sum up, I find out the authorization problem, which decreased the early retention. Also, there is a significant portion of users encountered the issue while using Omlet.
After discussing with QA engineers, we assume users can't find where to authorize in the operating system. Checking retention of the different versions of the operating system confirmed our assumption. Users hardly find the place for authorization if they used a specific version.
In final, the result supported the PM and designer to increase the ratio of authorization and early retention.