Principles. Quick Feedback

12009

Overview

When problems are broken down, if we continue to work hard without paying attention to the feedback on the results of each completed task, we lose the other half of the benefits brought by problem decomposition, which is confirming risk reduction or elimination. Only through quick feedback can we understand the quality and effectiveness of the completed work as early as possible.

Application

During the code review process, the code submitted by developers is reviewed by other developers. Reviewers provide feedback, pointing out potential errors or areas for improvement. This timely feedback helps developers identify and fix issues early, improving code quality.

In Continuous Integration/Continuous Deployment (CI/CD), each code submission triggers an automated build and testing process. If the build or test fails, developers receive immediate feedback, allowing them to fix issues as quickly as possible. This quick feedback mechanism can prevent problems from accumulating during development, reducing the cost of resolving issues.

In product development, the principle of quick feedback is often reflected in user feedback. Development teams quickly launch new features or products to the market and then collect user feedback for timely adjustments and improvements. This approach is known as the Minimum Viable Product (MVP) strategy, which helps teams validate the product's value and direction early, avoiding wasted time and resources on the wrong path.

Validate One Point at a Time

Validate one requirement hypothesis at a time. Throughout the execution of the entire experimental plan, we must maintain an open mindset and continuously optimize these experimental plans. Always remind ourselves that our goal is to validate our hypotheses; the experimental plan is merely a means to validate our hypotheses, not our goal.

For example, an e-commerce website completely overhauls its product search results page, changing the display of the product list to a waterfall layout, similar to the infinite scroll effect of Baidu's image search results page. This means that when more than one page of product results is returned, all products can form a waterfall based on certain sorting rules. Users can continuously scroll down the page, and products will keep displaying until all search results are shown.

The team rolled up their sleeves and worked hard for a long time, and this overhaul was finally ready to meet users. Cautiously, they conducted an A/B test, and the results surprised them. In the "experimental group" with the waterfall layout, the number of users with continuous browsing behavior was 40, while in the "control group," this number was 80. The click-through rate for products dropped by about 10%, the purchase rate dropped by about 22.5%, and the item collection rate dropped by about 8%. This "waterfall layout" did not achieve the expected effect; instead, it was worse.

When they first received such results, everyone on the team believed there must be a defect in the software causing this outcome. Therefore, the team conducted various investigations, such as segmenting users by browser and geographic location. They sent someone to a public library to log into the website using an old computer. The team did find some software defects, but fixing them did not change the overall results.

They then suspected that the experience varied across different browsers, so they optimized for different browsers. Each optimization performed inconsistently across different browsers. After completing all conceivable optimizations, the situation did not improve. Ultimately, the team accepted the fact that the waterfall layout made the product worse. During this process, the team changed too many things at once, making it difficult to identify any clues to find the "culprit."

In fact, if we analyze carefully, we will find that the assumption behind the "waterfall layout" feature is: showing more products to users faster will improve their shopping experience, leading to more purchases. This assumption consists of two parts: (1) the more search results displayed to users, the better the conversion rate metrics of the page; (2) the faster users see the products, the better the conversion rate metrics of the page.

In fact, we can find faster validation methods to test these two hypotheses. For the first hypothesis, displaying twice the current number of products on the existing page to see if it improves the conversion rate of the detail page. The implementation cost of this validation plan should be much lower than the original infinite waterfall design. The conclusion is that the number of users browsing the product landing page only increased slightly, and this increase was unstable, while there was no change in the product purchase conversion rate.

For the second hypothesis, it may be slightly more challenging. Because improving the product display speed again under the premise of existing performance may pose a greater technical challenge. However, we can take the opposite approach, artificially adding a delay time (e.g., 200 ms) on the page to verify whether the user purchase rate decreases. Of course, this is not a perfect experimental plan. However, as long as a small traffic test is conducted to avoid significantly impacting the overall website revenue, it can be an acceptable method for validating the hypothesis of "delay time affecting user purchasing behavior." The final result of the experiment was "no impact on the data."