Revenue A/B Tests

In this guide, we’ll walk you through the process of using the Telemetry SDK to monitor an A/B test where you compare the revenue impact of a new feature between a test group and a control group. By the end of this guide, you’ll be able to log, analyze, and compare revenue data for your A/B test to determine the effectiveness of the new feature.

Prerequisites

• A valid API key for Telemetry

• Basic understanding of JavaScript and Node.js

• An A/B test setup with a defined test and control group

Step 1: Install Telemetry SDK

First, you need to install the Telemetry SDK in your project. If you haven’t done so already, run the following command:

npm install telemetry-sh

Step 2: Initialize Telemetry

After installing the SDK, import and initialize Telemetry in your project. Replace YOUR_API_KEY with your actual Telemetry API key.

import telemetry from "telemetry-sh";

telemetry.init("YOUR_API_KEY");

Step 3: Log Revenue Data for Test and Control Groups

To monitor the revenue generated by the test and control groups, you’ll need to log relevant data points such as group type, revenue, and timestamp. Here’s an example function to log this data:

const logRevenue = (groupType, revenue) => {
  telemetry.log("ab_test_revenue", {
    group_type: groupType, // "test" or "control"
    revenue: revenue,
    timestamp: new Date().toISOString()
  });
};

// Example usage
logRevenue("test", 150.75);    // Log revenue for the test group
logRevenue("control", 120.50); // Log revenue for the control group

Step 4: Automate Revenue Logging

You should set up your system to automatically log revenue data whenever a transaction occurs. Depending on your setup, this might be in the checkout process, a purchase event, or wherever revenue is generated. Here’s an example:

const processTransaction = (groupType, amount) => {
  // Additional logic for processing the transaction
  
  // Log the revenue data
  logRevenue(groupType, amount);
};

// Example transactions
processTransaction("test", 200.00);    // Transaction in the test group
processTransaction("control", 180.00); // Transaction in the control group

Step 5: Query and Analyze A/B Test Results

Once you’ve logged sufficient data, you can query it using Telemetry’s query API to analyze the results of your A/B test. For example, to compare the total and average revenue between the test and control groups:

const results = await telemetry.query(`
  SELECT
    group_type,
    SUM(revenue) AS total_revenue,
    AVG(revenue) AS avg_revenue,
    COUNT(*) AS transaction_count
  FROM
    ab_test_revenue
  GROUP BY
    group_type
`);

console.log(results);

Step 6: Analyze Statistical Significance

To determine if the differences in revenue between the test and control groups are statistically significant, you might want to conduct a statistical test, such as a t-test. While this can’t be directly done in Telemetry, you can export the data and analyze it using statistical tools or software like Python's scipy library.

Step 7: Explore Data with Telemetry's UI

Telemetry’s UI allows you to visualize and explore your A/B test data interactively. Visit Telemetry Dashboard and log in with your credentials to create dashboards, charts, and more based on your A/B test results.

Conclusion

By following these steps, you can effectively monitor and analyze the revenue impact of your A/B test using the Telemetry SDK. This setup helps you determine whether the new feature positively affects revenue, providing valuable insights for decision-making.