Implement Component A/B Testing

1. Overview

The AB test is to make two (A/B) or more (A/B/n) versions for the application, and at the same time dimension, let visitor groups (target groups) with the same (similar) composition randomly visit these Version, collect user experience data and business data of each group, and finally analyze and evaluate the best version, and formally adopt it; in software development, product requirements are realized through a variety of technical means. A/B test experiment provides a Value approach to evaluate the impact of new features on customer behavior.

In terms of technology itself, A/B testing pays attention to providing different services for different clients. The two differences here are very important.

Different Clients

Generally, clients are classified in a certain way, such as HTTP protocol. Usually, the Header request header and Cookie are set according to user information to distinguish different clients.

Different Services

Generally refers to different versions of the application, which are different components on the Kato platform.

Kato currently supports the A/B testing practice of the HTTP protocol, which is currently the most widely used protocol.

Services need to be A/B tested, and it is necessary to distinguish whether they are internal services or external services. The A/B test feature of the internal service is provided by the ServiceMesh layer, and the external service is provided by the application gateway.


2. A/B Test for External Services

A/B testing of external services is the most commonly used scenario, because external services are services that directly face users; business programs need to inject user identification information into cookies through certain business strategies or inject into Header through mobile apps In the request header; the Kato application gateway can identify these identifiers and match the corresponding service to the user according to the policy configured by the user.

Prerequisites

  1. The two versions of test components in operation are simulated as two versions of the same business program. Refer to source code construction to directly build [the project](https:/ /github.com/Aaron-23/teststatic) (the branch master and devel are used here to distinguish different versions),
  2. Have a test domain name, and both components are bound to this domain name.

Steps

Method 1: Identify the client through the Header request header

Add the following two HTTP access policies through Gateway -> Access Policy Management:

Domain NameRequest HeaderService
www.test.gridworkz.comNoneExternal Service 1
www.test.gridworkz.comuser:testExternal Service 2

This method is suitable for the server under the C/S architecture, such as the interaction between mobile APP and API.

Method 2: Identify the client through Cookie

Add the following two HTTP access policies through Gateway -> Access Policy Management:

Domain NameCookieService
www.test.gridworkz.comNoneExternal Service 1
www.test.gridworkz.comuser=testExternal Service 2

This method is suitable for web services and other HTTP request services.

Show results

Simulate requesting external services, please note that the domain name should be correctly configured for DNS resolution or local HOST file

When identifying the client through the Header request header

# Simulation request external service 1
$ curl www.test.gridworkz.com
<h1>hello ~ this is V1</h1>
# Simulate request for external service 2
$ curl -H user:test www.test.gridworkz.com
<h1>hello ~ this is V2</h1>

When identifying the client through a cookie

# Simulation request external service 1
$ curl www.test.gridworkz.com
<h1>hello ~ this is V1</h1>
# Simulate request for external service 2
$ curl --cookie "user=test" www.test.gridworkz.com
<h1>hello ~ this is V2</h1>

3. A/B Testing of Internal Services

Internal services do not directly serve users, but generally provide API support for other services. Its communication path does not pass through the application gateway, so A/B control cannot be performed through the application gateway. All services running on Kato are servicing by default. Management is carried out in a way, and service communication governance is carried out with ServiceMesh’s governance thought. A/B testing is one of the functions of service governance.

Prerequisites

  1. Have the two version test components used in the A/B test of the above external service;
  2. The communication address for simulating external service requesting internal service must be the host name (top-level domain name), such as requesting user service API, the request address: http://user/***, it will be the default in Kato To resolve the host name (top-level domain name), we usually recommend the communication address and port to read the environment variables, only need to set the connection information variable on the internal service That’s it.

Steps

As mentioned above, we have created internal service 1 and internal service 2 components in advance. The communication between the internal services of the Kato platform needs to establish a dependency relationship to complete internal service registration and service discovery. In the current state, internal service 2 It is still in an independent state. Before adding it to the subordinate dependency of External Service 1, we need to know this question:

The two internal services are essentially the same business and use the same service port. If there is a port conflict by default, it cannot be relied on at the same time. At this time, we need to open a network that works on layer 7 for external service 1 Governance plug-ins (provided by the platform by default), the working principle of plug-ins will reuse port 80, implement advanced routing through different domain names and other HTTP elements to select the lower-level dependent services used.

  1. External Service 1 depends on Internal Service 2, for the operation method refer to Service Communication,
  2. Open the export network management plug-in for External Service 1,
  3. Configure the routing strategy, similar to the configuration method based on the application gateway, except that it only supports the header-based processing method.

Show results

After the above configuration is completed, the internal service also has the ability of A/B testing, and the interface processed through the Header will present different content when it is called.

Data Feedback

The most critical link in the A/B testing process is data feedback and timely adjustment of strategies; the business-level [performance analysis monitoring] provided by Kato (/docs/user-manual/plugin-manage/tcm-plugin/) can provide you with Real-time request situation analysis to assist your decision-making; In addition, if you have your own monitoring method, please adjust the strategy reasonably according to your monitoring results. All the above control strategies can be modified and take effect dynamically.

Existing Defects and Improvement Plan

  1. The current internal service A/B test needs to be configured for each service. Global configuration is not currently supported. Subsequent versions will support ServiceMesh’s global configuration.
  2. At present, there is no process control for A/B testing, and the process will highlight iteration in subsequent versions.
  3. Automatic coordination of monitoring data and testing process to achieve automated A/B testing

You may also need to read:

Based on Kato’s rolling release, grayscale release and blue-green release practice

One-click online/rollback based on version number