Component A/B Testing Practice

Function Description

Controlled experiments, also called random experiments and A/B testing; in software development, product requirements are realized through a variety of technical means, A/B Test experiments provide a valuable way to evaluate the impact of new features on customer behavior; the ability to run A/B test experiments on websites and services, so that a more scientific method can be used to evaluate the value of ideas at different stages of the planning process; Halt the debate of which design is better? Which copywriter is good? Which execution strategy is effective, it is better to let real users and data tell you the answer.

In terms of technology itself, A/B testing emphasizes providing different services to different clients. The two differences here are very important.

Different Clients

Generally, clients are classified in some way, such as HTTP protocol. Usually, the Header request header and Cookie are set according to user information to distinguish different clients.

Different Services

Generally refers to different versions of different applications, which are different components on the Kato platform.

Kato currently supports the A/B testing practice of the HTTP protocol, which is currently the most widely used protocol.

The service needs to be A/B tested, and it is necessary to distinguish whether it is an internal service or an external service. The A/B test feature of the internal service is provided by the ServiceMesh layer, and the external service is provided by the application gateway.

Let’s take the following scenarios as an example to conduct A/B testing practices on external services external services 2 internal services internal services 2.

A/B Testing for External Services

A/B testing of external services is the most commonly used scenario, because external services are services that directly face users. The business program needs to inject user identification information into the Cookie through a certain business strategy or into the Header request header through the mobile APP. The Kato application gateway can recognize these identifiers and provide users with corresponding services according to the policies configured by the users.

As shown in the example above, External Service External Service 2 I have created in advance and simulated as two versions of the same business program

Add the following two HTTP access policies through Application Gateway -> Access Policy:

Usage 1: Identify the client through the Header request header

Domain NameRequest HeaderService
www.test.comNoneExternal Service
www.test.comuser:testExternal Service 2

Test Method:

# Simulate requesting an external service, please note that for the domain name, please follow the instructions in the Add Access Policy document to correctly set DNS resolution settings or local HOST file settings
curl www.test.com
# Simulation request external service 2
curl -H user:test www.test.com

This method is suitable for the server under the C/S architecture. For example, the interaction between mobile APP and API.

Usage 2: Identify the client through Cookie

Domain NameCookieService
www.test.comNoneExternal Service
www.test.comuser=testExternal Service 2

Test Method:

# Simulate requesting external services
curl www.test.com
# Simulation request external service 2
curl --cookie "user=test" www.test.com

This method is suitable for web services and other HTTP request services.

A/B testing for Internal Services

Internal services do not directly serve users, but generally provide API support for other services. Its communication path does not pass through application gateways, so A/B control cannot be performed through application gateways. All services running on Kato are servicing by default. Management is carried out in a way, and service communication management is carried out with ServiceMesh’s governance ideas. A/B testing is one of the functions of service governance.

In our case above, there are internal services and internal services 2, which we have created in advance. The communication between internal services needs to establish a dependency relationship to complete the internal service registration and service discovery. In the initial state, the internal service 2 is still in an independent state. We need to know the subordinate dependencies of the external service to which it is added. Such a problem:

The two internal services are essentially the same business and use the same service port. If there is a port conflict by default, you cannot rely on them at the same time. At this time, we need to open a network that works on the 7th layer for the external service Governance plug-in (provided by the platform by default), the working principle of the plug-in will reuse port 80, implement advanced routing through different domain names and other HTTP elements to select the lower-level dependent services used; therefore, the premise of using this plug-in is:

Simulate external service The communication address for requesting internal service must be the host name (top-level domain name)

For example, to request the user service API, the request address: http://user/***

The host name (top-level domain name) will be resolved by default in the Kato environment

We usually recommend that the communication address and port read the environment variables, just set the connection information variable on the internal service.

  • Configure routing strategy, which is similar to the configuration method based on application gateway, except that it only supports header-based processing.

After the above configuration is completed, the internal service also has the ability of A/B testing.

Data Feedback

The most critical link in the A/B testing process is to pay attention to data feedback and adjust strategies in time. The business-level performance analysis monitoring provided by Kato can provide you with real-time request analysis to assist your decision-making. In addition, if you have your own monitoring method, please adjust the strategy reasonably according to your monitoring results. All the above control strategies can be modified and take effect dynamically.

Existing Defects and Improvement Plan

  1. The current A/B test of internal services needs to be configured for each service. Global configuration is not currently supported, and subsequent versions will support ServiceMesh global configuration.
  2. At present, there is no process control for A/B testing. In subsequent versions, processization will highlight iteration.
  3. Automatic coordination of monitoring data and testing process to achieve automated A/B testing