Kato Technical Architecture
Kato practices the application-centric concept, absorbs excellent community solutions, and forms a data center logical technology architecture that combines application control, application runtime, and cluster control. It combines management nodes, computing nodes, gateway nodes, The physical architecture that the storage node divides the class node type forms a highly available and highly scalable data center architecture system.
Based on the data center architecture, it is divided into application management control terminals and resource management control terminals according to the target population to serve application development operators and resource users respectively. Multi-data center mode that supports users to manage and control multiple data centers at the same time.
Data Center Service Component Description
Data Center API Service
API service is an abstract core control service at the data center level. It provides Restful-style API services externally. It is the only entry point for data center control requests. The security control is based on TLS two-way security authentication. Distribute to the client after issuing the certificate independently.
After the API service request enters, the router module distributes the request, and the ordinary request enters the Handle request processing module. This type of request is mainly the request for operating the database model. The API service supports Mysql database and YugabyteDB databases based on ORM. The requests to enter the Proxy are divided into API requests and Websocket requests. The Discover module discovers its proxy target through the etcd service and forwards the request. Therefore, the API provided by other components can be forwarded by the API service agent through service registration.
Application Gateway Service
The application gateway is the only entrance for external traffic to enter the internal components of Kato tenants. It provides HTTP, HTTPs routing, TCP/UDP services, load balancers, advanced routing (A/B testing, grayscale release), virtual IP support and other functions.
The design of the application gateway is inspired by the NGINX Ingress Controller, based on the Kubernetes resource (Ingress, Service, Secret, Endpoint) model monitored from the kube-apiserver and converted into the configuration of the Proxy controller, providing an automatic working, real-time effective, dynamic Hot-configured layer 7 and layer 4 gateway services.
Application gateways can increase concurrency and basic performance through horizontal expansion, and the common configuration takes effect simultaneously in all gateway nodes. For 74IP+
The architecture goal of the single-node application gateway service is to support multiple data sources, support multiple IPs, support high concurrency, support multiple servers, and support dynamic configuration changes. Based on this, the application gateway architecture is designed as follows:
The application gateway service integrates the Openresty service as a front-end traffic proxy service. Based on Lua’s implementation of function extensions to Openresty, Lua-Config-Controller implements Openresty’s dynamic configuration, Upstream management, and load balancing strategy implementation. Lua-Monitor-Controller implements request data The record and summary report, Lua-Security-Controller realizes the security control of the request. The three Lua modules are driven by Openresty-Controller. The Metric-Controller module aggregates various monitoring data of the gateway and exposes the Metric-Server specified by Prometheus. Openresty-Controller is driven by data. The data comes from Gateway-Model-Controller. There is no coupling between the two layers. It implements standard interface specifications. We can implement drivers based on other proxy services, such as F5, as needed.
The data model source of the application gateway supports the ingress model of kubernetes or the simplified model found in etcd, and the data discovery service drives the dynamic work of the entire gateway. IP is an important resource of the gateway. After the IP-Port-Pool module is started, it will obtain all available IP addresses and ports of the current cluster, and establish awareness of IP address changes. The IP-Manage module will determine whether the gateway strategy is based on the IP and port data. Take effect on the current node.
Application Construction Service
The Kato application construction service processes the CI process, parses, compiles, and packages the input source including
source code or
Docker image or
application market application, and finally generates the version medium of the application (component).
In the traditional sense, the complete CI process will include: design, coding, packaging, testing and release. Since its launch, Docker images have gradually become a new form of packaging many application codes. Existing CI products have been very mature in source code testing and Pipline, such as Jenkins, Gitlab, etc., so Kato can directly connect to third-party services in the pre-processing of source code or Docker images, and the third-party services are processed. The source code or mirror image is then connected to the Kato-Chaos module for application abstraction.
The input source of Chaos is a code repository that supports Git and Svn protocols, and a Docker mirror repository. If it is source code, Chaos intelligently judges the type of source code, such as Java, PHP, Python, Dockerfile, etc., selects the corresponding BuildingPack (source code builder) according to different source code types for source code compilation, and recognizes the operating environment requirements defined in the source code. , Application ports, environment variables and other parameters form an abstract configuration prototype of the application. Source code types other than Dockerfile will be compiled into application code environment packages (SLUG) and stored in distributed storage, and other generated Docker local images are stored in the data center mirror repository, combined with various attribute information of the application to form application component attributes.
The application construction service is a stateless architecture and is designed as a task executor driven by a message system (rbd-mq). The Git client and mirror repository client are implemented, and the SVN client is integrated to obtain component construction source resources. The complete log is recorded during task execution, and the event client inputs the log to the rbd-event-log service through the message flow. Each source code construction process is controlled by Chaos to call the native Docker API to create and build containers and maintain the container life cycle.
The source code construction process is a very resource-consuming process. Therefore, the application construction service supports multi-node deployment to increase the number of concurrently supported construction tasks. The maximum number of concurrent construction supported by each node is determined by the number of node CPU cores or manually set by operation and maintenance personnel.
- For the BuildingPack compiled from source code, refer to Support Documents for Each Language.
- Application construction service supports multi-point high-availability deployment, and multi-point deployment obtains application construction tasks from message middleware.
Application Runtime Control Service
The application runtime control service instantiates the Kato-Application Model into a Kubernetes resource model, assigns various resources required for application operation, and completes the running state part of the application life cycle, which can be understood as a CD control service. The design points of this service It is the life cycle supervision that supports a large number of applications.
The application life cycle may experience start and stop, upgrade and rollback. Different application types require different control strategies. For example, stateless applications can be upgraded disorderly, while the upgrade control strategy of stateful applications will be more complicated. The Worker service implements life cycle controllers of various component types, including start, stop, update, and scaling waiting.
Application operation requires a variety of external environment support, such as the need to allocate tenant IP, port and other network resources, need to allocate persistent storage resources according to application settings, such as shared file storage to allocate storage directories, and block storage to allocate storage resources based on various plug-ins. Establish service discovery and load balancing strategies based on application dependency properties to provide mesh plugins. A scheduling policy is generated according to the application attributes and the Kubernetes cluster is called to schedule the application to run.
Currently Kato uses the following resource types in Kubernetes: Deployment, Statefulset, Service, Ingress, Secret, ConfigMap, Pod. For users, there is no need to understand these resources, nor are they reflected in the product, they are just a carrier of application operation.
The function of the worker component is divided into a stateful part and a stateless part. In order to implement the cluster deployment of the worker component, the worker conducts the election of the master node, and the service of the election as the master node will provide the application storage provider and storage statistics service.
Worker component is a module of kubernetes-controller controller extension and scheduler extension. Currently, there are local storage providers, shared storage providers for stateful components, etc., and it will also obtain existing resources in the Kubernetes cluster for Kato application components to choose from .
The Worker component can support multi-node deployment to ensure high availability.
The design goal of Kato cluster monitoring service is to comprehensively monitor the four dimensions of application business performance level, application container resource level, cluster node level, and management service level. It provides data storage, query, visualization and alarms functionally, and supports performance. Large-scale clusters.
Kato encapsulates the Monitor component based on Prometheus. By automatically discovering various monitoring objects of applications, clusters, and cluster node services from etcd and Kubernetes clusters and completing the configuration of Prometheus monitoring targets, the monitoring targets are included in the scope of Prometheus monitoring. Each component of Kato also implements Prometheus exporter end exposure monitoring indicators. Automatic monitoring of multiple dimensions is realized without user intervention.
The Monitor component can automatically configure some default alarm rules. It is planned to support obtaining custom alarm rule resources from Kubernetes to generate Prometheus alarm rules to make them dynamically effective, thereby supporting custom alarm rules. The alarm information is processed by the third-party Alter-Manage system.
Prometheus does not support cluster deployment by default, and cannot directly increase the monitoring capabilities by expanding nodes. In order to improve the processing capacity of the monitoring system, it currently supports remote storage of stored data to improve data storage capacity; supports multiple node configuration synchronization to provide service availability; supports adjusting the monitoring data storage window and reducing the occupation of memory resources. It is planned to support clustering and partitioning of monitoring targets, thereby supporting expansion of nodes to expand monitoring capabilities, and providing a unified query API to shield the underlying data partitions.
Message Middleware Service
The MQ component is a lightweight distributed, message persistent and globally consistent message middleware based on Etcd. This component maintains asynchronous task messages and provides multi-topic message publishing and subscription capabilities.
There are many message middleware products on the market, such as ActiveMQ, RabbitMQ, ZeroMQ, etc., according to our needs: distributed, persistent, global consistency, lightweight, orderly and other key attributes, the existing message middleware products or More or less dissatisfied.
Etcd is a simple, safe, fast, and reliable key-value storage service, which plays an important role in our entire system architecture. It is very convenient to implement a basic message system based on the existing features of etcd. A key is used for each subscription topic, and the timing and exclusive mechanism are controlled by etcd, so that incoming messages can be consumed sequentially. The client can subscribe to messages very simply through the watch mechanism provided by etcd. Based on this, we have packaged two types of API: restful and grpc. The client can conveniently operate the PUB/SUB of the message.
Event and Log Processing Service
The logs and message information that the Kato platform needs to process include three categories: user asynchronous operation logs, application construction logs, and application operation logs.
User asynchronous operation log: For user operation logs, we need to track the final state of each operation in a distributed manner. Currently, the eventlog component judges the final state of the operation according to the log aggregation of each operation. In the process of processing an asynchronous task, other components send process log records to the eventlog cluster through the gRPC message flow.
Application build log: Mainly display the output information of source code build or Docker image build, which comes from the application build service.
Application log is currently divided into two forms:
(1) Standard output/error output log
For standard output logs, the computing node log collector obtains the log stream through Docker Daemon, and then separates the business. By default, it is based on TCP data stream communication to realize the real-time delivery of container logs of all computing nodes to the Eventlog component in accordance with the application-level aggregation for storage And real-time push to the UI, the log collector will realize the docking with the third-party log system in subsequent versions to directly send the logs to the third-party log processing platform.
(2) Business log (access log) output to a persistent file
For business logs output to the persistent directory, they generally need to be automatically analyzed, such as connecting to the ELK system, so the log processing plug-in is installed in the plug-in system to collect the log files of the persistent directory and send it to a third-party log analysis service.
The log processing design requirement for clusters is to support log aggregation, storage, real-time push and query. And can support a larger log volume.
According to the above log type analysis, the operation log is a limited log stream, and its main requirement is to implement storage and push clients. The processing link for this type of log is mainly that grpc message flow accepts logs from various component clients, aggregates and stores the logs in the unit of event, and simultaneously propagates in the cluster to provide distributed logs to clients Subscription capabilities. Realize the effect of single-node storage and multi-node readable.
The application running log is an unlimited log stream, and there are many log sources, which are determined by the amount of applications in the current cluster and the amount of logs generated by each application. Therefore, in order to support a huge amount of application logs, the eventlog service implements partitioned log processing capabilities, and flexibly adjusts the log processing nodes based on the component (service_id). The running log of the same component is passed by the log collection service of each computing node The TCP message stream sends logs to the designated eventlog service node. After the eventlog service receives the logs, it aggregates, buffers, and places the logs. At the same time, it provides real-time push capability to the subscription client. When the client is subscribed, the rbd-api service loads the specified eventlog service node according to the subscribed service_id information.
The cluster nodes of the Eventlog service need to communicate with each other, including operation log stream sharing and processing capability reporting to determine the processing node of the application running log. Therefore, the nodes of the Eventlog service realize inter-node communication through the ZeroMQ subscription mode. ZeroMQ is a high-performance message library, based on this, to ensure the communication performance between the Eventlog services. Eventlog’s worker nodes make full use of the memory buffer mechanism when processing logs to increase the throughput of log messages while reducing the pressure on the disk.
Eventlog service is a stateful cluster service that can support horizontal expansion to improve the processing capacity of application logs.
Cluster and Node Management Services
The Node component is the basic service formed by the Kato cluster, and all nodes in the cluster need to run this component. It provides key capabilities such as node information collection, cluster service maintenance, application log collection, and application runtime support.
The Node component has two roles, namely the master role and the worker role. Generally, the Node running on the management node has the master role. The master role maintains the status and information aggregation of all nodes, and provides API query services. The worker role node regularly reports the status to the master node.
The node controller first acts as the guardian task of the current node running service, which is similar to Kubelet in this respect. The services or health check items that each node needs to run are defined in the /opt/kato/conf directory in the format of yaml. After the controller is started, it will read the configuration in this directory. For the services that need to be started, the systemd daemon is called to start the service. For the health check items, a health check controller is generated according to the configuration. The design here is mainly to realize the automatic operation and maintenance of the cluster and expand the monitoring project of kubernetes nodes. The node controller will maintain the status of all configuration items and report to the master node. The abnormal service controller will try to restart and restore the service according to the rules. If it cannot be restored, it will suggest that the master node set the current node to an unhealthy state and leave the scope of scheduling available until the node is restored.
In terms of monitoring, Node exposes various monitoring indicators of the node’s operating system and hardware level (integrated promethes node-exporter), and as a collector of application performance analysis data, it converts and exposes application performance analysis indicators. All exposed monitoring indicators are collected by the rbd-monitor service.
The Node service running on all computing nodes establishes the operating environment support for running applications in the tenant network, especially ServiceMesh support and plug-in dynamic configuration query support. The Node service provides general configuration discovery APIs and service discovery APIs to support the application architecture of the current node. On this basis, it provides the XDS Server service and provides dynamic configuration for the built-in ServiceMesh architecture.
The node controller log collection module implements the collection of running logs of all application containers on the current node. By obtaining the container list and log driver configuration in real time from Dockerd, a log copyer driver for each container is generated. The log copyer driver has multiple implementations. The default implementation is to transfer logs to the eventlog service. It also supports the drivers of other log collection services, such as ES.
Application Web Terminal Control Service
This component realizes the function of connecting to the container console via the web. This component communicates with the UI through WebSocket, and users can send various shell commands by simulating the Web terminal. Webcli executes the commands in the container through the exec method provided by kube-apiserver and returns the results to the Web terminal.
The rbd-webcli service can intercept and clean all user commands. Prevent malicious command injection, and realize the terminal-to-browser through the communication mode.
Metadata Storage Service
Kato data center metadata currently supports storage in Mysql database or YugabyteDB database. The database mainly stores the basic metadata of the application model and does not record the running state data of the application. Running state data is mainly maintained by the Kubernetes cluster or recorded in the ETCD service.
5.1.8 and previous versions support Mysql 5.6
5.1.9 and later versions support Mysql 5.7 8.0
Kubernetes Master Service
Kubernetes Master includes three components of Kube-apiserver, Kube-ControllerManager, and Kube-Scheduler version 1.10.11.
DNS service provides DNS resolution service for the cluster, based on the secondary development of Kube-DNS.
Mirror Warehouse Service
Based on the open source Distribution project, it is used for container image storage in the current data center.
Package Repository (Artifactory)
Based on the open source Artifactory project, it serves all third-party class libraries and file packages required for application construction based on source code, storage or proxy application construction. It is a necessary component for source code construction, which can be connected to other package repositories within the enterprise.
Business Logic Layer Component Description
As the key module of Kato’s application-centric abstraction, the application console UI component is oriented to application developers and application users. Based on the Django+Ant design front-end and back-end separation architecture design, it provides users with application abstraction, application group abstraction, data center abstraction, and application market abstraction to provide interactive experience. Realize a complete application creation and management process, application delivery sharing process, etc.
Resource Management Console (Enterprise Edition)
The resource console UI component provides Kato cluster resource management. It is planned to support resource management capabilities for docking with IaaS and is designed for operation and maintenance personnel. Focus on the management of node physical resources, cluster resources, management service resources, actual application resources, and tenant resources. Kato is a key display platform for automated operation and maintenance capabilities.