Kato's Method of Building a Java Maven Project

Apache Maven is a cross-platform project management tool. Mainly serve for project construction, project management and project information management based on the Java platform. The so-called project construction is a series of processes such as cleaning, compiling, testing, reporting, packaging, and deployment of the project, which is called project construction.

For more Maven introduction, please refer to the official document: https://maven.apache.org/

Standard Java projects managed by Maven generally have several product packaging methods:

Jar Archive

This type of packaging method used to be mainly used for public class library projects, and now it is also used as a packaging method for service projects of architectures such as SpringBoot. Projects marked as Jar packages generally do not need to rely on middleware and can run directly.

War Archive

In the past, the main packaging method of traditional Java services was run in middleware such as Tomcat.

There are generally two types of project organization in Maven:

Single Module

This type of method is generally adopted by small projects, and the main project directly defines the configuration of the project’s dependencies and packaging forms.

Multiple Modules

This type of method is currently widely used and more flexible project organization. A parent project contains two sub-projects (modules), a core module and a webapp module. The webapp module depends on the core module. This is a very common engineering division method, that is, the core module contains the core business logic of a certain field, and the webapp module creates a front-end website by calling the service classes in the core module. This separates the core business logic from the front-end presentation. If you later decide to develop and support mobile apps, the core module can be reused.

A multi-module project can build a specified service module separately, and other modules that it depends on will also be built when it is built separately.

Maven dependency package management is its core function. For dependency package management, we need to pay attention to the following points:

Local Maven Repository

The local repository exists on the local disk of the compilation environment, which can be considered as the first-level cache of dependent packages. The storage path is configured in the settings.xml file localRepository

Remote Repository

Remote repositories include Maven central repositories, self-built private servers or repositories provided by other cloud vendors. Users can configure in pom.xml or settings.xml files. Warehouse authentication information can only be in settings Configured in .xml.

Remote Repository Proxy or Mirror

You can configure mirrors in the settings.xml configuration file to set up, such as using a private server to cache the mirroring of all external repositories.

<mirrors>
     <mirror>
         <id>nexus</id>
         <name>internal nexus repository</name>
         <url>http://192.168.0.1:8081/nexus/content/groups/public/</url>
        <mirrorOf>*</mirrorOf>
   </mirror>
</mirrors>

Kato Recognizes Maven Projects

The Kato rbd-chaos service obtains the project source code from the code repository specified by the user through the Git and SVN source code management protocol, and reads the pom.xml file from the project home directory. If it is read, the current project is considered Java-Maven type, the source code will be packaged and constructed using the source code compilation method corresponding to the Java-Maven type.

Multi-module Code Detection

During the service creation process, the source code inspection task will be initiated. The source code inspection program will read the pom.xml file. If the multi-module configuration is read from the file, it means that the current code repository is of a multi-module type and will be batched. Create multiple Kato components.

The detection program recursively reads the pom.xml configuration of all modules, selects all modules whose packaging method is jar or war, and automatically generates the default Maven build command (BUILD_MAVEN_CUSTOM_GOALS) And the service start command (BUILD_PROCFILE), these two commands require the user to determine whether they are correct and correct them. The following describes the customization methods of these two parameters in detail:

The build command looks like the following form:

clean install -pl module_name -am

Kato creates a corresponding service for each module, so users need to manually judge which are public class library projects according to their own project conditions, and cancel the creation of public class library class modules during the batch creation process. Each service has a corresponding build command, if it is incorrect, the user needs to modify it.

Single Module Code Detection

If the identified project is a single-module project, it directly enters the service construction process.

Kato Builds Maven Components

When the user builds the service from UI trigger or Webhook trigger, the building task will be generated by the data center API service and sent to the message system. The rbd-chaos service competes for the building task from the message system. If the current rbd-chaos node is performing more than If the maximum value is reached (the default maximum value is 2 times the number of node CPU cores), task competition will be suspended. After getting the task, start to get the project code through git or svn client and cache it. The next build task will update the code based on the cached code. Then start the builder container to pass in the code and build parameters (specific build parameters are described below) to execute the build task.

The execution of the build task has the following steps:

1. JDK and Maven Version Selection

The system.properties file can be defined in the main directory of the project source code to specify the version of the JDK and the version of Maven. The format is as follows:

java.runtime.version=1.9
maven.version=3.2.5

If there is a system.properties file in the code, the code detection phase will read the JDK version defined by this file and assign default values ​​to the BUILD_RUNTIMES variable.

Users can set the following variables on the Kato platform to select the version of JDK and Maven:

  • BUILD_RUNTIMES (OPEN-JDK version) optional value 1.6 1.7 1.9 10 11
  • BUILD_ENABLE_ORACLEJDK (whether to use ORACLEJDK) default is no.
  • BUILD_ORACLEJDK_URL (ORACLEJDK download path) specified by the user.
  • BUILD_RUNTIMES_MAVEN (Maven version), the default is 3.3.9 If the JDK version is 1.6, the default is 3.2.5

Note that all variables described in this article starting with BUILD_ can be set in the service environment variables, and most other variables have also been reflected in the build source settings.

1.1 Download of JDK

If ORACLEJDK support is enabled, download the JDK through the specified BUILD_ORACLEJDK_URL path. If not specified, the Maven build script builds the JDK download path based on the version specified by the above setting variables by default, such as:

http://lang.gridworkz/jdk/openjdk1.8.0_20.tar.gz

The lang.gridworkz domain name is the internal domain name of the Kato data center and is provided by the rbd-repo service (artifactory), which means that the resources are downloaded from the rbd-repo service inside the data center. The rbd-repo service is just a layer of static resource cache. After it receives the request for the first time, it will download the corresponding resource from the cloud OSS provided by Kato, and then cache it locally.

Note that if you encounter a JDK download failure problem, you need to confirm whether the rbd-repo can be connected to the Internet, and whether Kato provides the specified version of JDK download.

1.2 Download of Maven

The Maven download is based on the principle of the JDK. The download path is constructed according to the specified version and downloaded from the rbd-repo service.

2. Maven Environment Settings

2.1 Processing of settings.xml File

settings.xml file can define remote repository and authentication information, mirror repository information, etc. Users can customize the configuration by setting the following variables:

  • BUILD_MAVEN_SETTINGS_PATH directly defines the local path of settings.xml, this path users can only use the source code directory, that is, the path starts with /app, for example, if it is in the main source directory, the path should be: /app/settings. xml
  • BUILD_MAVEN_SETTINGS_URL defines the remote download path of settings.xml. Because the settings.xml file may contain account information and is not suitable for storage in the code repository, it can be stored in the object storage to provide the download path.

The build script will first use the BUILD_MAVEN_SETTINGS_PATH variable, and then use the BUILD_MAVEN_SETTINGS_URL variable. If none is defined, the default settings.xml file will be used. The default settings.xml defines the use of maven.gridworkz as a global proxy for all remote libraries. The purpose of this is to be able to cache all the dependent packages that users need to use in the project as a second-level cache in the build process. Configure as follows:

  <mirrors>
    <mirror>
       <id>gridworkz-repo</id>
       <name>gridworkz repo</name>
       <url>maven.gridworkz</url>
       <mirrorOf>*</mirrorOf>
  </mirror>
  </mirrors>

The user can also set the above configuration through the following parameters:

  • BUILD_MAVEN_MIRROR_DISABLE is set to true, the above configuration will not take effect. No longer use the rbd-repo service to cache maven packages.
  • MAVEN_MIRROR_OF sets the agent range, the default is *, all agents.
  • MAVEN_MIRROR_URL sets the proxy repository service address, the default is maven.gridworkz provided by the rbd-repo service, and the user can set to switch to the private server address that supports mirror in the enterprise.
2.2 Cache Processing

Kato provides a first-level cache for the build environment of each service. The Maven project provides a cache for the maven installation directory, configuration directory, and local repository directory. The user clears the cache by setting the following variables.

  • After NO_CACHE is set, the next build process will remove the cached resources and then download again. After removing the variables, it will not be cleaned up the next time.

    Note that users of the secondary cache existing in the rbd-repo service can access the UI of the rbd-repo service to clean up.

3. Execute Maven Build Command

The compiler first needs to construct the compile command. The user can customize the compile command of Maven with the following parameters:

  • BUILD_MAVEN_CUSTOM_OPTS default value is -DskipTests Ignore unit tests
  • BUILD_MAVEN_CUSTOM_GOALS default value is clean dependency:list install

Therefore, if you do not set any parameters, the default build command for Kato is:

mvn -B -s setting.xml file path -DskipTests clean dependency:list install

The -B parameter means to run in a non-interactive way, -s specifies the setting.xml configuration file, and the path is determined by the logic described above.

The command used by most users for local testing is mvn clean install without the dependency:list parameter.

If it is a multi-module service, the BUILD_MAVEN_CUSTOM_GOALS parameter will be automatically changed when the service is created, similar to the following command:

clean install -pl module_name -am

Most standard projects can be constructed normally. If the user’s multi-module project does not meet the requirements, the user must modify the BUILD_MAVEN_CUSTOM_GOALS parameter.

After the build command is confirmed, the compiler will execute the Maven build and output the build process log in real time.

Note, please be sure to use the above command to execute the test locally before using Kato to build the project.

4. Operating Environment Processing

After mvn completes the compilation and packaging, there are two media, war package and jar package.

How the generated media runs, that is, the startup command configuration, is defined in the code main directory Procfile file. If the code is defined, the configuration content will be recognized when the service is created to initialize the BUILD_PROCFILE variable. For the specification of the Procfile file, please refer to Document

When BUILD_PROCFILE is not defined, the Kato compilation script will make the following judgments to generate the default startup command configuration:

Determine Whether it is a War Package

According to whether the pom.xml file contains <packaging>war</packaging> to determine whether the current project is a war project, if so, use the following run command to run:

web: java $JAVA_OPTS -jar ./webapp-runner.jar --port $PORT target/*.war
Is it a Springboot Project?

According to whether the pom.xml file contains <groupId>org.springframework.boot and <artifactId>spring-boot and the packaging method is jar package, it is recognized as a springboot project. If it is the default running command:

web: java -Dserver.port=$PORT $JAVA_OPTS -jar target/*.jar
Is it a wildfly_swarm Project?

According to whether the pom.xml file contains <groupId>org.wildfly.swarm to identify whether it is a wildfly_swarm project, if it is, the default running command is:

web: java -Dswarm.http.port=$PORT $JAVA_OPTS -jar target/*.jar

Manually configure BUILD_PROCFILE to directly set the startup command is the most accurate and direct way.

You need to set the startup command yourself in several scenarios

  • The generated package is not in the target/ directory. For example, the package built during multi-module construction may be in the target directory of the submodule. When the same service is created, Kato has been set according to the recognition result.
  • The project settings do not comply with Kato’s automatic identification strategy.
  • The startup command needs to be customized.

As can be seen from the above automatically generated startup command, Kato uses the webapp-runner project when running War. If you need to set the Tomcat version or other parameters, The startup command needs to be set according to the relevant parameters of webapp-runner. For example, when setting the session path:

web: java $JAVA_OPTS -jar ./webapp-runner.jar --path /test --port $PORT target/*.war

Other setting parameter reference document: tomcat parameter setting

5. Running Environment Packaging

After the above process is completed, the compiled script will package the code directory. The packaging command is as follows:

tar -z --exclude='.git' -X "$build_root/.slugignore" -C $build_root -cf $slug_file.

By default, the entire source code, JDK and other executable programs downloaded during the compilation process, and the media files generated by the compilation are packaged together. The purpose is not to omit any files, especially static files, configuration files, etc. If the user does not want to package the useless source code in the running state, the .slugignore file needs to be defined.

The .slugignore file is similar to the common .gitignore file syntax. It can be seen from the above packaging command that this file takes effect when the tar command is packaging.

6. Generate Docker Image

After all language types are compiled, a tar package will be generated. We provide a basic runner image to run these tar packages. The source code is at https://github.com/gridworkz/runner, and its workflow is as follows:

  1. Unzip the tar package to the /app directory.
  2. Analyze the Procfile file to obtain the software startup command.
  3. Set the environment variables for program operation, such as JAVA_OPTS.
  4. Execute the software startup command in the Procfile file.

Common Problems

Kato does not support the required JDK or Maven version.

According to the above description, if the JDK version you need to use is not in the current support list of Kato (mainly whether the Kato public OSS repository has downloaded packages), you can provide feedback from the community to add the required version or prepare your own JDK version for storage Cache locally in the rbd-repo repository.

Can Maven source code build in offline environment?

Since the build process needs to download resources such as JDK, Maven, and dependency packages from the public network, by default, it cannot be built using source code offline. You can operate rbd-repo to prepare the corresponding resources according to the above instructions. The enterprise version provides complete offline resources Rbd-repo service.

How to troubleshoot dependent package download failure.

The most likely failure to download dependent packages is limited to your network, or you use a private server repository and cannot be mirrored by the rbd-repo service. The most simplified way is to configure the relevant parameters to directly use the repository service you used in the past.

It can be built locally, but Kato cannot.

The above has fully explained the process and method of Kato Mvn construction. You need to differentiate and investigate the differences from your local construction one by one. The difficulty of the solution may depend on your understanding of the Maven build mechanism.

How to feedback other issues.

If you encounter a source code build problem and cannot solve it by yourself, you can move to github to submit your issues. We are very happy to solve various scenarios and provide the compatibility of Kato source code CI.