Overview:

The [Template Automation Series], which started in February, has been familiar with the automated construction of multiple virtual machine templates through a series of articles. However, the number of templates in the actual enterprise environment will far exceed these. At this time, the management and The update is still very complicated and cumbersome (although it has been greatly improved compared to before). Now I share my GitOps-based management template to further improve the efficiency of template construction and management. This article will introduce how to automate management of templates through GitLab CI/CD.

There is an important problem that needs to be solved for template preservation. It is not recognized by the template name when the cloud platform or other automatic calls are made. If the template is only updated by the name, other systems will not be able to correctly recognize the new template. vCenter 6.5 began to support template updates. This feature is solving the problems we encountered, and Packer also supports this feature (OVF template) in the version at the end of August. There is another advantage of using the vCenter content library. The content library supports subscription. If the enterprise contains multiple vCenter environments, it only needs to be built once.

Through the previous article, everyone is already familiar with template construction. In order to improve the reading experience of mobile devices, this manual will not introduce detailed template configuration. You can view it directly by visiting https://github.com/6547709/gitops-packer Code and configuration.

Based on GitOps to automatically build vSphere template features:

Use Gitlab to store template configuration files;
Realize automated template construction based on Gitlab CI/CD;
Semantic version management (feet, fix) based on Git submission records, the version number is automatically incremented and stored in the Notes of the template;
Perform CI/CD tasks regularly to achieve template variation;
Use the vCenter content library storage template with -latest as the suffix;
The vCenter content library template is automatically updated every time it is built, and the ID remains unchanged to ensure that the vRA cloud platform or other tools call the latest template;
All passwords and configurations are defined through .gitlab-ci.yml;
Provide multiple templates such as Windows2016/2019, Ubuntu1804/1910/2004, CentOS7/8;
Basic optimization of all templates (refer to the automatic response file of the corresponding template);
The Windows template uses the ISO image to integrate the latest patches, shortening the deployment time;
Tested in vSphere 7.0 and Packer 1.6.4 environment.

Related tools:

Packer: It is an open source automated virtual machine template building tool that supports private and public clouds, covering almost all environments.

vSphere: VMware enterprise-level virtualization software, widely used by enterprise customers, has the characteristics of high stability, good performance, high security and ease of use.

govc: It is a cli tool based on govmomi that realizes remote management of vSphere.

https://github.com/vmware/govmomi

Packer-provisioner-windows-update: Windows Update for packer plug-in.

https://github.com/rgl/packer-provisioner-windows-update

Gitlab CI/CD: is a CI/CD tool that works with code integration.

https://docs.gitlab.com/ee/ci/

Semantic-delivery-gitlab: Mirroring is used to implement semantic version management.
Harbor: It is a private Docker image repository that stores Docker images used to execute Packer, Govc, and Genisoimage.

https://hub.docker.com/r/hutson/semantic-delivery-gitlab

Related code: contains all the files required by Gitlab CI/CD.

https://github.com/6547709/gitops-packer

For the configuration of each template, please refer to the historical article.
CI/CD can choose different tools according to the situation, and the principles are the same.

Environmental requirements#

Gitlab server and Gitlab-Runner server, where Runner runs in Docker mode;
A Windows/Linux/MacOS computer that can connect to vCenter Server;
Packer program, Govc program, Windows Update for Packer program;
Installation mirrors for each operating system;
VMware Tools, Windows pvscsi driver;
vCenter content library;
Semantic-delivery-gitlab mirror;

Step summary#

Build a Harbor mirror warehouse to store local mirrors;
Set up Gitlab and Gitlab CI/CD related environments, using Docker Runner mode;
Create a project in Gitlab, upload related code, modify related configuration;
Perform automated build testing;
Add timing tasks in Gitlab CI/CD;
carry out.

Build DockerIamge for executing Packer commands#

You need to download the three executable files of packer, govc and windows update for packer through the address in the relevant tool, and store them in the same directory as the Dockerfile. The Dockerfile is as follows:

FROM centos:8
LABEL maintainer="Alex Li <[email protected]>"
ENV PACKER_VERSION=1.6.4
ENV GOVC_VERSION=v0.23.0
ENV GOVC_INSECURE=true
ENV TIME_ZONE Asia/Shanghai
RUN ln -sf /usr/share/zoneinfo/${TIME_ZONE}/etc/localtime
ADD ./govc /usr/bin/
ADD ./packer /usr/bin/
ADD ./packer-provisioner-windows-update /usr/bin
RUN chmod a+x /usr/bin/govc /usr/bin/packer /usr/bin/packer-provisioner-windows-update && yum install -y  genisoimage
WORKDIR /tmp
CMD ["/bin/bash"]

Use Docker Build to build Docker Image and upload it to the container warehouse (private or public):

docker build -t harbor.corp.local/library/gitops-packer:v1.0.
docker push login harbor.corp.local
docker login -u admin harbor.corp.local
docker push harbor.corp.local/library/gitops-packer:v1.0

Upload the semantic-devlivery-gitlab image to Harbor

docker pull hutson/semantic-delivery-gitlab:9.1.0
docker tag hutson/semantic-delivery-gitlab:9.1.0 harbor.corp.local/library/semantic-delivery-gitlab:9.1.0
docker push harbor.corp.local/library/semantic-delivery-gitlab:9.1.0

Create Access Token in Gitlab

Log in to Gitlab with a personal account;
Go to user settings -> access token;
Fill in the token name, expiration date and selection authority -> create a personal access token;
Save Token for backup;

Create a project and submit all codes to the warehouse#

Detailed instructions are not given here. The following is a screenshot of the final warehouse content:

Modify the .gitlab-ci.yml configuration file#

This file is the main configuration file of Gitlab CI/CD. Common configuration parameters are defined in this file. There is no need to modify the packer and operating system automatic response files.

Tip 1: In order to make this manual easy to read, all sensitive information is also declared in this configuration file. It is strongly recommended to use Gitlab project variables for definition to avoid leakage of sensitive information.
Tip 2: The following code examples have been deleted, please get the complete code from Github.

# Environment variable definitions, it is not recommended to configure sensitive definitions (for example: password) in the production environment
variables:
 DOCKER_DRIVER:"overlay2"
 GITLAB_TOKEN:"xxxxxxxxxxxxx"
# Define the Docker image used to execute packer, govc and genisoimage, which needs to be built in advance
 PACKER_DOCKER_IMAGE:"harbor.corp.local/library/packer-gitops:v1.0"
# Define the vCenter-related information used to create the template. The password part is recommended to be defined in the project variable of Gitlab to improve security.
 VC_SERVER:"vcenter.corp.local"
 VC_USERNAME:"[email protected]"
 VC_PASSWORD:"VMware1!"
 VC_DATACENTER:"Labs-DC02"
 VC_CLUSTER:"DC02-Cluster"
 VC_CONTENT_LIBRARY:"DC02-VM-Templates"
 VC_FOLDER:"Templates"
 VC_DATASTORE:"SSD_DATASTORE"
# Define VM related configuration
 VM_NETWORK:"vlan100"
 VM_CPU:2
 VM_MEM:4096
 VM_DISK:81920
 VM_VIDEO_RAM:16384
 VM_HW_VERSION:17
# Define the storage path of the installation CD. The Windows system needs to adjust the automatic answer file according to the installation CD.
# - - - - - Part of the code is omitted here, and the complete code is obtained from github------ 
 OS_CENTOS8_ISO:"[SSD_DATASTORE 0-ISO/CentOS-8.2.2004-x86_64-dvd1.iso"
 OS_WINDOWS2016_ISO:"[SSD_DATASTORE] 0-ISO/cn_windows_server_2016_vl_x64_dvd.iso"
# Define the storage path of the CentOS8 auto-response CD, and each compilation will automatically overwrite the previous version.
 OS_CENTOS8_KS_ISO:"[SSD_DATASTORE] 0-ISO/centos8-ks.iso"
 OS_CENTOS8_GUI_KS_ISO:"[SSD_DATASTORE] 0-ISO/centos8-gui-ks.iso"
# Define VMware Tools installation path
 OS_WINDOWS_VMTOOLS:"[SSD_DATASTORE] 0-ISO/VMware-tools-windows-11.1.5.iso"
# Define the path of the Windows system using the pvscsi driver. This file can only be mounted by one VM at the same time. You need to configure one file for each Windows system.
 OS_WIN2016_DRIVER:"[SSD_DATASTORE] 0-ISO/win2016-pvscsi-Windows8.flp"
# Define GOVC environment variables, used to upload CentOS8 automatic response ISO (ks.cfg）
 GOVC_URL: ${VC_SERVER}
 GOVC_USERNAME: ${VC_USERNAME}
 GOVC_PASSWORD: ${VC_PASSWORD}
# This configuration is used to define the passwords of Linux root and ops users, Windows Administrator and ops users,The password is recommended to be defined in the Gitlab project variable to improve security.
 LINUX_SSH_PASSWORD:"VMware1!"
 WINDOWS_PASSWORD:"VMware1!"
# This variable is used to define the virtual machine name, which will end up with-latest is stored as a suffix in the vCenter content library.
 CENTOS8_VM_NAME:"CentOS8"
 WIN2016_VM_NAME:"Win2016"

# Define the Windows installation KEY and configure it according to different installation versions.
 WIN2016_KEY:"XXXXX-XXXXX-XXXXX-XXXXX-XXXX-XXXX"
 WIN2019_KEY:"XXXXX-XXXXX-XXXXX-XXXXX-XXXX-XXXX"
# Define CI/In the CD stage, the devliver stage is used to generate the version number, the validate stage is used to verify whether the packer configuration file is correct, build-The iso stage is used for ISO production of CentOS8 and automatically uploaded to the shared storage. The build stage is used for template construction, list-The library stage is used to list content library templates.
stages:- deliver
 - validate
 - build-iso
 - build
 - list-library

# Semantic version management is adopted, and the version number is increased based on the commit message and the Release document is generated. At this stage, packaging and deployment will not be carried out, only the version tag is added.
deliver:
 image:
 name: harbor.corp.local/library/sematic-delivery-gitlab:9.1.0
 entrypoint:[""]
 stage: deliver
 only:- master
 script:- semantic-delivery-gitlab --token ${GITLAB_TOKEN}

# This stage is used to verify that the packer configuration file is correct.
packer-validate:
 image:
 name: ${PACKER_DOCKER_IMAGE}
 stage: validate
 script:- packer validate ./CentOS8/centos-vsphere.json
 - packer validate ./Win2016/win2016-vsphere.json
 only:- tags
 dependencies:- deliver
# This stage is used to build the ISO file required by CentOS8 and upload it to the vSphere storage(Automatically overwrite the previous version)。
# for CentOS8。
CentOS8-ks-iso-build:
 image:
 name: ${PACKER_DOCKER_IMAGE}
 stage: build-iso
 script:- cd CentOS8
 - sed -i 's/__PASSWORD__/'"${LINUX_SSH_PASSWORD}"'/g'./ks.cfg
 - genisoimage -o centos8-ks.iso -V "OEMDRV"./ks.cfg
 - govc datastore.upload -ds ${VC_DATASTORE} centos8-ks.iso 0-ISO/centos8-ks.iso
 only:
 changes:- CentOS8/ks.cfg
# - - - - - Part of the code is omitted here, and the complete code is obtained from github------ 
# This stage is used to generate a virtual machine template, the template name is based on the variable definition in the job, and the final template is used-latest is the most suffix.
# for CentOS8。
packer-build-CentOS8:
 image:
 name: ${PACKER_DOCKER_IMAGE}
 stage: build
 variables:
 VM_NAME: ${CENTOS8_VM_NAME}
 script:- cd CentOS8
 - packer build --force centos-vsphere.json
 only:- tags
 dependencies:- packer-validate
# - - - - - Part of the code is omitted here, and the complete code is obtained from github------ 
# for Windows Server 2016。
packer-build-Win2016:
 image:
 name: ${PACKER_DOCKER_IMAGE}
 stage: build
 variables:
 VM_NAME: ${WIN2016_VM_NAME}
 OS_WINDOWS_DRIVER: ${OS_WIN2016_DRIVER}
 script:- cd Win2016
 - sed -i 's/__PASSWORD__/'"${WINDOWS_PASSWORD}"'/g'./Autounattend.xml
 - sed -i 's/__LICENSEKEY__/'"${WIN2016_KEY}"'/g'./Autounattend.xml
 - packer build --force win2016-vsphere.json
 timeout: 120m
 only:- tags
 dependencies:- packer-validate
# - - - - - Part of the code is omitted here, and the complete code is obtained from github------ 
# list vcenter content library。
list-content-library:
 image:
 name: ${PACKER_DOCKER_IMAGE}
 stage: list-library
 script:- govc library.info -json ${VC_CONTENT_LIBRARY}/*
 only:
 - tags

Standard git commit message format#

Standard and standardized commit messages can not only ensure the readability of the version history, but also understand the content and scope of each change, and will automatically generate documents on the Release page. Therefore, it is strongly recommended to use the standard commit message format and content.

fix: Use this flag when fixing code problems. Example: fix: Repair Windows template ISO file error. Version number change: 1.0.0->1.0.1

feat: Use this flag when adding new features or templates. Example: feat: Add Photon template. Version number change: 1.0.0->1.1.0

[ skip ci] When you do not want to automatically execute CI/CD, add this mark in the message. Example: fix: Update ReadME.[skip ci]. Version number change: no change

Verify the execution process and results#

When the changes are submitted, Gilab CI/CD will automatically execute the pipeline based on the configuration in .gitlab-ci.yml. The whole process is divided into 2 groups of 5 steps:

Perform semantic version management and add tags to the code;
Perform packer configuration file verification based on tag, CentOS8 automatically responds to ISO construction, image construction and content library content listing;

The execution process of the pipeline is shown in the figure below. It takes 25 minutes to complete the automated construction of 8 templates:

Add a scheduled task#

In the Gitlab CI/CD plan, add a weekly/monthly scheduled execution plan, as shown in the figure below after the addition is completed.

[ Optional]Windows mirror integration latest patch#

In the process of template building, Windows takes the longest time to build, in some cases up to 2 hours, which may trigger the timeout mechanism of Gitlab CI/CD and Packer, causing the task to fail. In order to improve the efficiency of template construction and reduce the error rate, it is recommended to package the template with the latest patch by yourself. The following is the general production process for reference.

Download DISM++ tool (made by Chinese people, graphical operation);
Unzip the system installation CD to the directory (d:\win2016-iso), extract the \sources\install.wim file in the unzipped directory, and copy it to another directory (d:\win2016-iso);
Open the DISM++ tool and load the install.wim file. The mount path must be created in advance (d:\win2016-iso\mnt);
Select the loaded install.wim system and open the session;
Use the system installation CD to install the system and perform system updates. After the update is complete, find the installed patch version number in the installed system update (add/remove programs);
Log in to the https://www.catalog.update.microsoft.com/ website, download the msu format file of the patch based on the good version, and save it in the directory (d:\win2016-iso\msu);
On the DISM++ tool page, control panel -> update management -> add (d:\win2016-iso\msu), and execute the installation, and wait for the installation to complete (a long time);
After the patch is installed, DISM++->File->Save as Image (d:\win2016-iso\new.wim) to store the new image;
Use new.wim to replace install.wim in the unzipped directory of the system installation disk (the name must be install.wim);
DISM++->Common Tools->Toolbox->ISO Generator, select the system installation disk and compressed directory to replace the install.wim file as the source, select the d:\win2016-iso\ directory as the target, and specify the name of the new CD and add tags;
DISM++->File->Uninstall image;
The system installation CD containing the specified patch is completed;
Upload the new ISO to the shared storage, and modify the .gitlab-ci.yml file to use the new ISO image path.

Tip 1: Although DISM++ provides the system update function, it seems abnormal for the Windows Server system, so it is recommended to add patches manually.
Tip 2: Through the DISM tool, you can also add the pvscsi driver to the installation CD, so there is no need to add the pvscsi driver.

carry out#

So far, the template GitOps in the vSphere environment has been implemented through Gitlab CI/CD, Govc and Packer. In the future, as long as the relevant configuration files are modified, the template construction will be automatically executed. Since this construction is executed concurrently, the efficiency is very high. Finally, I wish you all Can work easily and live happily.

source:

https://www.toutiao.com/i6882207358122983948/

GitOps—Automatically build virtual machine templates through CI/CD