GitLab installation and basic use

[ TOC]

0 x00 Foreword####

In the open source world, there is no end!

Description: GitLab is a very good project. This is an open source project that allows users to run a project management system similar to GitHub on their own servers.
You can use GitLab to develop private projects before public release.

Official website address: https://about.gitlab.com/
Help document: https://about.gitlab.com/install/
Component reference: https://docs.gitlab.com/ce/development/architecture.html

GitLab uses a traditional open source business model, and they have two products:

Free open source software, users can install on their own servers, and a hosting service similar to GitHub.
Free community version Gitlab CE and paid enterprise version Gitlab EE
The enterprise version is based on the community version but comes with other features for corporate customers. It is more or less similar to the services provided by WordPress.org or Wordpress.com.

Gitlab advantages:

There is an open source and free version, which can be uploaded and pulled for private development;
The Community Edition is highly scalable and can support 25,000 users on a single server or cluster
Some features of GitLab include: Git warehouse management, code reviews, issue tracking, activity sources and wikis
It is equipped with GitLab CI for continuous integration and delivery

0 x01 environment installation####

CentOS

Gitlab basic requirements: at least 4GB of free RAM to run GitLab

System: CentOS Linux release 8.0.1905 (Core)GitLab 12.9.2 (ac5568eb5d8) GitLab Shell 12.0.0 GitLab Workhorse v8.25.1 GitLab APIv 4 Ruby 2.6.5p114 Rails 6.0.2 PostgreSQL 10.12 Gitaly Servers

Gitlab installation method:

The official recommendation is to use Omnibus for quick installation (using the rpm package for installation and deployment (recommended to download directly from the mirror source in China)); #here is the community version as an example curl https://packages.gitlab.com/install/repositories/gitlab/gitlab- ce/script.rpm.sh | sudo bash sudo EXTERNAL_URL="https://gitlab.example.com" dnf install -y gitlab-ee
Use yum for installation and deployment;

Installation process:
Step1. Use rpm to install the community version of gitlab (replace the update source and choose by yourself):

sudo dn8 install -y curl policycoreutils openssh-server wget
# You can use wget to download and install the rpm package
wget https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el8/gitlab-ce-12.9.2-ce.0.el8.x86_64.rpm
# dnf install gitlab-ce-12.9.2-ce.0.el8.x86_64.rpm
rpm -ivh gitlab-ce-12.9.2-ce.0.el8.x86_64.rpm

Step2. After the installation is complete, the following shows that the installation is successful:

Thank you for installing GitLab!
GitLab was unable to detect a valid hostname for your instance.
Please configure a URL for your GitLab instance by setting `external_url`
configuration in/etc/gitlab/gitlab.rb file.
Then, you can start your GitLab instance by running the following command:
sudo gitlab-ctl reconfigure

Step3. Modify the default configuration of gitlab

$vim /etc/gitlab/gitlab.rb
# Modify it to your own domain name or IP, in single quotes, and don’t change the previous http (and add the domain name to hosts)
external_url 'http://gitlab.weiyigeek.top'    

# Mail configuration, no mail server can close the mail service function
# gitlab_rails['smtp_enable']=false
# gitlab_rails['smtp_address']=""
# gitlab_rails['smtp_port']=587
# gitlab_rails['smtp_user_name']=""
# gitlab_rails['smtp_password']=""
# gitlab_rails['smtp_authentication']=""
# gitlab_rails['smtp_enable_starttls_auto']=
# gitlab_rails['smtp_tls']=
# gitlab_rails['gitlab_email_from']=''

Step4. After the modification is completed, the configuration needs to be reloaded and then restarted;

$ gitlab-ctl reconfigure
# [2020- 04- 15 T10:33:44+08:00] WARN: Please install an English UTF-8 locale for Chef to use, falling back to C locale and disabling UTF-8 support. #If you feel uncomfortable, just set the system language by yourself
# [2020- 04- 15 T10:33:45+08:00] WARN: Please install an English UTF-8 locale for Chef to use, falling back to C locale and disabling UTF-8 support.
# Starting Chef Client, version 14.14.29
# resolving cookbooks for run list:["gitlab"]
# Synchronizing Cookbooks:
# - redis(0.1.0)
# - package(0.1.0)
# - postgresql(0.1.0)
# - mattermost(0.1.0)
# - consul(0.1.0)
# - letsencrypt(0.1.0)
# - gitlab(0.0.1)
# - runit(4.3.0)
# - monitoring(0.1.0)
# - praefect(0.1.0)
# - gitaly(0.1.0)
# - registry(0.1.0)
# - acme(4.1.1)
# - nginx(0.1.0)
# - crond(0.1.0)
# Running handlers:
# Running handlers complete
# Chef Client finished,541/1460 resources updated in03 minutes 14 seconds
# gitlab Reconfigured!

$ gitlab-ctl restart
# [2020- 04- 15 T10:37:51+08:00] WARN: Please install an English UTF-8 locale for Chef to use, falling back to C locale and disabling UTF-8 support.
# ok: run: alertmanager:(pid 21410) 1s
# ok: run: gitaly:(pid 21422) 0s
# ok: run: gitlab-exporter:(pid 21429) 1s
# ok: run: gitlab-workhorse:(pid 21444) 0s
# ok: run: grafana:(pid 21463) 0s
# ok: run: logrotate:(pid 21474) 0s
# ok: run: nginx:(pid 21480) 0s
# ok: run: node-exporter:(pid 21563) 1s
# ok: run: postgres-exporter:(pid 21569) 0s
# ok: run: postgresql:(pid 21580) 0s
# ok: run: prometheus:(pid 21589) 0s
# ok: run: redis:(pid 21601) 0s
# ok: run: redis-exporter:(pid 21708) 0s
# ok: run: sidekiq:(pid 21716) 0s
# ok: run: unicorn:(pid 21727) 0s

Step5. If reconfigure fails, you need to systemctl enable gitlab-runsvdir && systemctl restart gitlab-runsvdir to restart the gitlab-runsvdir service

Step6. Open the browser to initialize the account and set the password. This password is the password of the root administrator account. After setting the password, it will automatically jump to the login page.

application:http://gitlab.weiyigeek.top/
account number:root
password:WeiyiGeek #8 and above

WeiyiGeek.

Step7. The login interface is successful, is there a crazy urge to write code;

WeiyiGeek.

Step8. English may not be friendly to novices who are not good at English, we will Chineseize (Note: version higher than 12.3.5 does not require Chineseization, directly set the custom language in the user settings to select simplified Chinese);

# Stop gitlab
gitlab-ctl stop

# Get the currently installed version patch
git clone https://gitlab.com/xhang/gitlab.git
cd gitlab
# View all branch versions
git branch -a
gitlab_version=$(cat /opt/gitlab/embedded/service/gitlab-rails/VERSION)
# Compare the Chinese label with the original label, export the diff file for patch
# git diff v${gitlab_version} v${gitlab_version}-zh >../${gitlab_version}-zh.diff
git diff remotes/origin/12-3-stable remotes/origin/12-3-stable-zh >../${gitlab_version}-zh.diff

# When patching, it will prompt that some patch files do not exist. Be sure to skip these files. Otherwise, you will get an error when you reconfig.
patch -d /opt/gitlab/embedded/service/gitlab-rails -p1 <../${gitlab_version}-zh.diff

# Recompile and start
gitlab-ctl reconfigure
gitlab-ctl start

Docker

Description: Adopt a low-privileged user to manage docker and deploy gitlab service with docker;

Environmental description:

$lsb_release -a
Description:    Ubuntu 20.04.1 LTS
Release:20.04
Codename:       focal

$docker version
Client: Docker Engine - Community
 Version:19.03.13
 API version:1.40
 Go version:        go1.13.15
 Git commit:        4484c46d9d
 Built:             Wed Sep 1617:02:522020
 OS/Arch:           linux/amd64
 Experimental:false

Server: Docker Engine - Community
 Engine:
 Version:19.03.13
 API version:1.40(minimum version 1.12)
 Go version:       go1.13.15
 Git commit:       4484c46d9d
 Built:            Wed Sep 1617:01:202020
 OS/Arch:          linux/amd64
 Experimental:false
 containerd:
 Version:1.3.7
 GitCommit:        8fba4e9a7d01810a393d5d25a3621dc101981175
 runc:
 Version:1.0.0-rc10
 GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
 Version:0.18.0
 GitCommit:        fec3683

Operating procedures:

# 1. Create a new low-privileged user and add the user to the docker group
sudo useradd -m -g docker WeiyiGeek-gitlab && sudo passwd WeiyiGeek-gitlab
sudo gpasswd -a WeiyiGeek-gitlab docker

# 2. Switch to WeiyiGeek with low permissions-gitlab users pull the latest community version of gitlab
[ email protected]:~$ docker pull gitlab/gitlab-ce
# Using default tag: latest
# latest: Pulling from gitlab/gitlab-ce
# 4 f53fa4d2cf0: Pull complete
# 6 af7c939e38e: Pull complete
# 903 d0ffd64f6: Pull complete
# 04 feeed388b7: Pull complete
# 25 d5e5c7360d: Pull complete
# 0 cc025692f2b: Pull complete
# eac308723fda: Pull complete
# 2135 ce2185ba: Pull complete
# 3 d2db784a8b0: Pull complete
# d2942fac7230: Pull complete
# Digest: sha256:b7daf0c109a62e776f5f72b728a01191059a51f43b5df82c53ef997e877a784f
# Status: Downloaded newer image for gitlab/gitlab-ce:latest
# docker.io/gitlab/gitlab-ce:latest

# 3. Set the persistent directory and run gitlab-server
$ mkdir -vp ~/{config,logs,data}
# mkdir: created directory '/home/WeiyiGeek-gitlab/config'
# mkdir: created directory '/home/WeiyiGeek-gitlab/logs'
# mkdir: created directory '/home/WeiyiGeek-gitlab/data'
$ docker run -d -p 443:443-p 80:80-p 22:22--name gitlab-server --restart always -v /home/WeiyiGeek-gitlab/config:/etc/gitlab -v /home/WeiyiGeek-gitlab/logs:/var/log/gitlab -v /home/WeiyiGeek-gitlab/data:/var/opt/gitlab gitlab/gitlab-ce
512 c45077bafaf1c617ddd6e43f4b8a9a147ca3c8b9e8e889b14d58f151647fe

# 4. Use docker logs to see if gitlab initialization is complete
[ email protected]:~$ docker logs -f --tail 100 512c
==> /var/log/gitlab/puma/puma_stdout.log <=={"timestamp":"2020-10-10T06:14:37.385Z","pid":767,"message":"PumaWorkerKiller: Consuming 3219.65625 mb with master and 4 workers."}

# 5. Host name configuration
sudo tee /etc/hosts <<'EOF'192.168.10.222 gitlab.weiyigeek.top
EOF

PS: How to configure the configuration file address /etc/WeiyiGeek-gitlab/config, please refer to the following article, no more description here;

# Modified key-value pair
sudo egrep -v "^#|^$"/home/WeiyiGeek-gitlab/config/gitlab.rb
[ sudo] password for WeiyiGeek:

external_url 'http://gitlab.WeiyiGeek.top'
user['git_user_name']="Gitlab"
user['git_user_email']="[email protected]"、
gitlab_rails['gitlab_ssh_host']='gitlab.WeiyiGeek.top'
gitlab_rails['gitlab_shell_ssh_port']=2222
gitlab_rails['gitlab_shell_git_timeout']=800
gitlab_rails['time_zone']='Asia/Shanghai'
gitlab_rails['gitlab_email_enabled']=true
gitlab_rails['gitlab_email_from']='[email protected]'
gitlab_rails['gitlab_email_display_name']='Gitlab'
gitlab_rails['gitlab_email_reply_to']='[email protected]'
gitlab_rails['gitlab_email_subject_suffix']='WeiyiGeek-IT'
gitlab_rails['object_store']['enabled']=false
gitlab_rails['object_store']['connection']={}
gitlab_rails['object_store']['storage_options']={}
gitlab_rails['object_store']['proxy_download']=false
gitlab_rails['object_store']['objects']['artifacts']['bucket']= nil
gitlab_rails['object_store']['objects']['external_diffs']['bucket']= nil
gitlab_rails['object_store']['objects']['lfs']['bucket']= nil
gitlab_rails['object_store']['objects']['uploads']['bucket']= nil
gitlab_rails['object_store']['objects']['packages']['bucket']= nil
gitlab_rails['object_store']['objects']['dependency_proxy']['bucket']= nil
gitlab_rails['object_store']['objects']['terraform_state']['bucket']= nil
gitlab_rails['smtp_enable']=true
gitlab_rails['smtp_address']="smtp.exmail.qq.com"
gitlab_rails['smtp_port']=465
gitlab_rails['smtp_user_name']="[email protected]"
gitlab_rails['smtp_password']="Hm595Yb"
gitlab_rails['smtp_domain']="WeiyiGeek.com.top"
gitlab_rails['smtp_authentication']="login"
gitlab_rails['smtp_tls']=true

# Restart the gitlab container after modification
$ docker restart gitlab

0 x02 Operating Instructions####

Gitlab related operations and instructions:

# View Gitlab version
cat /opt/gitlab/embedded/service/gitlab-rails/VERSION

/etc/gitlab/gitlab.rb          #gitlab configuration file
/opt/gitlab                    #gitlab program installation directory
/var/opt/gitlab                #gitlab directory data directory(Via gitlab.The submodule configuration file modified by rb is stored)/var/opt/gitlab/git-data       #Store warehouse data
/var/opt/gitlab/backups        #Default backup directory

# The service structure of Gitlab:
gitlab-shell:Used to process Git commands and modify the authorized keys list
gitlab-workhorse:Lightweight reverse proxy server
logrotate:Log file management tool
nginx:Static web server
postgresql:database
redis:Cache database
sidekiq:Used to execute queue tasks in the background(Asynchronous execution)
unicorn:The GitLab Rails application is hosted on this server.

gitlab-ctl command

Basic commands:

# Common commands
gitlab-ctl reconfigure         #Generate configuration and restart service
gitlab-ctl start               #Start all gitlab components:
gitlab-ctl status              #View the current running status of all services in gitlab
gitlab-ctl stop                #Stop gitlab service
gitlab-ctl pg-upgrade          #Upgrade to the latest version of PostgreSQL

# Separate start|stop|Restart a service
gitlab-ctl start nginx            #Start nginx component
gitlab-ctl stop postgresql        #Stop all gitlab postgresql components:
gitlab-ctl restart unicorn        #Restart related data connection services
gitlab-ctl restart sidekiq        #Restart related data connection services
gitlab-ctl restart  gitlab-workhorse #Restart all gitlab gitlab-workhorse component:

# Log management
gitlab-ctl tail         #View logs of all services
gitlab-ctl tail redis   #Check redis logs in real time (the same goes for extending a certain service)

gitlab-rails command

Basic example:

# gitlab reset password
gitlab-rails console 
> u=User.where(id:1).first        #This is for the administrator, you can also use email, etc.
> u.password ='your_password'    #The password has format restrictions, I only know 8 or more digits, otherwise it will fail to save
> u.password_confirmation ='your_password'> u.save

gitlab-rake command

Basic example:

gitlab-rake gitlab:check SANITIZE=true--trace #Check gitlab
gitlab-rake db:migrate #Database relationship upgrade
gitlab-rake cache:clear #Clean up the redis cache

0 x03 basic configuration####

Gitlab initialization settings#####

0) User theme and language settings
Process: Personal avatar in the upper right corner ->settings->Preferences->Theme settings|Syntax highlighting theme|Custom language and regional related settings

**1) Turn off the Gravatar avatar function for **Network Acceleration
Setting -> Gravatar enabled(Expand Expand) -> not Checked Gravatar enabled -> Sava Changes

WeiyiGeek.

2) User management and registration related settings
Description: Whether users register themselves and set password policies and registered email domain names (white|blacklist) and other information, if closed, the homepage will not display registration;

WeiyiGeek.

Create user: Admin Area-> User -> New User -> Item limit (the number of creations is default) -> Can create group (internal recommended cancellation) -> Create user (user mailbox will receive A registration email)
Impersonate user impersonate: you can imitate user login without logging out and then logging in;

WeiyiGeek.

3) Code push reminder function
Description: Add a project to send an email to the designated mailbox to remind the notification after the code is submitted. This is still helpful for code review, at least know who pushed the new code, what content has been updated, etc.

The administrator logs in to the gitlab interface Admin area-> Service Templates -> Emails on push, check Active, Recipients fill in the people who need to push the update of this project, and separate multiple spaces or commas;

WeiyiGeek.

4) Set the port of the ssh shell
Used to process Git commands and modify the authorized keys list

gitlab_rails['gitlab_shell_ssh_port']=9022

5) Modify the gitlab repository (repositories) storage directory
Description: After installation, the repository (repositories) should be placed on a large hard disk, so you need to modify the directory corresponding to the repository.
( Note: I first delete all repositories and then do the following operations. I also delete all the files under the user name in the /var/opt/gitlab/git-data/repositories/username directory. Only the directories are deleted. Empty folder left)

# View disk space(You can operate according to the disk you added)
$df -h
/dev/mapper/cl-home   24G  954M   24G    4%/home

# Create a new warehouse directory
$mkdir -p /home/gitlab/git-data

# Configure the data storage location of gitlab as/home directory
$vi /etc/gitlab/gitlab.rb
git_data_dirs({"default"=>{"path"=>"/home/gitlab/git-data"}})

# Reload configuration
$sudo gitlab-ctl reconfigure

# After reloading, we found that there is data stored under the warehouse address we changed
$ls /home/gitlab/git-data/
repositories

6) Create a development team
Click Create a group on the main page after logging in to Gitlab -> enter the relevant group information to create;
Invite members to join the group: Members->Group members->Invite memeber -> select users to join the newly created group and role permissions;

WeiyiGeek.

7) Project creation and initialization
The composition of Gitlab's git address is the same as github (don't tell me you don't know): gitlab address + user/group + custom name

There are three levels of visible types of Gitlab projects.

Private project: This level is that only the project owner or authorized people can access the project, or these people are members of the project team.
Internal project: As long as you have a user name and password, can log in to the Gitlab server where the project is located, and can access the project.
Public projects: As long as you know the specific location of the project is the path, you can access the project, They use guest permissions by default.

For example, the operation process of creating a project in the above development group project is as follows:

Enter the development group -> New Project -> set the project name -> create a project
It is recommended to add SSH key to Gitlab without password push and pull [[email protected]](https://weiyigeek.top/cdn-cgi/l/email-protection#6f0f08061b2f08061b030e0d41180a061606080a0a04411b001f): newproject/secopsdev.git: Setting -> SSH;

$ ls ~/.ssh/
id_ed25519  id_ed25519.pub  id_rsa  id_rsa.pub  known_hosts

WeiyiGeek.

Initialization of the project:

# Basic Settings(Prerequisite you need to download git)
git config --global user.name "WeiyiGeek"
git config --global user.email "[email protected]"

# Create a new repository
git clone [email protected]:newproject/secopsdev.git
cd secopsdev
touch README.md
git add README.md
git commit -m "add README"

# Push existing folder
cd existing_folder
git init
git remote add origin http://gitlab.weiyigeek.top/newproject/secopsdev.git
git add .
git commit -m "Initial commit"

# Push existing Git repositories
cd existing_repo
git remote rename origin old-origin  #Rename the local branch
git remote add origin http://gitlab.weiyigeek.top/newproject/secopsdev.git

It should be noted that the default cannot be submitted. This is caused by the protection of the Master branch. If you want to directly make changes to the master branch but do not want to create a sub-branch, you can use the following methods, provided that the user of Maintainers must upload the Master branch After setting;
Project -> Settings -> Repository -> Brach (Branch permission settings) or delete protection;

WeiyiGeek.

8) User project permission control
Rights management understanding:

(1). When creating a new user, you can choose a common authority and administrator authority.
(2). You can associate a group or a member when building a git library
(3). The people added to the group can set the Guest Reporter Developer Master Owner
(4). The same member associated in different git libraries can set different permissions
(5). Different permissions cannot be set for the same group in different git libraries
(6). A git library can only be associated with one group member
(7). A group member can be associated with multiple git repositories with the same permissions
(8). A member can be associated with multiple git libraries or group members, and the permissions can be different
(9). Only above Reporter has code permission
(10). Enterprise access control:
Developer: developer permissions
Code review MDE: master authority

Project group & x project authority management:

Project Name -> Memebers-> User members and Choose a role permission -> and expiration date setting
Refer to the official documentation website lists the permissions corresponding to Guest, Reporter, Developer, Maintainer(Master), Owner, such as our By default, the Dev setting has no push permission to the protected branch;

9) Broadcast information settings
Description: The broadcast information will be displayed to all users. It can be used to notify users of regular system maintenance, upcoming planned upgrades and other information. It will be displayed on the user login interface and when the user submits the code.
Area Admin -> Messages -> Broadcast Information (just set it)

WeiyiGeek.

Gitlab data backup and recovery

Description: GitLab is a version management system of the company's project code. The data is very important, so it must be backed up.

Gitlab provides two kinds of backups:

Local backup
Remote backup

Local backup

#1. The default directory for GitLab backup is
/var/opt/gitlab/backups 

#2. Modify the backup directory
$vim /etc/gitlab/gitlab.rb
gitlab_rails['backup_path']='/home/backups'
# Backup retention for 7 days
gitlab_rails['backup_keep_time']=604800

#3. Reload the configuration after modifying the configuration
$gitlab-ctl reconfigure

#4. Execute backup command
$gitlab-rake gitlab:backup:create 
# 2020- 04- 1609:44:52+0800- - Dumping database ...
# Dumping PostgreSQL database gitlabhq_production ...[DONE]
# 2020- 04- 1609:44:53+0800- - done
# 2020- 04- 1609:44:53+0800- - Dumping repositories ...
# 2020- 04- 1609:44:53+0800- - done
# 2020- 04- 1609:44:53+0800- - Dumping uploads ...
# 2020- 04- 1609:44:53+0800- - done
# 2020- 04- 1609:44:53+0800- - Dumping builds ...
# 2020- 04- 1609:44:53+0800- - done
# 2020- 04- 1609:44:53+0800- - Dumping artifacts ...
# 2020- 04- 1609:44:53+0800- - done
# 2020- 04- 1609:44:53+0800- - Dumping pages ...
# 2020- 04- 1609:44:53+0800- - done
# 2020- 04- 1609:44:53+0800- - Dumping lfs objects ...
# 2020- 04- 1609:44:53+0800- - done
# 2020- 04- 1609:44:53+0800- - Dumping container registry images ...
# 2020- 04- 1609:44:53+0800- - [ DISABLED]
# Creating backup archive: 1587001493_2020_04_16_12.9.2_gitlab_backup.tar ... done
# Uploading backup archive to remote storage  ... skipped

#5. View backup files,The compressed package includes the entire complete part of GitLab,Approximate format xxxxxxxx_gitlab_backup.tar
$ls /var/opt/gitlab/backups
1587001493_2020_04_16_12.9.2_ gitlab_backup.tar

Supplementary note:

Realize automatic backup through task plan crontab#Enter the command crontab -e sudo crontab -e 0 2 * * * /opt/gitlab/bin/gitlab-rake gitlab:backup:create #Restart the service systemctl restart crond
It is worth noting that because gitlab.rb and gitlab-secrets.json contain sensitive data, they cannot be backed up in the form of the above backup. They can only be backed up by manual backup;

Remote backup
Description: In fact, it is to add the pub public key of the backup server on the gitlab server, and then the backup server accesses the Gitlab server for pulling. Of course, you can also use rsync for backup;

# Suppose the environment is as follows:
# Backup server:192.168.1.2
# GitLab server:192.168.1.250

#1. Backup server creation authorized_keys file
mkdir -p /root/.ssh
touch /root/.ssh/authorized_keys 
chmod 400/root/.ssh/authorized_keys

#2. The Gitlab server generates the rsa public key and uploads the certificate to the backup server (the least-privileged user is recommended in the actual environment)
$ssh-keygen -t rsa -C "[email protected]"
# Enter file in which to save the key(/root/.ssh/id_rsa):Error path
# Enter passphrase(empty for no passphrase):Public key password
# Enter same passphrase again:verify password
# Your public key has been saved in/root/.ssh/id_rsa.pub. #save route
# The key fingerprint is:
# SHA256:YahioDs7XZ9No********RS9jWX1NNXo04i4ZHOLY [email protected]
# The key's randomart image is:
# +- - - [ RSA 2048]----+
# |..++|
# |.+. o o.+|
# |. o.oo=.+.|
# |....=. E.++|
# | o o..+ oS=..|
# |.=+.+ o ..|
# |+*..=+|
# | oo= o o .|
# |. o.+|
# +- - - - [ SHA256]-----+

$ls /root/.ssh/
id_rsa  id_rsa.pub

$scp /root/.ssh/id_rsa.pub [email protected]:/root/.ssh/

#3. Backup server will id_rsa.pub file content is appended to authorized_keys file, and modify authorized_The permissions of the keys file
$cat /root/.ssh/id_rsa.pub >>/root/.ssh/authorized_keys

#4. gitlab backup script(Main executive authority appeal)Copy the backup file to the server
$cat >/home/gitlab/auto_backup_to_remote.sh<<'END'
DATE=`date +"%Y_%m_%d"`
BackupIP=192.168.1.2
BackupUser=root
BackupPATH=/var/opt/gitlab/backups

# Delete the local backup 7 days ago
find ${BackupPATH}-type f -mtime +7-name '*.tar'-exec rm {} \;

# If you do not need to back up files, you can also back up the warehouse directory
find ${BackupPATH}-name *${DATE}*-exec scp -r {} ${root}@${BackupIP}:/tmp/ \;
END

$chmod +x /home/gitlab/auto_backup_to_remote.sh

#5. Add time schedule(Execute at 1 am every day), Restart crond service
cat >>/etc/crontab <<END
01***/home/gitlab/auto_backup_to_remote.sh
END

systemctl restart crond

Gitlab data recovery
For example: above we have to backup: /var/opt/gitlab/backups/1587001493_2020_04_16_12.9.2_gitlab_backup.tar;
The operation process is as follows:

# Stop related data connection services (to ensure that there is no new connection to the database and there will be no write data)
gitlab-ctl stop unicorn
gitlab-ctl stop sidekiq

# Specify the recovery file will automatically go to the backup directory to find the format:1587001493_2020_04_16_12.9.2
cd /var/opt/gitlab/backups/&& chmod -R 777*
gitlab-rake gitlab:backup:restore BACKUP=1587001493_2020_04_16_12.9.2

# Start Gitlab
gitlab-ctl restart

Gitlab add email address#####

Email password recovery:

Because gitlab is running all the time, it is necessary to record and back up the configuration file before modification;

cp /etc/gitlab/gitlab.rb{,.bak}

Modify the configuration file: /etc/gitlab/gitlab.rb Here use the company's corporate mailbox to send emails;

$egrep -v "^$|^#"/etc/gitlab/gitlab.rb 
external_url 'http://gitlab.weiyigeek.top'
gitlab_rails['time_zone']='Asia/Shanghai'
gitlab_rails['gitlab_email_enabled']=true
gitlab_rails['gitlab_email_from']='[email protected]'
gitlab_rails['gitlab_email_display_name']='Gitlab CE'
gitlab_rails['gitlab_email_reply_to']='[email protected]'
gitlab_rails['smtp_enable']=true
gitlab_rails['smtp_address']="smtp.exmail.qq.com"
gitlab_rails['smtp_port']=465
gitlab_rails['smtp_user_name']="[email protected]"
gitlab_rails['smtp_password']="email Password"
gitlab_rails['smtp_domain']="weiyigeek.top"
gitlab_rails['smtp_authentication']="login"
gitlab_rails['smtp_enable_starttls_auto']=true
gitlab_rails['smtp_tls']=true
user['git_user_email']="[email protected]"

Save the modification and update the configuration;

# Just refresh the modified configuration file information to the configuration file
gitlab-ctl reconfigure

In fact, a configuration file gitlab.rb is modified above. The configuration in it will be called by this command and written into multiple configuration files to the installation directory: /var/opt/gitlab/gitlab-rails/etc/

$ls /var/opt/gitlab/gitlab-rails/etc/
database.yml         gitlab_shell_secret      gitlab.yml      resque.yml   smtp_settings.rb #You can also directly modify
gitlab_pages_secret  gitlab_workhorse_secret  rack_attack.rb  secrets.yml  unicorn.rb

If you modify the smtp_settings.rb file directly, just restart gitlab-ctl restart.

Console view configuration and test sending

$gitlab-rails console
--------------------------------------------------------------------------------
 GitLab:12.9.2(ac5568eb5d8) FOSS
 GitLab Shell:12.0.0
 PostgreSQL:10.12--------------------------------------------------------------------------------
Loading production environment(Rails 6.0.2)irb(main):001:0> ActionMailer::Base.delivery_method
=>: smtp

irb(main):002:0> ActionMailer::Base.smtp_settings
=>{: authentication=>:login,:address=>"smtp.exmail.qq.com",:port=>465,:user_name=>"[email protected]",:password=>"Pa****20",:domain=>"weiyigeek.top",:enable_starttls_auto=>true,:tls=>true,:ca_file=>"/opt/gitlab/embedded/ssl/certs/cacert.pem"}irb(main):003:0> Notify.test_email('[email protected]','GitLab test mail','GitLab page information').deliver_now
Notify#test_email: processed outbound mail in2.0ms
Delivered mail [email protected](1543.7ms)
Date: Wed,15 Apr 202021:12:37+0800
From: Gitlab CE <[email protected]>
Reply-To: Gitlab CE <[email protected]>

After completing the above operations, go back to the login page and select Forgot your password? Enter the email address of your account to receive the email, and modify the new password according to the email address;

WeiyiGeek.sendemail

Supplementary note:*

A gitlab repository is configured on some ECS private network hosts, because by default these private network ECS hosts are disabled with port 25, so the configuration of gitlab.rb has been changed accordingly, using the 465 protocol. #Only modify the following two lines, and update the configuration after modification gitlab_rails['smtp_port'] = 25 gitlab_rails['smtp_tls'] = true

Gitlab add LDAP authentication

LDAP configuration instructions: http://gitlab.weiyigeek.top/help/administration/auth/ldap

Step1. Modify the gitlab configuration file to set the parameters for connecting to the LDAP server, that is, the DN in bindPass

vim /etc/gitlab/gitlab.rb
gitlab_rails['ldap_enabled']=true
# gitlab_rails['prevent_ldap_sign_in']=false
###!**remember to close this block with'EOS' below**
gitlab_rails['ldap_servers']= YAML.load <<-'EOS'
 main: # 'main' is the GitLab 'provider ID'ofthis LDAP server
  label:'LDAP'
  host:'10.10.107.245'
  port:389
  timeout:10
  uid:'uid'
  bind_dn:'cn=admin,dc=WeiyiGeek,dc=com,dc=cn'
  password:'WeiyiGeek'
  encryption:'plain' # "start_tls" or "simple_tls" or "plain"
  verify_certificates:false
  smartcard_auth:false
  active_directory:true
  allow_username_or_email_login:true
  lowercase_usernames:false
  block_auto_created_users:false
  base:'dc=WeiyiGeek,dc=com,dc=cn'
  # user_filter:'(&(objectclass=inetOrgPerson)(memberof=cn=gitlab,ou=Group,dc=WeiyiGeek,dc=com,dc=cn))'
  attribute:
  username:['uid']
  email:['mail']
  name:'cn'
  first_name:['sn']
EOS

Step2. LDAP establishes groupOfUniqueNames attribute to obtain user groups and bind users;

dn: cn=gitlab,ou=Group,dc=WeiyiGeek,dc=com,dc=cn
uniqueMember: uid=gituser,ou=People,dc=WeiyiGeek,dc=com,dc=cn

dn: uid=gituser,ou=People,dc=WeiyiGeek,dc=com,dc=cn
objectClass: posixAccount
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: person
loginShell:/bin/bash
homeDirectory:/home/gituser
userPassword:: e1NTSEF9UGVyM21xc1dJcnV3K1d2bWRiVmVpd3RWZHVVeVN6Tks=
uid: gituser
cn: gituser
uidNumber:10000
gidNumber:10000
sn: gituser
mail:[email protected]

Step3. Login test if the LDAP user does not exist or the filter entry does not exist, an error will be reported Could not authenticate you from Ldapmain because "Invalid credentials for jenkuser".

WeiyiGeek.

Step4. The login success interface is as follows:

WeiyiGeek.

0 x04 Into the pit solution

(0) Common mistakes of user pull and push code to gitlab

Error 1: The requested URL returned error: 403
Reason: It means that we don't have permission to pull/push related code and need to modify the permission of the related gitlab account to Development or Maintianer;
Error 2: To gitlab.weiyigeek.top:newproject/secopsdev.git! [Remote rejected] master -> master (pre-receive hook declined) error: failed to push some refs to'[email protected]:newproject/secopsdev.git '
Reason: Because the gitlab user permission for uploading code is the developer, the default is to create a private project. The default is to turn on the protection mechanism for the master branch. You need to set the Branch branch of the project to Allowed to merge | Allowed to push as Development +Maintiner;#The final master branch and dev branch are tested successfully last time remote: To gitlab.weiyigeek.top:newproject/secopsdev.git * [new branch] dev -> dev Branch'dev' set up to track remote branch'dev 'from'origin'.

(1) Forgotten gitlab password reset method

Method 1: Reset via gitlab-rails
$gitlab-rails console Loading production environment (Rails 6.0.2)> u=User.where(id:1).first #This is for the administrator, you can also use email, etc.> u.password ='your_password' #Password has Format limitation, I only know that more than 8 digits, otherwise it will fail to save> u.password_confirmation ='your_password'> u.save
Method 2: Retrieve the administrator by initializing the mail setting or check the unsent mail in the local postfix, here is the root user’s [[email protected]](https://weiyigeek.top/cdn-cgi/l/email -protection#9afbfef7f3f4daffe2fbf7eaf6ffb4f9f5f7) as an example, generally the default is not changed (just a method that does not make much sense in practice);

It is assumed here that the service cannot be linked to the external network (in order to reproduce the following operations);

# (1) View the mail queue that cannot be sent
$postqueue -p
- Queue ID---Size------Arrival Time-----Sender/Recipient-------
# Because there is no such address, it cannot be sent out and can only stay in the mail queue.
908 DCCAC62     6871 Wed Apr 1519:54:32[email protected] #(Actually, we retrieve the password email)(Host or domain name not found. Name service error for name=example.com type=MX: Host not found,try again)[email protected]

# (2) Mail queue storage location
$postconf -d | grep queue_directory  #There is no modification by default
queue_directory =/var/spool/postfix

# (3) Go to the directory and find the queue ID number of the mail above
$find /var/spool/postfix/-name 908DCCAC62
/var/spool/postfix/defer/9/908DCCAC62
/var/spool/postfix/deferred/9/908DCCAC62

# (4) Use the tool that comes with postfix to view the content of the email and find the place to change the password
$postcat /var/spool/postfix/deferred/9/908DCCAC62
*** ENVELOPE RECORDS /var/spool/postfix/deferred/9/908DCCAC62 ***
message_size:68712741068710
message_arrival_time: Wed Apr 1519:54:322020
create_time: Wed Apr 1519:54:322020
named_attribute: rewrite_context=local
sender:[email protected]
named_attribute: dsn_orig_rcpt=rfc822;[email protected]
original_recipient:[email protected].....
If you did not perform this request, you can safely ignore this email.
Otherwise, click the link below to complete the process:
# Find the following password to modify the connection, and copy it out to the browser to reset the password
http://gitlab.weiyigeek.top/users/password/edit?reset_password_token=gd9MyL7FaSt5R2F3_qA_

WeiyiGeek.

(2) 502 error caused by GitLab database and its solution
Description: Open GitLab's homepage address http:/gitlab.weiyigeek.top/ error 502, restart or check the status and find that it is normal sudo gitlab-ctl status, and check that the port number is not occupied, check the refresh configuration gitlab- ctl reconfigure found an error;

The error message is as follows:

# gitlab-ctl reconfigure 
Running handlers:
There was an error running gitlab-ctl reconfigure:

bash[migrate gitlab-rails database](gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with[0], but received '1'---- Begin output of"bash""/tmp/chef-script20190308-65247-12ck9rp"----
STDOUT: rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
 Is the server running locally and accepting
 connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:52:in`block (3 levels) in <top (required)>
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure(See full trace by running task with--trace)
STDERR:---- End output of"bash""/tmp/chef-script20190308-65247-12ck9rp"----
Ran "bash""/tmp/chef-script20190308-65247-12ck9rp" returned 1

# gitlab-ctl tail

WeiyiGeek.

(3) gitlab-ce version upgrade record
In order to ensure data security, we will slowly upgrade step by step. Using official sources may be slower. You can use domestic sources (using Omnibus to upgrade directly to rpm package deployment): https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum

Upgrade method: first upgrade to the last version of the current major version (here is my suggestion, generally the last few versions are fine), and then upgrade across major versions, first upgrade from the lower version of the major version, and then to the end Version upgrade, and so on; because I tried to upgrade directly across too many versions, and there were exceptions. In fact, it was mainly the data import and backup problem of the database. If the new version is too much related to the old version, an error will be reported.

For example, my upgrade process is very long and I also did a simple test

# upgraded version
8.3.1-->8.3.7-->8.8.5-->8.17.4-->9.5.6-->10.0.6-->10.8.5-->11.0.3

# Out of service
gitlab-ctl stop unicorn
gitlab-ctl stop sidekiq
gitlab-ctl stop nginx

# Create data backup to prevent update failure(Very important)
gitlab-rake gitlab:backup:create

# Install the upgrade package
# yum update gitlab-ce #Upgrade GitLab-ce version(This method is not recommended)
rpm -Uvh gitlab-7.4.2_omnibus-1.el6.x86_64.rpm 

# Reload configuration&Restart Gitlab service
gitlab-ctl reconfigure
gitlab-ctl restart

For example, whether there is any abnormality in the data backed up by the project, user information, response speed, etc., I personally think that it is relatively reliable.

In fact, as long as we have upgraded before, we won't be like this anymore. We upgraded from such a low version all at once. It is generally recommended to keep the same version number. Too new may have bugs, and too old may have many historical problems.

After the upgrade is complete, you will find several more processes. It may have higher requirements for system hardware, such as memory requirements.
Suggestion: If you simply want a code repository for non-essential functional requirements, it would be good to use it personally between 11.x.-12.3.x.

**(4) gitlab-rails database failed to initialize **
Environmental description

Centos7:3.10.0-327.el7.x86_64
gitlab version: the original is 8.8.Version 5 is now upgraded to a newer version;

Problem: After uninstalling the old version of gitlab on the same Centos7 machine, and then reinstalling the new version of gitlab-ce, the configuration was modified after the installation, and the following error occurred when initializing the configuration:

Running handlers:
There was an error running gitlab-ctl reconfigure:

bash[migrate gitlab-rails database](gitlab::database_migrations line 49) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with[0], but received '1'---- Begin output of"bash""/tmp/chef-script20190628-7065-vx17en"----
STDOUT: rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
 Is the server running locally and accepting
 connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:49:in`block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure(See full trace by running task with--trace)
STDERR:---- End output of"bash""/tmp/chef-script20190628-7065-vx17en"----
Ran "bash""/tmp/chef-script20190628-7065-vx17en" returned 1

Reason: Because the old data was not deleted when the old version was uninstalled, when installing directly, the program detects the configuration file and skips the data file without overwriting. The new version uses the data file of the old version [because the version is too different More] will not be recognized.

Solution: After uninstalling the old version, remember to delete the old installation data configuration file, installation directory, data directory, etc, and reinstall the new version of gitlab-ce to start normally.

rm -rf /var/opt/gitlab//opt/gitlab//etc/gitlab/

(5) Gitlab database postgresql updates account information
Problem description: Because the internal Ldap authentication server hard disk is broken, the Ldap service running on it cannot perform centralized authentication on the Jenkins and Gitlab platforms. As a result, the account on Gitlab cannot log in to the platform, so it cannot submit the pull Code too.

Solutions:*

Method 1: Reconfigure an Ldap service to import the previous database, migrate to the new machine, and modify the Gitlab authentication service address.
Method 2: Modify the Gitlab database directly and modify the authentication method of the account.

The process of selecting method 2 for operation is as follows:

Step1. Modify the database configuration of gitlab and enable remote access (default local access) 12345678910111213141516171819202122232425262728#1. Modify the configuration file and add the last two lines egrep -v "^#|^

After connecting, find the corresponding identities table and modify the corresponding field. In fact, it is to modify the user_id in the field. What I do here is to change the user_id of the user to a negative number, so that it will not be found during Ldap authentication, so standard authentication (This is done without restarting Gitlab).

WeiyiGeek.

Supplementary note:

After the update, if the user still has the problem of not being able to log in, at this time, the user can log in to gitlab on the Web side, choose to forget the password, and reset a password by forgetting the password. (The premise is that you have realized the password retrieval function)
After completing all operations, remember to comment out or delete the remote configuration, and then restart the postgresql service.

0 x05 Script Appendix####

GitLab one-click installation configuration script

#! /bin/bash
# Desc:Gitlab code server automated deployment
# Author:WeiyiGeek
# SupportOS:CentOS7 / CentOS8

GITLAB_BASEDOMAIN=weiyigeek.top
GITLAB_VERSION=12.9.2
GITlABOS7=gitlab-ce-${GITLAB_VERSION}-ce.0.el7.x86_64.rpm
GITlABOS8=gitlab-ce-${GITLAB_VERSION}-ce.0.el8.x86_64.rpm

GITLABRUNNER_VERSION=12.9.0-1
GITLABRUNNER_NAME=gitlab-runner-${GITLABRUNNER_VERSION}.x86_64.rpm	
CheckOSVersion=$(uname -r | grep -c el8)

## [ Mirror source settings]functionrepoChange(){
 mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.$(date +"%Y%m%d").backup
 if[ $CheckOSVersion -eq 1];then
 # CentOS8 source
 curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-8.repo
 # Install the epel configuration package and replace the address with the address of the Aliyun mirror station
 dnf install -y https://mirrors.aliyun.com/epel/epel-release-latest-8.noarch.rpm
 sed -i 's|^#baseurl=https://download.fedoraproject.org/pub|baseurl=https://mirrors.aliyun.com|'/etc/yum.repos.d/epel*
 sed -i 's|^metalink|#metalink|'/etc/yum.repos.d/epel*
 dnf clean all
 dnf makecache
 else
 # CentOS7 source
 curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
 sed -i -e '/mirrors.cloud.aliyuncs.com/d'-e '/mirrors.aliyuncs.com/d'/etc/yum.repos.d/CentOS-Base.repo
 wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
# Add to trust the GPG public key in GitLab
sudo cat >/etc/yum.repos.d/gitlab-ce.repo <<EOF
[ gitlab-ce]
name=Gitlab CE Repository
baseurl=https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7/
gpgcheck=0
enabled=1
gpgkey=https://packages.gitlab.com/gpg.key
EOF
 yum clean all
 yum makecache
 fi
}

#[ yum installation]functionyumInstall(){
 # View available versions neng'b
 yum list gitlab-ce --showduplicates
 # Install the latest version by default
 yum install -y gitlab-ce
 # Install specified version 12.3.5
 # yum install gitlab-ce-12.3.5-ce.0.el7.x86_64.rpm
}

#[ rpm installation-Recommended way]functionOmnibusInstall(){if[ $CheckOSVersion -eq 1];then
 wget -O $GITlABOS8 https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el8/${GITlABOS8}
 rpm -i $GITlABOS8
 else
 wget -O $GITlABOS7 https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7/${GITlABOS7}
 rpm -i  $GITlABOS7
 fi
} functiongitlabSetting(){
 sed -i "s#example.com#${GITLAB_BASEDOMAIN}#g"/etc/gitlab/gitlab.rb
 echo "127.0.0.1 gitlab.${GITLAB_BASEDOMAIN}">/etc/hosts
} functionuseage(){
 echo -e "\e[32m# Description:Gitlab automated installation and deployment script"
 echo -e "usage: $0 [rpm|yum] #Specify rpm installation or yum installation"
 echo -e "Author:WeiyiGeek\e[0m"}

#[ Less than 12.3.Set only for x version]functionChinesization(){
 # Stop gitlab
 gitlab-ctl stop

 # Get the currently installed version patch
 git clone https://gitlab.com/xhang/gitlab.git
 cd gitlab
 gitlab_version=$(cat /opt/gitlab/embedded/service/gitlab-rails/VERSION)

 # Generate the corresponding version patch file
 git diff remotes/origin/12-3-stable remotes/origin/12-3-stable-zh >../${gitlab_version}-zh.diff

 # When patching, it will prompt that some patch files do not exist. Be sure to skip these files. Otherwise, you will get an error when you reconfig.
 patch -d /opt/gitlab/embedded/service/gitlab-rails -p1 <../${gitlab_version}-zh.diff
 gitlab-ctl reconfigure
 gitlab-ctl restart
}

## [ Installation configuration script entry function]functionmain(){
 # Turn off Sellinux
 echo "Current Selinux: $(getenforce)"
 setenforce 0
 sed -i 's/SELINUX=enforcing/SELINUX=disabled/g'/etc/selinux/config
 echo "Set up Selinux: $(getenforce)"

 # Environment dependent installation
 repoChange
 sudo yum install -y curl policycoreutils openssh-server wget postfix git htop ncdu net-tools
 systemctl enable postfix
 systemctl start postfix

 # Protective wall setting
 sudo firewall-cmd --permanent --add-service=http
 sudo firewall-cmd --permanent --add-service=https
 sudo systemctl reload firewalld

 # Choose installation method
 if[ $1=="rpm"];then
 OmnibusInstall
 elif [ $1=="yum"];then 
 yumInstall
 else
 usage
 fi

 gitlabSetting
 gitlab-ctl reconfigure
 gitlab-ctl start
 gitlab-ctl status
}

#[ Parameter verification]if[ $# -ne 1];then
 usage
else 
 main $1
fi