Prerequisites
同之前的 prequistes，只针对 Linux package (Omnibus) 的相关说明
升级
确定方案
个人使用的话，无所谓在乎测试与生产环境
官方建议确保知晓
- GitLab 的最初安装方式
- node 的操作系统
- 确定是单 node 还是在多 node 上安装的 GitLab 实例
- 以及其它未提及的已知问题
升级
用 apt 安装的话升级比较简单
# Ubuntu/Debian
sudo apt update && sudo apt install gitlab-ee
# RHEL/CentOS 7 and Amazon Linux 2
sudo yum install gitlab-ee
# RHEL/Almalinux 8/9 and Amazon Linux 2023
sudo dnf install gitlab-ee
# SUSE
sudo zypper install gitlab-eePreparing to unpack .../23-gitlab-ee_16.10.0-ee.0_arm64.deb ...
gitlab preinstall: Automatically backing up only the GitLab SQL database (excluding everything else!)
2024-03-21 15:50:04 UTC -- Dumping database ...
Dumping PostgreSQL database gitlabhq_production ... [DONE]
2024-03-21 15:50:10 UTC -- Dumping database ... done
2024-03-21 15:50:10 UTC -- Dumping repositories ... [SKIPPED]
2024-03-21 15:50:10 UTC -- Dumping uploads ... [SKIPPED]
2024-03-21 15:50:10 UTC -- Dumping builds ... [SKIPPED]
2024-03-21 15:50:10 UTC -- Dumping artifacts ... [SKIPPED]
2024-03-21 15:50:10 UTC -- Dumping pages ... [SKIPPED]
2024-03-21 15:50:10 UTC -- Dumping lfs objects ... [SKIPPED]
2024-03-21 15:50:10 UTC -- Dumping terraform states ... [SKIPPED]
2024-03-21 15:50:10 UTC -- Dumping container registry images ... [DISABLED]
2024-03-21 15:50:10 UTC -- Dumping packages ... [SKIPPED]
2024-03-21 15:50:10 UTC -- Dumping ci secure files ... [SKIPPED]
2024-03-21 15:50:10 UTC -- Creating backup archive: 1711036204_2024_03_21_16.9.2-ee_gitlab_backup.tar ...
2024-03-21 15:50:10 UTC -- Creating backup archive: 1711036204_2024_03_21_16.9.2-ee_gitlab_backup.tar ... done
2024-03-21 15:50:10 UTC -- Uploading backup archive to remote storage microcharon/backup ...
[fog][WARNING] fog: the specified s3 bucket name(microcharon/backup) is not a valid dns name, which will negatively impact performance. For details see: https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html
2024-03-21 15:50:11 UTC -- Uploading backup archive to remote storage microcharon/backup ... done (encrypted with AES256)
2024-03-21 15:50:11 UTC -- Deleting old backups ... [SKIPPED]
2024-03-21 15:50:11 UTC -- Deleting tar staging files ...
2024-03-21 15:50:11 UTC -- Cleaning up /var/opt/gitlab/backups/backup_information.yml
2024-03-21 15:50:11 UTC -- Cleaning up /var/opt/gitlab/backups/db
2024-03-21 15:50:11 UTC -- Deleting tar staging files ... done
2024-03-21 15:50:11 UTC -- Deleting backups/tmp ...
2024-03-21 15:50:11 UTC -- Deleting backups/tmp ... done
2024-03-21 15:50:11 UTC -- Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data
and are not included in this backup. You will need these files to restore a backup.
Please back them up manually.
2024-03-21 15:50:11 UTC -- Backup 1711036204_2024_03_21_16.9.2-ee is done.
2024-03-21 15:50:11 UTC -- Deleting backup and restore PID file ... done
gitlab preinstall: Automatically backing up /etc/gitlab
Could not find '/etc/gitlab/config_backup' directory. Creating.
Running configuration backup
Creating configuration backup archive: gitlab_config_1711036212_2024_03_21.tar
/etc/gitlab/
/etc/gitlab/ssl/
/etc/gitlab/ssl/code.microcharon.dev.key
/etc/gitlab/ssl/microbin.dev.crt
/etc/gitlab/ssl/microbin.dev.key
/etc/gitlab/ssl/letsencrypt_account_private_key.pem
/etc/gitlab/ssl/code.microcharon.dev.crt
/etc/gitlab/ssl/code.microcharon.dev.key-staging
/etc/gitlab/trusted-certs/
/etc/gitlab/gitlab-secrets.json
/etc/gitlab/gitlab.rb
Configuration backup archive complete: /etc/gitlab/config_backup/gitlab_config_1711036212_2024_03_21.tar
Keeping all older configuration backups
Unpacking gitlab-ee (16.10.0-ee.0) over (16.9.2-ee.0) ...卸载
此部分讲解如何卸载 GitLab 的所有资料文件并不留残留
sudo gitlab-ctl stop
sudo apt-get purge gitlab-ee
sudo rm -rf /etc/gitlab
sudo rm -rf /var/opt/gitlab
sudo rm -rf /var/log/gitlab
sudo apt-get autoremove备份与恢复
此部分讲解如何备份 GitLab 的资料以及如何恢复，并测试该备份系统
备份
备份分为配置备份 (configuration backup) 和应用备份 (application backup)
配置备份
配置备份路径固定在 /etc/gitlab/config_backup 下
配置备份中官方提供备份文件生命周期设置，取消此行注释并 gitlab-ctl reconfigure
## Limit backup lifetime to 7 days - 604800 seconds
gitlab_rails['backup_keep_time'] = 604800由此重新备份配置时 (gitlab-ctl backup-etc) 若带上参数 --delete-old-backups 即出现如下输出
root@hetzner-fsn1-002:/etc/gitlab# gitlab-ctl backup-etc --delete-old-backups
Running configuration backup
Creating configuration backup archive: gitlab_config_1713364561_2024_04_17.tar
/etc/gitlab/
/etc/gitlab/ssl/
/etc/gitlab/ssl/code.microcharon.dev.key
/etc/gitlab/ssl/microbin.dev.crt
/etc/gitlab/ssl/microbin.dev.key
/etc/gitlab/ssl/letsencrypt_account_private_key.pem
/etc/gitlab/ssl/code.microcharon.dev.crt
/etc/gitlab/ssl/code.microcharon.dev.key-staging
/etc/gitlab/trusted-certs/
/etc/gitlab/gitlab-secrets.json
/etc/gitlab/gitlab.rb
Configuration backup archive complete: /etc/gitlab/config_backup/gitlab_config_1713364561_2024_04_17.tar
Removing configuration backups older than 2024-04-10 22:36:01 +0800 ...
Removed /etc/gitlab/config_backup/gitlab_config_1711036212_2024_03_21.tar
Removed /etc/gitlab/config_backup/gitlab_config_1711950867_2024_04_01.tar应用数据备份
应用数据备份默认在 /var/opt/gitlab/backups 下，最简单的备份指令如下
gitlab-backup create配置 S3 备份，这里也只推荐用 AWS 的 S3 服务，当然其它 provider 也可使用，见官方 doc
gitlab_rails['backup_upload_connection'] = {
'provider' => 'AWS',
'region' => 'us-east-1',
'aws_access_key_id' => 'your_access_key',
'aws_secret_access_key' => 'your_secret_key',
# # # If IAM profile use is enabled, remove aws_access_key_id and aws_secret_access_key
# 'use_iam_profile' => false
}
gitlab_rails['backup_upload_remote_directory'] = 'your_s3_bucket_name'
# gitlab_rails['backup_multipart_chunk_size'] = 104857600恢复
先恢复配置文件，再恢复应用数据
配置文件的恢复
文件 gitlab-secrets.json 和 gitlab.rb 还原至 /etc/github/ 目录下，若有 TLS keys 及 SSL 证书则可选还原（/etc/gitlab/ssl, /etc/gitlab/trusted-certs）
成功恢复配置文件后至少执行 1 次 gitlab-ctl reconfigure
应用数据的恢复
继配置文件所有恢复后，再开始进行应用数据的恢复
#默认设置的 backup_path 为 /var/opt/gitlab/backups
cp /your/file/path/<backup-id>_gitlab_backup.tar /var/opt/gitlab/backups/
#默认权限为 git
chown git:git /your/file/path/<backup-id>_gitlab_backup.tar停止相关的进程
gitlab-ctl stop puma
gitlab-ctl stop sidekiq
#验证状态
sudo gitlab-ctl status# 此命令将覆盖 GitLab 数据库的内容
# 注意：名称中省略了“_gitlab_backup.tar”，backup-id 如 1715929662_2024_05_17_17.0.0-ee
gitlab-backup restore BACKUP=<backup-id>重启并检查 GitLab
gitlab-ctl restart
gitlab-rake gitlab:check SANITIZE=true实际操作
光知道如何备份以及如何恢复还不够，需要实际上手才知道完整流程，以避免未来的意外事件以及因灾难导致的数据丢失等问题
这里将会以另一个 GitLab 实例（AWS EC2）作为例子进行示范
确定 GitLab 实例版本
毕竟有要求在这里：Restore GitLab | GitLab
一定要确认新安装的 GitLab 实例版本要与原 GitLab 实例的版本相同
root@ip-172-31-31-185:~# gitlab-rake gitlab:env:info
System information
System: Debian 12
Proxy: no
Current User: git
Using RVM: no
Ruby Version: 3.1.5p253
Gem Version: 3.5.9
Bundler Version:2.5.9
Rake Version: 13.0.6
Redis Version: 7.0.15
Sidekiq Version:7.1.6
Go Version: unknown
GitLab information
Version: 17.0.0-ee
Revision: 8c75d0bf4a4
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 14.11
URL: https://git.microcharon.com
HTTP Clone URL: https://git.microcharon.com/some-group/some-project.git
SSH Clone URL: [email protected]:some-group/some-project.git
Elasticsearch: no
Geo: no
Using LDAP: no
Using Omniauth: yes
Omniauth Providers:
GitLab Shell
Version: 14.35.0
Repository storages:
- default: unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Gitaly
- default Address: unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version: 17.0.0
- default Git Version: 2.44.0.gl2备份配置与应用数据文件
root@ip-172-31-31-185:~# gitlab-ctl backup-etc
Could not find '/etc/gitlab/config_backup' directory. Creating.
Running configuration backup
Creating configuration backup archive: gitlab_config_1715929627_2024_05_17.tar
/etc/gitlab/
/etc/gitlab/gitlab-secrets.json
/etc/gitlab/ssl/
/etc/gitlab/ssl/git.microcharon.com.key
/etc/gitlab/ssl/git.microcharon.com.crt-staging
/etc/gitlab/ssl/git.microcharon.com.key-staging
/etc/gitlab/ssl/git.microcharon.com.crt
/etc/gitlab/gitlab.rb
/etc/gitlab/initial_root_password
/etc/gitlab/trusted-certs/
Configuration backup archive complete: /etc/gitlab/config_backup/gitlab_config_1715929627_2024_05_17.tar
Keeping all older configuration backups
root@ip-172-31-31-185:~# gitlab-backup create
2024-05-17 07:07:42 UTC -- Dumping database ...
2024-05-17 07:07:42 UTC -- Dumping PostgreSQL database gitlabhq_production ...
2024-05-17 07:07:46 UTC -- [DONE]
2024-05-17 07:07:46 UTC -- Dumping database ... done
2024-05-17 07:07:46 UTC -- Dumping repositories ...
{"command":"create","gl_project_path":"root/test","level":"info","msg":"started create","pid":25107,"relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git","storage_name":"default","time":"2024-05-17T07:07:46.852Z"}
{"command":"create","gl_project_path":"root/test","level":"info","msg":"completed create","pid":25107,"relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git","storage_name":"default","time":"2024-05-17T07:07:46.887Z"}
{"command":"create","gl_project_path":"root/test.wiki","level":"info","msg":"started create","pid":25107,"relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.wiki.git","storage_name":"default","time":"2024-05-17T07:07:46.921Z"}
{"command":"create","gl_project_path":"root/test.wiki","level":"info","msg":"completed create","pid":25107,"relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.wiki.git","storage_name":"default","time":"2024-05-17T07:07:46.933Z"}
2024-05-17 07:07:46 UTC -- Dumping repositories ... done
2024-05-17 07:07:46 UTC -- Dumping uploads ...
2024-05-17 07:07:46 UTC -- Dumping uploads ... done
2024-05-17 07:07:46 UTC -- Dumping builds ...
2024-05-17 07:07:46 UTC -- Dumping builds ... done
2024-05-17 07:07:46 UTC -- Dumping artifacts ...
2024-05-17 07:07:46 UTC -- Dumping artifacts ... done
2024-05-17 07:07:46 UTC -- Dumping pages ...
2024-05-17 07:07:46 UTC -- Dumping pages ... done
2024-05-17 07:07:46 UTC -- Dumping lfs objects ...
2024-05-17 07:07:46 UTC -- Dumping lfs objects ... done
2024-05-17 07:07:46 UTC -- Dumping terraform states ...
2024-05-17 07:07:46 UTC -- Dumping terraform states ... done
2024-05-17 07:07:46 UTC -- Dumping container registry images ...
2024-05-17 07:07:46 UTC -- Dumping container registry images ... done
2024-05-17 07:07:46 UTC -- Dumping packages ...
2024-05-17 07:07:46 UTC -- Dumping packages ... done
2024-05-17 07:07:46 UTC -- Dumping ci secure files ...
2024-05-17 07:07:46 UTC -- Dumping ci secure files ... done
2024-05-17 07:07:46 UTC -- Creating backup archive: 1715929662_2024_05_17_17.0.0-ee_gitlab_backup.tar ...
2024-05-17 07:07:46 UTC -- Creating backup archive: 1715929662_2024_05_17_17.0.0-ee_gitlab_backup.tar ... done
2024-05-17 07:07:46 UTC -- Uploading backup archive to remote storage ... [SKIPPED]
2024-05-17 07:07:46 UTC -- Deleting old backups ... [SKIPPED]
2024-05-17 07:07:46 UTC -- Deleting tar staging files ...
2024-05-17 07:07:46 UTC -- Cleaning up /var/opt/gitlab/backups/backup_information.yml
2024-05-17 07:07:46 UTC -- Cleaning up /var/opt/gitlab/backups/db
2024-05-17 07:07:46 UTC -- Cleaning up /var/opt/gitlab/backups/repositories
2024-05-17 07:07:46 UTC -- Cleaning up /var/opt/gitlab/backups/uploads.tar.gz
2024-05-17 07:07:46 UTC -- Cleaning up /var/opt/gitlab/backups/builds.tar.gz
2024-05-17 07:07:46 UTC -- Cleaning up /var/opt/gitlab/backups/artifacts.tar.gz
2024-05-17 07:07:46 UTC -- Cleaning up /var/opt/gitlab/backups/pages.tar.gz
2024-05-17 07:07:46 UTC -- Cleaning up /var/opt/gitlab/backups/lfs.tar.gz
2024-05-17 07:07:46 UTC -- Cleaning up /var/opt/gitlab/backups/terraform_state.tar.gz
2024-05-17 07:07:46 UTC -- Cleaning up /var/opt/gitlab/backups/registry.tar.gz
2024-05-17 07:07:46 UTC -- Cleaning up /var/opt/gitlab/backups/packages.tar.gz
2024-05-17 07:07:46 UTC -- Cleaning up /var/opt/gitlab/backups/ci_secure_files.tar.gz
2024-05-17 07:07:46 UTC -- Deleting tar staging files ... done
2024-05-17 07:07:46 UTC -- Deleting backups/tmp ...
2024-05-17 07:07:46 UTC -- Deleting backups/tmp ... done
2024-05-17 07:07:46 UTC -- Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data
and are not included in this backup. You will need these files to restore a backup.
Please back them up manually.
2024-05-17 07:07:46 UTC -- Backup 1715929662_2024_05_17_17.0.0-ee is done.
2024-05-17 07:07:46 UTC -- Deleting backup and restore PID file ... done执行还原
首先还原配置文件和密钥文件，官方要求必须先还原密钥文件 /etc/gitlab/gitlab-secrets.json
即还原至 /etc/gitlab 目录下
root@ip-172-31-19-181:/etc/gitlab# ls -l
total 188
drwxr-xr-x 2 root root 4096 May 17 07:52 config_backup
-rw------- 1 root root 19095 May 17 08:02 gitlab-secrets.json
-rw-r--r-- 1 root root 154251 May 17 07:57 gitlab.rb
-rw------- 1 root root 749 May 17 07:32 initial_root_password
drwxr-xr-x 2 root root 4096 May 17 08:02 ssl
drwxr-xr-x 2 root root 4096 May 17 07:57 trusted-certs运行一次 gitlab-ctl reconfigure 后进行下一步
还原应用数据文件
#将应用数据文件 copy 到 /var/opt/gitlab/backups/ 目录下，并更改权限为 git
root@ip-172-31-19-181:~# cp /home/admin/1715929662_2024_05_17_17.0.0-ee_gitlab_backup.tar /var/opt/gitlab/backups/
root@ip-172-31-19-181:~# chown git:git /var/opt/gitlab/backups/1715929662_2024_05_17_17.0.0-ee_gitlab_backup.tar
# 关闭 puma, sidekiq 进程，并查看状态
root@ip-172-31-19-181:~# gitlab-ctl stop puma
root@ip-172-31-19-181:~# gitlab-ctl stop sidekiq
# This command will overwrite the contents of your GitLab database!
# NOTE: "_gitlab_backup.tar" is omitted from the name
root@ip-172-31-19-181:~# gitlab-backup restore 1715929662_2024_05_17_17.0.0-ee中途会提示
root@ip-172-31-19-181:/var/opt/gitlab/backups# gitlab-backup restore 1715929662_2024_05_17_17.0.0-ee
Transfering ownership of /var/opt/gitlab/gitlab-rails/shared/registry to git
2024-05-17 08:04:59 UTC -- Unpacking backup ...
2024-05-17 08:04:59 UTC -- Unpacking backup ... done
2024-05-17 08:04:59 UTC -- Restoring database ...
2024-05-17 08:04:59 UTC -- Be sure to stop Puma, Sidekiq, and any other process that
connects to the database before proceeding. For Omnibus
installs, see the following link for more information:
https://git.microcharon.com/help/raketasks/backup_restore.html#restore-for-omnibus-gitlab-installations
Before restoring the database, we will remove all existing
tables to avoid future upgrade problems. Be aware that if you have
custom tables in the GitLab database these tables and all data will be
removed.
Do you want to continue (yes/no)? yes
2024-05-17 08:05:05 UTC -- Removing all tables. Press `Ctrl-C` within 5 seconds to abort
2024-05-17 08:05:10 UTC -- Cleaning the database ...
2024-05-17 08:05:14 UTC -- done
2024-05-17 08:05:14 UTC -- Restoring PostgreSQL database gitlabhq_production ...
...
...
...
This task will now rebuild the authorized_keys file.
You will lose any data stored in the authorized_keys file.
Do you want to continue (yes/no)? yes
2024-05-17 08:05:34 UTC -- Deleting tar staging files ...
2024-05-17 08:05:34 UTC -- Cleaning up /var/opt/gitlab/backups/backup_information.yml
2024-05-17 08:05:34 UTC -- Cleaning up /var/opt/gitlab/backups/db
2024-05-17 08:05:34 UTC -- Cleaning up /var/opt/gitlab/backups/repositories
2024-05-17 08:05:34 UTC -- Cleaning up /var/opt/gitlab/backups/uploads.tar.gz
2024-05-17 08:05:34 UTC -- Cleaning up /var/opt/gitlab/backups/builds.tar.gz
2024-05-17 08:05:34 UTC -- Cleaning up /var/opt/gitlab/backups/artifacts.tar.gz
2024-05-17 08:05:34 UTC -- Cleaning up /var/opt/gitlab/backups/pages.tar.gz
2024-05-17 08:05:34 UTC -- Cleaning up /var/opt/gitlab/backups/lfs.tar.gz
2024-05-17 08:05:34 UTC -- Cleaning up /var/opt/gitlab/backups/terraform_state.tar.gz
2024-05-17 08:05:34 UTC -- Cleaning up /var/opt/gitlab/backups/registry.tar.gz
2024-05-17 08:05:34 UTC -- Cleaning up /var/opt/gitlab/backups/packages.tar.gz
2024-05-17 08:05:34 UTC -- Cleaning up /var/opt/gitlab/backups/ci_secure_files.tar.gz
2024-05-17 08:05:34 UTC -- Deleting tar staging files ... done
2024-05-17 08:05:34 UTC -- Deleting backups/tmp ...
2024-05-17 08:05:34 UTC -- Deleting backups/tmp ... done还原执行完成结束后，再配置，重启 Gitlab 实例，并检查
gitlab-ctl reconfigure
gitlab-ctl restartroot@ip-172-31-19-181:/etc/gitlab# gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.35.0 ? ... OK (14.35.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes
Tables are truncated? ... skipped
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Cable config exists? ... yes
Resque config exists? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
1/1 ... yes
Redis version >= 6.2.14? ... yes
Ruby version >= 3.0.6 ? ... yes (3.1.5)
Git user has default SSH configuration? ... yes
Active users: ... 1
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x-8.x or OpenSearch version 1.x ... skipped (Advanced Search is disabled)
All migrations must be finished before doing a major upgrade ... skipped (Advanced Search is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
之前测试创建的项目被成功还原，同管理页中之前配置的选项也被成功还原
todo:
使用 cron 任务来备份文件
参考资料
Create a GitLab upgrade plan | GitLab
How to Uninstall GitLab on Ubuntu | Linux Tutorials for Beginners (webhostinggeeks.com)
Back up and restore GitLab | GitLab
GitLab備份及復原流程筆記 :: Hello Friend, I'm Yesting! (suyenting.github.io)