上周更新系统,看到Gitlab-CE有新版本,直接就更新了。今天写好代码要推送的时候,发现服务端竟然挂了,提示502错误。
登陆到服务器,运行 gitlab-ctl status
查看状态,组件都正常运行。唯一的意外是:ssh登录服务器时,会出现”LANG cannot change locale (en_US.utf8)” 的提示。暂时想不到其他原因,就先解决这个问题。
运行 locale -a
命令,发现系统中确实没有 en_US.utf8
的本地化设置文件。于是编辑 /etc/locale.gen
文件,取消 “en_US.UTF-8 UTF-8” 这一行的注释,运行 locale-gen
,生成 en_US.utf8 相关文件。
接着设置语言和本地化设置:
export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8
断开ssh重新登陆,bash的警告提示消失了。运行 gitlab-ctl restart
重启Gitlab,代码能正常推送,网站能打开,似乎问题都解决了。
但是好事多磨,过了没多久,发现网页都是500错误。虽然能通过git正常推送和拉去代码,但是网页用不了也挺烦人的。想到上个月为了节省服务器资源,把Premethus、Grafana等用不到的组件都关了,会不会是这个原因导致的?
于是编辑 /etc/gitlab/gitlab.rb
文件,把组件又启用。运行 gitlab-ctl reconfigure
重新配置,意外发现了,脚本无法运行成功,出现了如下提示:
Recipe: gitlab::database_migrations
* ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
* rails_migration[gitlab-rails] action run
* bash[migrate gitlab-rails database] action run
================================================================================
Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Command execution failed. STDOUT/STDERR suppressed for sensitive resource
Resource Declaration:
---------------------
suppressed sensitive resource output
Compiled Resource:
------------------
suppressed sensitive resource output
System Info:
------------
chef_version=15.17.4
platform=ubuntu
platform_version=20.04
ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client
================================================================================
Error executing action `run` on resource 'rails_migration[gitlab-rails]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
51: rails_migration "gitlab-rails" do
52: rake_task 'gitlab:db:configure'
53: logfile_prefix 'gitlab-rails-db-migrate'
54: helper migration_helper
55:
56: environment env_variables
57: dependent_services dependent_services
58: notifies :run, "execute[clear the gitlab-rails cache]", :immediately
59: notifies :run, "ruby_block[check remote PG version]", :immediately
60:
61: only_if { migration_helper.attributes_node['auto_migrate'] }
62: end
Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'
rails_migration("gitlab-rails") do
action [:run]
default_guard_interpreter :default
declared_type :rails_migration
cookbook_name "gitlab"
recipe_name "database_migrations"
rake_task "gitlab:db:configure"
logfile_prefix "gitlab-rails-db-migrate"
helper "*sensitive value suppressed*"
environment "*sensitive value suppressed*"
dependent_services ["runit_service[puma]", "sidekiq_service[sidekiq]"]
only_if { #code block }
end
System Info:
------------
chef_version=15.17.4
platform=ubuntu
platform_version=20.04
ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client
Running handlers:
There was an error running gitlab-ctl reconfigure:
rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
Running handlers complete
Chef Infra Client failed. 0 resources updated in 21 seconds
网上一查,发现和上次碰到的 Gitlab更新设置总是500错误 问题一样,又是Gitlab官方的锅。但是这次解决问题比之前更麻烦,步骤为:
运行 gitlab-rake db:migrate
,出现错误没关系,中间会输出一行用于终止后台任务的命令;
复制这条命令或者上一步控制台输出的指令: gitlab-rake gitlab:background_migrations:finalize[ProjectNamespaces::BackfillProjectNamespaces,projects,id,'[null\\,"up"]']
。运行这条指令可能会失败,如果失败,请重新运行上面的指令,成功说明问题解决了;
再次运行 gitlab-rake db:migrate
,确认不会再报错;
运行 gitlab-ctl reconfigure
;
重启Gitlab:gitlab-ctl restart
。
需要注意的是,这个bug是在Gitlab-CE 14.10.0中引入,在14.10.1版本被修复。如果运行过reconfigure但没解决错误,是无法直接升级到14.10.1版本的。