obd无法启动ob集群提示:obagent start failed

【 使用环境 】测试环境
【 OB or 其他组件 】
ob社区版4.1.0
系统centos7.9
部署方式obd白屏安装
【 使用版本 】
【问题描述】清晰明确描述问题
【复现路径】问题出现前后相关操作

错误提示

部署时提示

只有完成按钮点击完成以后尝试使用obd命令启动

obd cluster start myoceanbase

一开始提示obproxy启动失败,折腾了几次也没改什么,之后在star集群只提示obagent start failed

Check before start observer ok
Check before start obproxy ok
Check before start obagent ok
Check before start ocp-express ok
Start observer ok
observer program health check ok
Connect to observer ok
Start obproxy ok
obproxy program health check ok
Connect to obproxy ok
Initialize obproxy-ce ok
Start obagent ok
obagent program health check x
[WARN] failed to start xx.xx.xx.168 obagent
[ERROR] obagent start failed
Wait for observer init ok
+------------------------------------------------+
|                    observer                    |
+--------------+---------+------+-------+--------+
| ip           | version | port | zone  | status |
+--------------+---------+------+-------+--------+
| xx.xx.xx.140 | 4.1.0.0 | 2881 | zone3 | ACTIVE |
| xx.xx.xx.168 | 4.1.0.0 | 2881 | zone1 | ACTIVE |
| xx.xx.xx.173 | 4.1.0.0 | 2881 | zone2 | ACTIVE |
+--------------+---------+------+-------+--------+
obclient -hxx.xx.xx.140 -P2881 -uroot -p'xxxx' -Doceanbase -A

+------------------------------------------------+
|                    obproxy                     |
+--------------+------+-----------------+--------+
| ip           | port | prometheus_port | status |
+--------------+------+-----------------+--------+
| xx.xx.xx.168 | 2883 | 2884            | active |
+--------------+------+-----------------+--------+
obclient -hxx.xx.xx.168 -P2883 -uroot -p'xxxx' -Doceanbase -A
See https://www.oceanbase.com/product/ob-deployer/error-codes .
Trace ID: b2364af4-ed8c-11ed-a6a8-fa163ea8e553
If you want to view detailed obd logs, please run: obd display-trace b2364af4-ed8c-11ed-a6a8-fa163ea8e553

配置

obd cluster edit-config myoceanbase

user:
  username: root
  password: xxxxxx
  port: 22
oceanbase-ce:
  version: 4.1.0.0
  release: 100000202023040520.el7
  package_hash: d598937b1cfb1df85e2c2231acf024e4994db533
  xx.xx.xx.168:
    zone: zone1
  xx.xx.xx.173:
    zone: zone2
  xx.xx.xx.140:
    zone: zone3
  servers:
  - xx.xx.xx.168
  - xx.xx.xx.173
  - xx.xx.xx.140
  global:
    appname: myoceanbase
    root_password: xxxxxxx
    mysql_port: 2881
    rpc_port: 2882
    home_path: /root/myoceanbase/oceanbase
    ocp_meta_tenant_log_disk_size: 7G
    enable_syslog_recycle: true
    enable_syslog_wf: false
    max_syslog_file_count: 4
    memory_limit: 26G
    datafile_size: 402G
    system_memory: 5G
    log_disk_size: 80G
    cpu_count: 16
obproxy-ce:
  version: 4.1.0.0
  package_hash: 2a9d9bf67f179dcca2a8c9e7c77373d94e7e2abe
  release: 7.el7
  servers:
  - xx.xx.xx.168
  global:
    prometheus_listen_port: 2884
    listen_port: 2883
    home_path: /root/myoceanbase/obproxy
    skip_proxy_sys_private_check: true
    enable_strict_kernel_release: false
    enable_cluster_checkout: false
  depends:
  - oceanbase-ce
obagent:
  version: 1.3.0
  package_hash: d57fbb4962b2fbecb6282358c59295fdfba4d6ac
  release: 22.el7
  servers:
  - xx.xx.xx.168
  - xx.xx.xx.173
  - xx.xx.xx.140
  global:
    monagent_http_port: 8088
    mgragent_http_port: 8089
    home_path: /root/myoceanbase/obagent
    ob_monitor_status: active
  depends:
  - oceanbase-ce
ocp-express:
  version: 1.0.0
  package_hash: 5b29837dc1f575c7d1840ab8e59064ddd407bd4b
  release: 100000452023041914.el7
  servers:
  - xx.xx.xx.168
  global:
    port: 8180
    home_path: /root/myoceanbase/ocpexpress
    memory_size: 872M
  depends:
  - obagent
  - oceanbase-ce
  - obproxy-ce

相关日志

agentctl.log

/.config_secret.key CryptoMethod:plain}
2023-05-08T18:21:18.64601+08:00 INFO [4151,1063a252681fe66d] caller=config/config_reader_writer.go:218:loadYamlFilesFromPath: read config files from path /root/myoceanbase/obagent/conf/config_properties
2023-05-08T18:21:18.64619+08:00 INFO [4151,1063a252681fe66d] caller=config/config_reader_writer.go:224:loadYamlFilesFromPath: read config from path /root/myoceanbase/obagent/conf/config_properties, files length 6
2023-05-08T18:21:18.64879+08:00 INFO [4151,1063a252681fe66d] caller=config/config_reader_writer.go:118:decodeConfigPropertiesGroup: config key monagent.ob.gateway.user is not defined in sdk, use valueType string
2023-05-08T18:21:18.64887+08:00 INFO [4151,1063a252681fe66d] caller=config/config_reader_writer.go:118:decodeConfigPropertiesGroup: config key monagent.ob.gateway.password is not defined in sdk, use valueType string
2023-05-08T18:21:18.64941+08:00 WARN [4151,1063a252681fe66d] caller=config/config_property_meta.go:40:mergeConfigProperties: config key monagent.monitordb.password is still set as default value
2023-05-08T18:21:18.64948+08:00 WARN [4151,1063a252681fe66d] caller=config/config_property_meta.go:40:mergeConfigProperties: config key monagent.ob.monitor.password is still set as default value
2023-05-08T18:21:18.64954+08:00 INFO [4151,1063a252681fe66d] caller=sdk/sdk_init.go:76:initConfigs: init config properties success.
2023-05-08T18:21:18.64957+08:00 INFO [4151,1063a252681fe66d] caller=config/config_reader_writer.go:218:loadYamlFilesFromPath: read config files from path /root/myoceanbase/obagent/conf/module_config
2023-05-08T18:21:18.64969+08:00 INFO [4151,1063a252681fe66d] caller=config/config_reader_writer.go:224:loadYamlFilesFromPath: read config from path /root/myoceanbase/obagent/conf/module_config, files length 16
2023-05-08T18:21:18.66779+08:00 INFO [4151,] caller=agent/admin.go:95:NewAdmin: NewAdmin with config {RunDir:/root/myoceanbase/obagent/run ConfDir:/root/myoceanbase/obagent/conf LogDir:/root/myoceanbase/obagent/log BackupDir:/root/myoceanbase/obagent/backup TempDir:/root/myoceanbase/obagent/tmp TaskStoreDir:/root/myoceanbase/obagent/task_store AgentPkgName:obagent PkgExt:rpm PkgStoreDir:/root/myoceanbase/obagent/pkg_store StartWaitSeconds:10 StopWaitSeconds:10 AgentdPath:/root/myoceanbase/obagent/bin/ob_agentd}
2023-05-08T18:21:18.66789+08:00 INFO [4151,] caller=agent/admin.go:142:Lock: process 4151 fetching admin lock
2023-05-08T18:21:18.66791+08:00 INFO [4151,] caller=agent/admin.go:107:cleanDanglingLock: process 4151 try cleaning dangling admin lock
2023-05-08T18:21:18.66814+08:00 INFO [4151,] caller=agent/admin.go:168:Lock: process 4151 got admin lock
2023-05-08T18:21:18.6682+08:00 INFO [4151,] caller=agent/admin.go:192:StartAgent: starting agent
2023-05-08T18:21:18.66823+08:00 INFO [4151,] caller=agent/admin.go:583:progressStart: startAgent start: []
2023-05-08T18:21:18.66825+08:00 WARN [4151,] caller=agent/admin.go:584:progressStart: updateProgress: missing storedStatus fields: task_token=
2023-05-08T18:21:18.66829+08:00 INFO [4151,] caller=agent/admin.go:218:startAgent: starting agentd with config file '/root/myoceanbase/obagent/conf/agentd.yaml'
2023-05-08T18:21:19.68181+08:00 INFO [4151,] caller=agent/admin.go:340:AgentStatus: AgentStatus
2023-05-08T18:21:19.71952+08:00 INFO [4151,] caller=agent/admin.go:352:AgentStatus: check agentd status got: {State:running Ready:false Version:1.3.0-SNAPSHOT Pid:4158 Socket:/root/myoceanbase/obagent/run/ob_agentd.4158.sock Services:map[ob_mgragent:{Status:{State:stopped Version: Pid:0 StartAt:-6795364578871345152 Ports:[]} Socket:/root/myoceanbase/obagent/run/ob_mgragent.0.sock EndAt:-6795364578871345152} ob_monagent:{Status:{State:running Version:1.3.0-SNAPSHOT Pid:4164 StartAt:1683541279347961532 Ports:[8088]} Socket:/root/myoceanbase/obagent/run/ob_monagent.4164.sock EndAt:-6795364578871345152}] Dangling:[] StartAt:1683541278788079906}
2023-05-08T18:21:20.71991+08:00 INFO [4151,] caller=agent/admin.go:340:AgentStatus: AgentStatus
2023-05-08T18:21:20.74459+08:00 INFO [4151,] caller=agent/admin.go:352:AgentStatus: check agentd status got: {State:running Ready:false Version:1.3.0-SNAPSHOT Pid:4158 Socket:/root/myoceanbase/obagent/run/ob_agentd.4158.sock Services:map[ob_mgragent:{Status:{State:stopped Version: Pid:0 StartAt:-6795364578871345152 Ports:[]} Socket:/root/myoceanbase/obagent/run/ob_mgragent.0.sock EndAt:-6795364578871345152} ob_monagent:{Status:{State:running Version:1.3.0-SNAPSHOT Pid:4164 StartAt:1683541279347961532 Ports:[8088]} Socket:/root/myoceanbase/obagent/run/ob_monagent.4164.sock EndAt:-6795364578871345152}] Dangling:[] StartAt:1683541278788079906}
2023-05-08T18:21:21.74493+08:00 INFO [4151,] caller=agent/admin.go:340:AgentStatus: AgentStatus
2023-05-08T18:21:21.76546+08:00 INFO [4151,] caller=agent/admin.go:352:AgentStatus: check agentd status got: {State:running Ready:false Version:1.3.0-SNAPSHOT Pid:4158 Socket:/root/myoceanbase/obagent/run/ob_agentd.4158.sock Services:map[ob_mgragent:{Status:{State:stopped Version: Pid:0 StartAt:-6795364578871345152 Ports:[]} Socket:/root/myoceanbase/obagent/run/ob_mgragent.0.sock EndAt:-6795364578871345152} ob_monagent:{Status:{State:running Version:1.3.0-SNAPSHOT Pid:4164 StartAt:1683541279347961532 Ports:[8088]} Socket:/root/myoceanbase/obagent/run/ob_monagent.4164.sock EndAt:-6795364578871345152}] Dangling:[] StartAt:1683541278788079906}
2023-05-08T18:21:22.76596+08:00 INFO [4151,] caller=agent/admin.go:340:AgentStatus: AgentStatus
2023-05-08T18:21:22.78987+08:00 INFO [4151,] caller=agent/admin.go:352:AgentStatus: check agentd status got: {State:running Ready:false Version:1.3.0-SNAPSHOT Pid:4158 Socket:/root/myoceanbase/obagent/run/ob_agentd.4158.sock Services:map[ob_mgragent:{Status:{State:stopped Version: Pid:0 StartAt:-6795364578871345152 Ports:[]} Socket:/root/myoceanbase/obagent/run/ob_mgragent.0.sock EndAt:-6795364578871345152} ob_monagent:{Status:{State:running Version:1.3.0-SNAPSHOT Pid:4164 StartAt:1683541279347961532 Ports:[8088]} Socket:/root/myoceanbase/obagent/run/ob_monagent.4164.sock EndAt:-6795364578871345152}] Dangling:[] StartAt:1683541278788079906}
2023-05-08T18:21:23.79082+08:00 INFO [4151,] caller=agent/admin.go:340:AgentStatus: AgentStatus
2023-05-08T18:21:23.81331+08:00 INFO [4151,] caller=agent/admin.go:352:AgentStatus: check agentd status got: {State:running Ready:false Version:1.3.0-SNAPSHOT Pid:4158 Socket:/root/myoceanbase/obagent/run/ob_agentd.4158.sock Services:map[ob_mgragent:{Status:{State:stopped Version: Pid:0 StartAt:-6795364578871345152 Ports:[]} Socket:/root/myoceanbase/obagent/run/ob_mgragent.0.sock EndAt:-6795364578871345152} ob_monagent:{Status:{State:running Version:1.3.0-SNAPSHOT Pid:4164 StartAt:1683541279347961532 Ports:[8088]} Socket:/root/myoceanbase/obagent/run/ob_monagent.4164.sock EndAt:-6795364578871345152}] Dangling:[] StartAt:1683541278788079906}
2023-05-08T18:21:24.81388+08:00 INFO [4151,] caller=agent/admin.go:340:AgentStatus: AgentStatus
2023-05-08T18:21:24.83928+08:00 INFO [4151,] caller=agent/admin.go:352:AgentStatus: check agentd status got: {State:running Ready:false Version:1.3.0-SNAPSHOT Pid:4158 Socket:/root/myoceanbase/obagent/run/ob_agentd.4158.sock Services:map[ob_mgragent:{Status:{State:stopped Version: Pid:0 StartAt:-6795364578871345152 Ports:[]} Socket:/root/myoceanbase/obagent/run/ob_mgragent.0.sock EndAt:-6795364578871345152} ob_monagent:{Status:{State:running Version:1.3.0-SNAPSHOT Pid:4164 StartAt:1683541279347961532 Ports:[8088]} Socket:/root/myoceanbase/obagent/run/ob_monagent.4164.sock EndAt:-6795364578871345152}] Dangling:[] StartAt:1683541278788079906}
2023-05-08T18:21:25.83952+08:00 INFO [4151,] caller=agent/admin.go:340:AgentStatus: AgentStatus
2023-05-08T18:21:25.86298+08:00 INFO [4151,] caller=agent/admin.go:352:AgentStatus: check agentd status got: {State:running Ready:false Version:1.3.0-SNAPSHOT Pid:4158 Socket:/root/myoceanbase/obagent/run/ob_agentd.4158.sock Services:map[ob_mgragent:{Status:{State:stopped Version: Pid:0 StartAt:-6795364578871345152 Ports:[]} Socket:/root/myoceanbase/obagent/run/ob_mgragent.0.sock EndAt:-6795364578871345152} ob_monagent:{Status:{State:running Version:1.3.0-SNAPSHOT Pid:4164 StartAt:1683541279347961532 Ports:[8088]} Socket:/root/myoceanbase/obagent/run/ob_monagent.4164.sock EndAt:-6795364578871345152}] Dangling:[] StartAt:1683541278788079906}
2023-05-08T18:21:26.86315+08:00 INFO [4151,] caller=agent/admin.go:340:AgentStatus: AgentStatus
2023-05-08T18:21:26.88445+08:00 INFO [4151,] caller=agent/admin.go:352:AgentStatus: check agentd status got: {State:running Ready:false Version:1.3.0-SNAPSHOT Pid:4158 Socket:/root/myoceanbase/obagent/run/ob_agentd.4158.sock Services:map[ob_mgragent:{Status:{State:stopped Version: Pid:0 StartAt:-6795364578871345152 Ports:[]} Socket:/root/myoceanbase/obagent/run/ob_mgragent.0.sock EndAt:-6795364578871345152} ob_monagent:{Status:{State:running Version:1.3.0-SNAPSHOT Pid:4164 StartAt:1683541279347961532 Ports:[8088]} Socket:/root/myoceanbase/obagent/run/ob_monagent.4164.sock EndAt:-6795364578871345152}] Dangling:[] StartAt:1683541278788079906}
2023-05-08T18:21:27.88521+08:00 INFO [4151,] caller=agent/admin.go:340:AgentStatus: AgentStatus
2023-05-08T18:21:27.90968+08:00 INFO [4151,] caller=agent/admin.go:352:AgentStatus: check agentd status got: {State:running Ready:false Version:1.3.0-SNAPSHOT Pid:4158 Socket:/root/myoceanbase/obagent/run/ob_agentd.4158.sock Services:map[ob_mgragent:{Status:{State:stopped Version: Pid:0 StartAt:-6795364578871345152 Ports:[]} Socket:/root/myoceanbase/obagent/run/ob_mgragent.0.sock EndAt:-6795364578871345152} ob_monagent:{Status:{State:running Version:1.3.0-SNAPSHOT Pid:4164 StartAt:1683541279347961532 Ports:[8088]} Socket:/root/myoceanbase/obagent/run/ob_monagent.4164.sock EndAt:-6795364578871345152}] Dangling:[] StartAt:1683541278788079906}
2023-05-08T18:21:28.91082+08:00 INFO [4151,] caller=agent/admin.go:340:AgentStatus: AgentStatus
2023-05-08T18:21:28.93115+08:00 INFO [4151,] caller=agent/admin.go:352:AgentStatus: check agentd status got: {State:running Ready:false Version:1.3.0-SNAPSHOT Pid:4158 Socket:/root/myoceanbase/obagent/run/ob_agentd.4158.sock Services:map[ob_mgragent:{Status:{State:stopped Version: Pid:0 StartAt:-6795364578871345152 Ports:[]} Socket:/root/myoceanbase/obagent/run/ob_mgragent.0.sock EndAt:-6795364578871345152} ob_monagent:{Status:{State:running Version:1.3.0-SNAPSHOT Pid:4164 StartAt:1683541279347961532 Ports:[8088]} Socket:/root/myoceanbase/obagent/run/ob_monagent.4164.sock EndAt:-6795364578871345152}] Dangling:[] StartAt:1683541278788079906}
2023-05-08T18:21:28.9312+08:00 ERROR [4151,] caller=agent/admin.go:242:startAgent: wait for agent ready timeout
2023-05-08T18:21:28.93123+08:00 INFO [4151,] caller=agent/admin.go:602:progressEnd: startAgent end
2023-05-08T18:21:28.93126+08:00 WARN [4151,] caller=agent/admin.go:604:progressEnd: updateProgress: missing storedStatus fields:, task_token=
2023-05-08T18:21:28.93131+08:00 ERROR [4151,] caller=agent/admin.go:197:StartAgent: start agent failed: Module=agent, kind=DEADLINE_EXCEEDED, code=wait_for_ready_timeout; 
2023-05-08T18:21:28.93142+08:00 INFO [4151,] caller=agent/admin.go:176:Unlock: process 4151 release admin lock
2023-05-08T18:21:28.93162+08:00 INFO [4151,] caller=agentctl/main.go:261:func2: agentctl error fields:, response="{\"successful\":false,\"message\":null,\"error\":\"Module=agent, kind=DEADLINE_EXCEEDED, code=wait_for_ready_timeout; \"}"

agentd.log

2023-05-08T17:40:07.02843+08:00 INFO [12391,ba73e944fb16b6bc] caller=http/http_command.go:47:func1: handling command request /api/v1/status <nil>
2023-05-08T17:40:07.05788+08:00 INFO [12391,06b9badb899c8fb1] caller=http/http_command.go:90:writeOk: command request /api/v1/status succeed
2023-05-08T17:40:08.06001+08:00 INFO [12391,bb2bf3823008881f] caller=http/http_command.go:47:func1: handling command request /api/v1/status <nil>
2023-05-08T17:40:08.08601+08:00 INFO [12391,4751369a614a0b08] caller=http/http_command.go:90:writeOk: command request /api/v1/status succeed
2023-05-08T18:21:18.80003+08:00 INFO [4158,] caller=agentd/main.go:51:run: starting agentd with config /root/myoceanbase/obagent/conf/agentd.yaml
2023-05-08T18:21:18.815+08:00 INFO [4158,] caller=agentd/limit_linux.go:20:newLimiter: create service ob_mgragent resource limit skipped, no limit in config
2023-05-08T18:21:18.83781+08:00 INFO [4158,] caller=agentd/limit_linux.go:31:newLimiter: create service ob_monagent resource limit done, cpu: 2, memory: 2GiB
2023-05-08T18:21:18.8384+08:00 ERROR [4158,] caller=agentd/watchdog.go:259:cleanupPidPattern: cleanup pid file /root/myoceanbase/obagent/run/ob_mgragent.pid got error fields:, error="Module=agent, kind=INTERNAL, code=read_pid_failed; [/root/myoceanbase/obagent/run/ob_mgragent.pid]\ncause:\nstrconv.Atoi: parsing \"\": invalid syntax"
2023-05-08T18:21:18.83872+08:00 INFO [4158,] caller=agentd/watchdog.go:81:Start: starting agentd
2023-05-08T18:21:18.95648+08:00 INFO [4158,] caller=agentd/watchdog.go:94:Start: starting socket listener on '/root/myoceanbase/obagent/run/ob_agentd.4158.sock'
2023-05-08T18:21:18.95689+08:00 INFO [4158,] caller=agentd/watchdog.go:104:Start: starting service 'ob_mgragent'
2023-05-08T18:21:18.95718+08:00 ERROR [4158,] caller=agentd/watchdog.go:107:Start: start service 'ob_mgragent' failed: Module=agentd, kind=INTERNAL, code=write_pid_failed; [/root/myoceanbase/obagent/run/ob_mgragent.pid]
cause:
open /root/myoceanbase/obagent/run/ob_mgragent.pid: file exists
2023-05-08T18:21:18.95723+08:00 INFO [4158,] caller=agentd/watchdog.go:104:Start: starting service 'ob_monagent'
2023-05-08T18:21:18.95735+08:00 INFO [4158,] caller=agentd/service.go:132:startProc: starting service fields:, service=ob_monagent
2023-05-08T18:21:18.9714+08:00 INFO [4158,] caller=agentd/service.go:139:startProc: service process started. pid: 4164 fields: service=ob_monagent
2023-05-08T18:21:18.97156+08:00 INFO [4158,] caller=agentd/watchdog.go:110:Start: agentd started
2023-05-08T18:21:19.68351+08:00 INFO [4158,4bb5fad6769bf611] caller=http/http_command.go:47:func1: handling command request /api/v1/status <nil>
2023-05-08T18:21:19.7186+08:00 INFO [4158,655cc53616a8a54d] caller=http/http_command.go:90:writeOk: command request /api/v1/status succeed
2023-05-08T18:21:20.72058+08:00 INFO [4158,77d572383c2d977e] caller=http/http_command.go:47:func1: handling command request /api/v1/status <nil>
2023-05-08T18:21:20.74395+08:00 INFO [4158,2c89db92aad72339] caller=http/http_command.go:90:writeOk: command request /api/v1/status succeed
2023-05-08T18:21:21.74554+08:00 INFO [4158,9f3f5d721fc0e48f] caller=http/http_command.go:47:func1: handling command request /api/v1/status <nil>
2023-05-08T18:21:21.76495+08:00 INFO [4158,9ed146e4208eac05] caller=http/http_command.go:90:writeOk: command request /api/v1/status succeed
2023-05-08T18:21:22.76657+08:00 INFO [4158,6440fb3706b7dc77] caller=http/http_command.go:47:func1: handling command request /api/v1/status <nil>
2023-05-08T18:21:22.78953+08:00 INFO [4158,ea0b68bb65b3f4b1] caller=http/http_command.go:90:writeOk: command request /api/v1/status succeed
2023-05-08T18:21:23.79152+08:00 INFO [4158,a08555a32ab64da4] caller=http/http_command.go:47:func1: handling command request /api/v1/status <nil>
2023-05-08T18:21:23.81286+08:00 INFO [4158,f1bb573deb8fd72c] caller=http/http_command.go:90:writeOk: command request /api/v1/status succeed
2023-05-08T18:21:24.81449+08:00 INFO [4158,4d88cc57a139b0de] caller=http/http_command.go:47:func1: handling command request /api/v1/status <nil>
2023-05-08T18:21:24.83881+08:00 INFO [4158,34e9902a971a3c1c] caller=http/http_command.go:90:writeOk: command request /api/v1/status succeed
2023-05-08T18:21:25.84063+08:00 INFO [4158,f263cad820ec57e2] caller=http/http_command.go:47:func1: handling command request /api/v1/status <nil>
2023-05-08T18:21:25.8625+08:00 INFO [4158,fa9c83665e60d8f1] caller=http/http_command.go:90:writeOk: command request /api/v1/status succeed
2023-05-08T18:21:26.86374+08:00 INFO [4158,2dcca8c67ca6ca74] caller=http/http_command.go:47:func1: handling command request /api/v1/status <nil>
2023-05-08T18:21:26.884+08:00 INFO [4158,7947c1680f111f2b] caller=http/http_command.go:90:writeOk: command request /api/v1/status succeed
2023-05-08T18:21:27.88576+08:00 INFO [4158,faa21106333ec664] caller=http/http_command.go:47:func1: handling command request /api/v1/status <nil>
2023-05-08T18:21:27.90925+08:00 INFO [4158,c3b7f66c1cd180c7] caller=http/http_command.go:90:writeOk: command request /api/v1/status succeed
2023-05-08T18:21:28.91141+08:00 INFO [4158,312f2faa8406d988] caller=http/http_command.go:47:func1: handling command request /api/v1/status <nil>
2023-05-08T18:21:28.93075+08:00 INFO [4158,68e8ae816e21bccb] caller=http/http_command.go:90:writeOk: command request /api/v1/status succeed

ob_monagent.error.log

time="2023-05-08T18:21:19+08:00" level=debug msg="config key monagent.pipeline.ob.trace.status is still set as default value:<nil>"
time="2023-05-08T18:21:19+08:00" level=debug msg="merge config meta and configfile config, config key monagent.pipeline.ob.trace.status"
time="2023-05-08T18:21:19+08:00" level=debug msg="config key monagent.alertmanager.address is still set as default value:<nil>"
time="2023-05-08T18:21:19+08:00" level=debug msg="merge config meta and configfile config, config key monagent.alertmanager.address"
time="2023-05-08T18:21:19+08:00" level=debug msg="merge config meta and configfile config, config key mgragent.log.maxage.days"
time="2023-05-08T18:21:19+08:00" level=debug msg="merge config meta and configfile config, config key monagent.mysql.sql.port"
time="2023-05-08T18:21:19+08:00" level=info msg="init config properties success."
time="2023-05-08T18:21:19+08:00" level=info msg="read config files from path /root/myoceanbase/obagent/conf/module_config"
time="2023-05-08T18:21:19+08:00" level=info msg="read config from path /root/myoceanbase/obagent/conf/module_config, files length 16"
time="2023-05-08T18:21:19+08:00" level=debug msg="read config from path /root/myoceanbase/obagent/conf/module_config, file common_module.yaml"
time="2023-05-08T18:21:19+08:00" level=debug msg="register module mgragent.config.meta, moduleType config.meta"
time="2023-05-08T18:21:19+08:00" level=debug msg="register module mgragent.proxy.config, moduleType proxy.config"
time="2023-05-08T18:21:19+08:00" level=debug msg="register module monagent.config.meta, moduleType config.meta"
time="2023-05-08T18:21:19+08:00" level=debug msg="register module mgragent.stat.config, moduleType stat.config"
time="2023-05-08T18:21:19+08:00" level=debug msg="register module monagent.stat.config, moduleType stat.config"
time="2023-05-08T18:21:19+08:00" level=debug msg="read config from path /root/myoceanbase/obagent/conf/module_config, file log_module.yaml"
time="2023-05-08T18:21:19+08:00" level=debug msg="register module monagent.log.config, moduleType monagent.log.config"
time="2023-05-08T18:21:19+08:00" level=debug msg="register module mgragent.log.config, moduleType mgragent.log.config"
time="2023-05-08T18:21:19+08:00" level=debug msg="read config from path /root/myoceanbase/obagent/conf/module_config, file mgragent_logquerier_module.yaml"
time="2023-05-08T18:21:19+08:00" level=debug msg="register module mgragent.logquerier, moduleType mgragent.logquerier"
time="2023-05-08T18:21:19+08:00" level=debug msg="read config from path /root/myoceanbase/obagent/conf/module_config, file mgragent_module.yaml"
time="2023-05-08T18:21:19+08:00" level=debug msg="register module mgragent.basic.auth, moduleType mgragent.basic.auth"
time="2023-05-08T18:21:19+08:00" level=debug msg="register module module.config.notify, moduleType module.config.notify"
time="2023-05-08T18:21:19+08:00" level=debug msg="read config from path /root/myoceanbase/obagent/conf/module_config, file monagent_basic_auth.yaml"
time="2023-05-08T18:21:19+08:00" level=debug msg="register module monagent.server.basic.auth, moduleType monagent.server.basic.auth"
time="2023-05-08T18:21:19+08:00" level=debug msg="read config from path /root/myoceanbase/obagent/conf/module_config, file monitor_host_log.yaml"
time="2023-05-08T18:21:19+08:00" level=debug msg="register module monitor.host.log, moduleType monagent.pipeline"
time="2023-05-08T18:21:19+08:00" level=debug msg="read config from path /root/myoceanbase/obagent/conf/module_config, file monitor_log_alarm.yaml"
time="2023-05-08T18:21:19+08:00" level=debug msg="register module monitor.log.alarm, moduleType monagent.pipeline"
time="2023-05-08T18:21:19+08:00" level=debug msg="read config from path /root/myoceanbase/obagent/conf/module_config, file monitor_mysql.yaml"
time="2023-05-08T18:21:19+08:00" level=debug msg="register module monitor.mysql, moduleType monagent.pipeline"
time="2023-05-08T18:21:19+08:00" level=debug msg="read config from path /root/myoceanbase/obagent/conf/module_config, file monitor_node_custom.yaml"
time="2023-05-08T18:21:19+08:00" level=debug msg="register module monitor.node.custom, moduleType monagent.pipeline"
time="2023-05-08T18:21:19+08:00" level=debug msg="read config from path /root/myoceanbase/obagent/conf/module_config, file monitor_node_host.yaml"
time="2023-05-08T18:21:19+08:00" level=debug msg="register module monitor.node.host, moduleType monagent.pipeline"
time="2023-05-08T18:21:19+08:00" level=debug msg="read config from path /root/myoceanbase/obagent/conf/module_config, file monitor_ob.yaml"
time="2023-05-08T18:21:19+08:00" level=debug msg="register module monitor.ob, moduleType monagent.pipeline"
time="2023-05-08T18:21:19+08:00" level=debug msg="read config from path /root/myoceanbase/obagent/conf/module_config, file monitor_ob_custom.yaml"
time="2023-05-08T18:21:19+08:00" level=debug msg="register module monitor.ob.custom, moduleType monagent.pipeline"
time="2023-05-08T18:21:19+08:00" level=debug msg="read config from path /root/myoceanbase/obagent/conf/module_config, file monitor_ob_log.yaml"
time="2023-05-08T18:21:19+08:00" level=debug msg="register module monitor.ob.err.log, moduleType monagent.pipeline"
time="2023-05-08T18:21:19+08:00" level=debug msg="read config from path /root/myoceanbase/obagent/conf/module_config, file monitor_observer_log.yaml"
time="2023-05-08T18:21:19+08:00" level=debug msg="register module monitor.ob.log, moduleType monagent.pipeline"
time="2023-05-08T18:21:19+08:00" level=debug msg="read config from path /root/myoceanbase/obagent/conf/module_config, file monitor_observer_trace.yaml"
time="2023-05-08T18:21:19+08:00" level=debug msg="register module monitor.trace.log, moduleType monagent.pipeline"
time="2023-05-08T18:21:19+08:00" level=debug msg="read config from path /root/myoceanbase/obagent/conf/module_config, file ob_logcleaner_module.yaml"
time="2023-05-08T18:21:19+08:00" level=debug msg="register module ob.logcleaner, moduleType ob.logcleaner"
time="2023-05-08T18:21:19+08:00" level=debug msg="register moduleType module.config.notify config callback start"
time="2023-05-08T18:21:19+08:00" level=debug msg="register moduleType module.config.notify config callback end"
time="2023-05-08T18:21:19+08:00" level=debug msg="register moduleType config.meta config callback start"
time="2023-05-08T18:21:19+08:00" level=debug msg="register moduleType config.meta config callback end"
time="2023-05-08T18:21:19+08:00" level=debug msg="register moduleType stat.config config callback start"
time="2023-05-08T18:21:19+08:00" level=debug msg="register moduleType stat.config config callback end"
time="2023-05-08T18:21:19+08:00" level=debug msg="module module.config.notify config sample:{Module:module.config.notify ModuleType:module.config.notify Disabled:false Process:ob_agentctl Config:[map[notifyAddress:api/v1/module/config/notify process:ob_mgragent] map[notifyAddress:api/v1/module/config/notify process:ob_monagent] map[local:true process:ob_agentctl]]}"
time="2023-05-08T18:21:19+08:00" level=debug msg="decrypt config key agent.http.basic.auth.password rawVal-length:4"
time="2023-05-08T18:21:19+08:00" level=info msg="sdk inited"
time="2023-05-08T18:21:19+08:00" level=debug msg="decrypt config key agent.http.basic.auth.password rawVal-length:4"
time="2023-05-08T18:21:19+08:00" level=debug msg="register moduleType monagent.log.config config callback start"
time="2023-05-08T18:21:19+08:00" level=debug msg="register moduleType monagent.log.config config callback end"
time="2023-05-08T18:21:19+08:00" level=debug msg="register moduleType monagent.pipeline config callback start"
time="2023-05-08T18:21:19+08:00" level=debug msg="register moduleType monagent.pipeline config callback end"
time="2023-05-08T18:21:19+08:00" level=debug msg="register moduleType monagent.server.basic.auth config callback start"
time="2023-05-08T18:21:19+08:00" level=debug msg="register moduleType monagent.server.basic.auth config callback end"
time="2023-05-08T18:21:19+08:00" level=debug msg="module monagent.log.config config sample:{Module:monagent.log.config ModuleType:monagent.log.config Disabled:false Process:ob_monagent Config:map[compress:${monagent.log.compress} filename:${ocp.agent.home.path}/log/monagent.log level:${monagent.log.level} maxage:${monagent.log.maxage.days} maxbackups:${monagent.log.maxbackups} maxsize:${monagent.log.maxsize.mb}]}"
time="2023-05-08T18:21:19+08:00" level=debug msg="decrypt config key agent.http.basic.auth.password rawVal-length:4"

【问题现象及影响】

【附件】

能放一下obd的日志吗

这个可以吗
obd_display-trace.txt (40.7 KB)

帮忙放一下mgragent.log ,和ob_monagent.error.log这些文件都在一个目录下

好的
mgragent.log (8.5 KB)
ob_mgragent.error.log (24.8 KB)

看日志应该是obagent起成功了,后面可能有把agent kill掉了,里面的pid文件没删除。
解决方案:
1、可以先kill掉白屏服务 然后在黑屏上执行obd cluster destory {deployname} -f 会清空环境
2、obd web 启动白屏重新部署

我们通过obd白屏已经部署很多次了,每次是OBProxy和OCPExpress错误,我重新又建了一个单机环境部署了一次,重新开了一个帖子:obd白屏安装错误:OBProxy与OCPExpress失败