ocp-express没有数据

bull020 · 2024 年6 月 7 日 20:53

环境：
os：ubuntu 2204
ob：4.3.0.1
OceanBase Deploy: 2.7.0

admin@ubuntu:/data/admin/obproxy/log$ obd mirror list local
+---------------------------------------------------------------------------------------------------------------------+
|                                                  local Package List                                                 |
+---------------------------+-----------+------------------------+---------+------------------------------------------+
| name                      | version   | release                | arch    | md5                                      |
+---------------------------+-----------+------------------------+---------+------------------------------------------+
| grafana                   | 7.5.17    | 1                      | x86_64  | 1bf1f338d3a3445d8599dc6902e7aeed4de4e0d6 |
| libobclient               | 2.2.1     | 2.el8                  | x86_64  | 97d7de0b610b296125ab511d19937a80745a090d |
| obagent                   | 4.2.2     | 100000042024011120.el8 | x86_64  | bf152b880953c2043ddaf80d6180cf22bb8c8ac2 |
| obclient                  | 2.2.2     | 1.el8                  | x86_64  | 725024d45f7ac20f0320ea9fbecf94b77f1cf626 |
| ob-configserver           | 1.0.0     | 2.el8                  | x86_64  | 664f93205c913d5dc84e0779d565768fd60f1d5e |
| ob-deploy                 | 2.7.0     | 5.el8                  | x86_64  | bcbbe9844e277da1414d1455d2bd66d1d78088a1 |
| obproxy-ce                | 4.2.3.0   | 3.el8                  | x86_64  | 2526073e3c652177b15093be611af94a469e0e21 |
| oceanbase-ce              | 4.3.0.1   | 100000242024032211.el8 | x86_64  | d475e3bfe98d2a741a7bbb50f5f76345d778c9b8 |
| oceanbase-ce-libs         | 4.3.0.1   | 100000242024032211.el8 | x86_64  | ebb224dbfdc2c0585f3f722ef052269959f54dc4 |
| oceanbase-diagnostic-tool | 1.6.2     | 12024031514.el8        | x86_64  | df49965c9af6d982b90d88544ead279d1a140352 |
| ocp-express               | 4.2.2     | 100000022024011120.el8 | x86_64  | e5c152ebdd65839ed5f5521ff6c73e6a29cb9e75 |
| openjdk-jre               | 1.8.0_322 | b09.el8                | x86_64  | 45e4b8c216b090a63583b70743c54e87b7b69aad |
| prometheus                | 2.37.1    | 10000102022110211.el8  | x86_64  | e4f8a3e784512fca75bf1b3464247d1f31542cb9 |
| ob-deploy                 | 2.7.0     | 5.el7                  | x86_64  | a0683dc9bd3499ff0d2c5678b4558e234abfa307 |
| obproxy-ce                | 4.2.1.0   | 11.el7                 | x86_64  | 0aed4b782120e4248b749f67be3d2cc82cdcb70d |
| oceanbase-ce              | 4.2.1.2   | 102010022023121415.el7 | x86_64  | 1873bbe80cbbe5d00d5f276a4f7302cfca677fb6 |
| oceanbase-ce-libs         | 4.2.1.2   | 102010022023121415.el7 | x86_64  | 3d3eeee6694be9b8debb0c3f8b5b6583c06e3b62 |
| oceanbase-ce-utils        | 4.2.1.2   | 102010022023121415.el7 | x86_64  | 8d9490944c0cf35efe36fee39d5b56c622a2c82f |
| oceanbase-diagnostic-tool | 1.5.2     | 22024010415.el7        | x86_64  | 4c0f0551be4ad04bc40a66763f66f76fb8bdf10c |
| ocp-agent-ce              | 4.2.2     | 20240315150922.el7     | aarch64 | 565b5ab22ff384c246b4a707bab845048274cc04 |
| ocp-agent-ce              | 4.2.2     | 20240315150922.el7     | x86_64  | 5d25b0d8b36f66a87ac824a4e04d4966302eaf75 |
| ocp-server-ce             | 4.2.2     | 20240315150922.el7     | noarch  | 84a65bf0ec004e73b772938696b2b896a0ef2b2f |
| openjdk-jre               | 1.8.0_322 | b09.el7                | x86_64  | 051aa69c5abb8697d15c2f0dcb1392b3f815f7ed |
+---------------------------+-----------+------------------------+---------+------------------------------------------+

数据库性能这块没有数据，怎么排查呢？

bull020 · 2024 年6 月 7 日 21:08

不知道是否跟obagent有关系呢？ monagent有日志报错：

admin@ubuntu:~/obagent/log$ tail -100f monagent.log  |grep ERROR
2024-06-07T21:00:15.30255+08:00 ERROR [14711,] caller=obcommon/connectivity.go:151:initDbConnections: target oceanbase ping failed fields: error="Error 1045: Access denied for user 'ocp_monitor'@'xxx.xxx.xxx.xxx' (using password: NO)"
2024-06-07T21:00:15.3026+08:00 ERROR [14711,] caller=inputs/ocp.go:40:func2: init connectivityInput failed fields: error="init DB connections failed: Error 1045: Access denied for user 'ocp_monitor'@'xxx.xxx.xxx.xxx' (using password: NO)"
2024-06-07T21:00:15.30264+08:00 ERROR [14711,] caller=engine/config_manager.go:202:handleConfigEvent: CreatePipelineV2s init source failed fields:, error="init DB connections failed: Error 1045: Access denied for user 'ocp_monitor'@'xxx.xxx.xxx.xxx' (using password: NO)"
2024-06-07T21:00:15.30266+08:00 ERROR [14711,5c66d8a3b0aa4b1e] caller=engine/config_manager.go:155:handleAddEvent: CreatePipelines failed fields: error="init DB connections failed: Error 1045: Access denied for user 'ocp_monitor'@'xxx.xxx.xxx.xxx' (using password: NO)"
2024-06-07T21:00:18.15988+08:00 ERROR [14711,] caller=plugins/registry.go:137:GetPlugin: init tableInput failed fields:, error="db ping: Error 1045: Access denied for user 'ocp_monitor'@'xxx.xxx.xxx.xxx' (using password: NO)"
2024-06-07T21:00:18.15992+08:00 ERROR [14711,] caller=engine/config_manager.go:202:handleConfigEvent: CreatePipelineV2s init source failed fields:, error="db ping: Error 1045: Access denied for user 'ocp_monitor'@'xxx.xxx.xxx.xxx' (using password: NO)"
2024-06-07T21:00:18.15994+08:00 ERROR [14711,5c66d8a3b0aa4b1e] caller=engine/config_manager.go:155:handleAddEvent: CreatePipelines failed fields: error="db ping: Error 1045: Access denied for user 'ocp_monitor'@'xxx.xxx.xxx.xxx' (using password: NO)"

bull020 · 2024 年6 月 7 日 21:13

bull020 · 2024 年6 月 7 日 21:26

这里输入密码后，monagent.log就没有错误输出。但是奇怪是数据库性能版块还是没有数据。
想再咨询下：
1）部署了ocp-express、ocp 两个组件，前者创建一个租户，后者创建2个租户：
ocp-express对应的租户配置：

    ocp_meta_db: ocp_express # The database name of ocp express meta
    ocp_meta_username: meta # The username of ocp express meta
    ocp_meta_password: obdemometa # The password of ocp express meta
    ocp_agent_monitor_password: obdemoagent # The password for obagent monitor user
    ocp_root_password: ObDemo_@123
    ocp_meta_tenant: # The config for ocp express meta tenant
      tenant_name: ocp
      max_cpu: 1
      memory_size: 2G
      log_disk_size: 7680M # The recommend value is (4608 + (expect node num + expect tenant num) * 512) M.

ocp对应的租户配置：

    ocp_meta_tenant:
      tenant_name: ocp_meta
      max_cpu: 2.0
      memory_size: 2G
      ocp_meta_username: root # User to use under ocp meta tenant
      ocp_meta_password: ocp_meta # Password used to connect to ocp meta tenant
      ocp_meta_db: meta_database # Database used to store ocp meta data
    # OCP monitor tenant definition, including tenant name, cpu and memory
    ocp_monitor_tenant:
      tenant_name: ocp_monitor
      max_cpu: 2.0
      memory_size: 2G
      ocp_monitor_username: root # User to use under ocp monitor tenant
      ocp_monitor_password: ocp_monitor # Password used to connect to ocp meta tenant
      ocp_monitor_db: monitor_database # Database used to store ocp meta data

问题：
1）ocp-express 连接的账号密码为什么不是这里：

ocp_meta_db: ocp_express # The database name of ocp express meta
    ocp_meta_username: meta # The username of ocp express meta
    ocp_meta_password: obdemometa # The password of ocp express meta
    ocp_agent_monitor_password: obdemoagent # The password for obagent monitor user
    ocp_root_password: ObDemo_@123
    ocp_meta_tenant: # The config for ocp express meta tenant
      tenant_name: ocp

为什么ocp-express web要输入是ocp配置的ocp_monitor租户和密码？

    ocp_meta_tenant:
      tenant_name: ocp_meta
      max_cpu: 2.0
      memory_size: 2G
      ocp_meta_username: root # User to use under ocp meta tenant
      ocp_meta_password: ocp_meta # Password used to connect to ocp meta tenant
      ocp_meta_db: meta_database # Database used to store ocp meta data
    # OCP monitor tenant definition, including tenant name, cpu and memory
    ocp_monitor_tenant:
      tenant_name: ocp_monitor
      max_cpu: 2.0
      memory_size: 2G
      ocp_monitor_username: root # User to use under ocp monitor tenant
      ocp_monitor_password: ocp_monitor # Password used to connect to ocp meta tenant
      ocp_monitor_db: monitor_database # Database used to store ocp meta data

这两个组件是不是有共同默认的租户呢？

王利博 · 2024 年6 月 8 日 23:14

看租户名称：是ocp_monitor租户。所以需要输入 ocp_meta_password: ocp_meta的密码

bull020 · 2024 年6 月 9 日 21:32

输入了，目前没有弹框提示，但是数据库性能 tab页还是没有数据呢

辞霜 · 2024 年6 月 11 日 11:19

看看agent日志是否还有报错，黑屏查一下monitor租户创建成功没。

bull020 · 2024 年6 月 11 日 11:33

2024-06-11T11:26:51.78516+08:00 ERROR [57046,] caller=inputs/ocp.go:40:func2: init connectivityInput failed fields:, error=“init DB connections failed: Error 1045: Access denied for user ‘ocp_monitor’@‘xxx.xxx.xxx.xxx’ (using password: NO)”
2024-06-11T11:26:51.78519+08:00 ERROR [57046,] caller=engine/config_manager.go:202:handleConfigEvent: CreatePipelineV2s init source failed fields: error=“init DB connections failed: Error 1045: Access denied for user ‘ocp_monitor’@‘xxx.xxx.xxx.xxx’ (using password: NO)”
2024-06-11T11:26:51.78522+08:00 ERROR [57046,c7188c2943b852f4] caller=engine/config_manager.go:155:handleAddEvent: CreatePipelines failed fields: error=“init DB connections failed: Error 1045: Access denied for user ‘ocp_monitor’@‘xxx.xxx.xxx.xxx’ (using password: NO)”

monagent.log 有这样日志报错，但是：

那就是obagent的配置的连接observer账号密码有误？

辞霜 · 2024 年6 月 11 日 17:05

使用ocp-express安装也是存在这些租户的，你是否是先安装的ocp再安装的ocp-express

王利博 · 2024 年6 月 11 日 17:13

升级下obd2.7.0为最新版obd2.9.0，再试试看。
参考方式：【SOP 系列 17】使用 OBD 在线/离线升级 OceanBase 集群

bull020 · 2024 年6 月 11 日 17:28

ocp-express是通过yaml配置，和observer、obagent、obconfigserver、obproxy 一起部署的，可以确认的是部署后没有 ocp_meta、ocp_monitor 这个两租户的，这个时候数据是正常的，数据库性能tab页是有数据的。默认的ocp-express租户也是yaml中配置的。如下：

    ocp_meta_db: ocp_express # The database name of ocp express meta
    ocp_meta_username: meta # The username of ocp express meta
    ocp_meta_password: obdemometa # The password of ocp express meta
    ocp_agent_monitor_password: obdemoagent # The password for obagent monitor user
    ocp_root_password: ObDemo_@123
    ocp_meta_tenant: # The config for ocp express meta tenant
      tenant_name: ocp
      max_cpu: 1
      memory_size: 2G
      log_disk_size: 7680M # The recommend value is (4608 + (expect node num + expect tenant num) * 512) M.

后来单独手工创建了两个租户ocp_meta、ocp_monitor,供ocp使用。
然后又通过docker方式部署了ocp社区版。
然后ocp-express 数据库性能 tab页就没有数据了

这个时候docker部署的ocp还没有托管observer，ocp web上还是空白。

bull020 · 2024 年6 月 11 日 17:31

所以我是怀疑 ocp-express和ocp部署是不是有一写交叉的东西，比如agent访问observer采集信息。

王利博 · 2024 年6 月 11 日 18:01

看看obagent进行在不在。ps -ef |grep agent 一般express使用的是obgaent ocp是ocp-agent

bull020 · 2024 年6 月 12 日 09:56

在的，主机性能tab页还是有数据的，如果obagent不在的话，这个页面就没有数据。

王利博 · 2024 年6 月 12 日 15:08

这两个不冲突。我这边看着是没问题

筛选时间调整试试看

bull020 · 2024 年6 月 12 日 15:18

王利博 · 2024 年6 月 12 日 15:23

提供下agent日志看看。

bull020 · 2024 年6 月 12 日 15:25

bull020 · 2024 年6 月 12 日 15:44

bull020 · 2024 年6 月 12 日 15:45

2024-06-04T16:04:37.87684+08:00 INFO [62841,3bffe1233aab90e6] caller=http/http_command.go:59:func1: handling command request /api/v1/status
2024-06-04T16:04:37.97942+08:00 INFO [62841,646d46be419d1d2d] caller=http/http_command.go:102:writeOk: command request /api/v1/status succeed
2024-06-04T16:04:52.13223+08:00 INFO [62841,a4de394712c92b43] caller=http/http_command.go:59:func1: handling command request /api/v1/status
2024-06-04T16:04:52.18723+08:00 INFO [62841,c4cdfb06d93a4576] caller=http/http_command.go:102:writeOk: command request /api/v1/status succeed
2024-06-04T16:04:52.52121+08:00 INFO [62841,74197e28f68e9798] caller=http/http_command.go:59:func1: handling command request /api/v1/status
2024-06-04T16:04:52.577+08:00 INFO [62841,7f26871ec15c45ae] caller=http/http_command.go:102:writeOk: command request /api/v1/status succeed
2024-06-04T16:40:57.25603+08:00 INFO [62841,9b7d60fc0f02398a] caller=http/http_command.go:59:func1: handling command request /api/v1/status
2024-06-04T16:40:57.30994+08:00 INFO [62841,74350d4752017db0] caller=http/http_command.go:102:writeOk: command request /api/v1/status succeed
2024-06-04T16:40:57.6474+08:00 INFO [62841,be2324bd6b63102b] caller=http/http_command.go:59:func1: handling command request /api/v1/status
2024-06-04T16:40:57.70469+08:00 INFO [62841,40f2c8e63ae53a0f] caller=http/http_command.go:102:writeOk: command request /api/v1/status succeed
2024-06-04T17:37:02.33465+08:00 INFO [62841,6fd5ee0c7c1105be] caller=http/http_command.go:59:func1: handling command request /api/v1/status
2024-06-04T17:37:02.39298+08:00 INFO [62841,7b696cc32c43bdf6] caller=http/http_command.go:102:writeOk: command request /api/v1/status succeed
2024-06-04T17:37:02.72622+08:00 INFO [62841,7b829c5c6f3e6958] caller=http/http_command.go:59:func1: handling command request /api/v1/status
2024-06-04T17:37:02.78199+08:00 INFO [62841,def6ca56e2dea6d8] caller=http/http_command.go:102:writeOk: command request /api/v1/status succeed
2024-06-04T19:32:18.35567+08:00 INFO [62841,4c97440b7e84b9a5] caller=http/http_command.go:59:func1: handling command request /api/v1/status
2024-06-04T19:32:18.41181+08:00 INFO [62841,23069aa516d7c402] caller=http/http_command.go:102:writeOk: command request /api/v1/status succeed
2024-06-04T19:32:18.75112+08:00 INFO [62841,c56225def6d04908] caller=http/http_command.go:59:func1: handling command request /api/v1/status
2024-06-04T19:32:18.80708+08:00 INFO [62841,0cb9dce2e92651e0] caller=http/http_command.go:102:writeOk: command request /api/v1/status succeed
2024-06-08T11:54:53.13172+08:00 INFO [10799,] caller=agentd/main.go:64:run: starting agentd with config /home/admin/obagent/conf/agentd.yaml
2024-06-08T11:54:53.13235+08:00 INFO [10799,] caller=agentd/limit_linux.go:32:newLimiter: create service ob_mgragent resource limit skipped, no limit in config
2024-06-08T11:54:53.13276+08:00 WARN [10799,] caller=agentd/limit_linux.go:37:newLimiter: create cgroup for service ob_monagent failed, fallback to watch limiter. only memory quota will affect!
2024-06-08T11:54:53.13311+08:00 INFO [10799,] caller=agentd/watchdog.go:93:Start: starting agentd
2024-06-08T11:54:53.1337+08:00 INFO [10799,] caller=agentd/watchdog.go:106:Start: starting socket listener on ‘/home/admin/obagent/run/ob_agentd.10799.sock’
2024-06-08T11:54:53.13384+08:00 INFO [10799,] caller=agentd/watchdog.go:116:Start: starting service ‘ob_mgragent’
2024-06-08T11:54:53.13393+08:00 INFO [10799,] caller=agentd/service.go:144:startProc: starting service fields:, service=ob_mgragent
2024-06-08T11:54:53.13438+08:00 INFO [10799,] caller=agentd/service.go:151:startProc: service process started. pid: 10805 fields:, service=ob_mgragent
2024-06-08T11:54:53.13444+08:00 INFO [10799,] caller=agentd/watchdog.go:116:Start: starting service ‘ob_monagent’
2024-06-08T11:54:53.13451+08:00 INFO [10799,] caller=agentd/service.go:144:startProc: starting service fields:, service=ob_monagent
2024-06-08T11:54:53.13485+08:00 INFO [10799,] caller=agentd/service.go:151:startProc: service process started. pid: 10806 fields: service=ob_monagent
2024-06-08T11:54:53.13488+08:00 INFO [10799,] caller=agentd/watchdog.go:122:Start: agentd started
2024-06-08T11:54:54.12192+08:00 INFO [10799,b55d557c9ff296d3] caller=http/http_command.go:59:func1: handling command request /api/v1/status
2024-06-08T11:54:54.23239+08:00 INFO [10799,600fc02b1038041e] caller=http/http_command.go:102:writeOk: command request /api/v1/status succeed
2024-06-08T11:55:02.64081+08:00 INFO [10799,4b5fb5c93edf85e3] caller=http/http_command.go:59:func1: handling command request /api/v1/status
2024-06-08T11:55:02.70041+08:00 INFO [10799,448bc1cadd2d520a] caller=http/http_command.go:102:writeOk: command request /api/v1/status succeed
2024-06-08T11:55:03.16305+08:00 INFO [10799,0658d507e7032256] caller=http/http_command.go:59:func1: handling command request /api/v1/status
2024-06-08T11:55:03.22513+08:00 INFO [10799,7e50b11b4f245f55] caller=http/http_command.go:102:writeOk: command request /api/v1/status succeed
2024-06-08T12:03:56.95963+08:00 INFO [10799,d130080e4625ef28] caller=http/http_command.go:59:func1: handling command request /api/v1/status
2024-06-08T12:03:57.01825+08:00 INFO [10799,847c4d42f3cb900c] caller=http/http_command.go:102:writeOk: command request /api/v1/status succeed
2024-06-08T12:03:57.48227+08:00 INFO [10799,ff978c8e76b83bc3] caller=http/http_command.go:59:func1: handling command request /api/v1/status
2024-06-08T12:03:57.54146+08:00 INFO [10799,1c108fe19e2e0924] caller=http/http_command.go:102:writeOk: command request /api/v1/status succeed
2024-06-08T12:05:56.58886+08:00 INFO [10799,6846ed88f2e8abb0] caller=http/http_command.go:59:func1: handling command request /api/v1/status
2024-06-08T12:05:56.64977+08:00 INFO [10799,2eda6671ed3446b3] caller=http/http_command.go:102:writeOk: command request /api/v1/status succeed
2024-06-08T12:11:19.67239+08:00 INFO [57039,] caller=agentd/main.go:64:run: starting agentd with config /home/admin/obagent/conf/agentd.yaml
2024-06-08T12:11:19.67306+08:00 INFO [57039,] caller=agentd/limit_linux.go:32:newLimiter: create service ob_mgragent resource limit skipped, no limit in config
2024-06-08T12:11:19.67353+08:00 WARN [57039,] caller=agentd/limit_linux.go:37:newLimiter: create cgroup for service ob_monagent failed, fallback to watch limiter. only memory quota will affect!
2024-06-08T12:11:19.67391+08:00 INFO [57039,] caller=agentd/watchdog.go:93:Start: starting agentd
2024-06-08T12:11:19.67467+08:00 INFO [57039,] caller=agentd/watchdog.go:106:Start: starting socket listener on ‘/home/admin/obagent/run/ob_agentd.57039.sock’
2024-06-08T12:11:19.67485+08:00 INFO [57039,] caller=agentd/watchdog.go:116:Start: starting service ‘ob_mgragent’
2024-06-08T12:11:19.67494+08:00 INFO [57039,] caller=agentd/service.go:144:startProc: starting service fields: service=ob_mgragent
2024-06-08T12:11:19.67536+08:00 INFO [57039,] caller=agentd/service.go:151:startProc: service process started. pid: 57045 fields:, service=ob_mgragent
2024-06-08T12:11:19.6754+08:00 INFO [57039,] caller=agentd/watchdog.go:116:Start: starting service ‘ob_monagent’
2024-06-08T12:11:19.67547+08:00 INFO [57039,] caller=agentd/service.go:144:startProc: starting service fields: service=ob_monagent
2024-06-08T12:11:19.67579+08:00 INFO [57039,] caller=agentd/service.go:151:startProc: service process started. pid: 57046 fields:, service=ob_monagent
2024-06-08T12:11:19.67583+08:00 INFO [57039,] caller=agentd/watchdog.go:122:Start: agentd started
2024-06-08T12:11:20.6627+08:00 INFO [57039,2ab4ec8639ca1cee] caller=http/http_command.go:59:func1: handling command request /api/v1/status
2024-06-08T12:11:20.7702+08:00 INFO [57039,162ec36ac952ed86] caller=http/http_command.go:102:writeOk: command request /api/v1/status succeed
2024-06-08T12:11:34.84217+08:00 INFO [57039,847c76371f5faef4] caller=http/http_command.go:59:func1: handling command request /api/v1/status
2024-06-08T12:11:34.91265+08:00 INFO [57039,000ed84a95918349] caller=http/http_command.go:102:writeOk: command request /api/v1/status succeed
2024-06-08T12:11:35.36829+08:00 INFO [57039,adb5badd4ce5a190] caller=http/http_command.go:59:func1: handling command request /api/v1/status
2024-06-08T12:11:35.42815+08:00 INFO [57039,0183432ac4cf87b6] caller=http/http_command.go:102:writeOk: command request /api/v1/status succeed