使用自签名 HTTPS 证书反代 OCP 站点时，无法通过 OBProxy 连接集群

OceanBase 技术问题
, ,
#1

使用环境

  • 测试环境
    • CentOS 7.9
  • 组件及版本
    • OceanBase 4.3.2
    • OCP 4.3.0
    • OBProxy 4.2.3 & 4.2.1

问题描述

通过 OCP 部署 OB 集群时，如果在 OCP 配置项 ocp.site.url 配置了 HTTPS 协议的内网域名，且使用的是自签名证书，那么集群就无法通过 OBProxy 访问了，大概是因为 OBProxy 无法获取到集群相关信息。

尝试添加自签名 CA 证书到系统级证书库，在 shell 中生效，在 OBProxy 中未生效。

异常现象如下：

  1. 直连 OB 集群的 OBServer 节点，完全正常；
  2. 通过 OBProxy 连接集群会提示 ERROR 4669 (HY000): cluster not exist 错误；
  3. 在 OCP 日志中会出现很多包含 server_ip={*Not IP address [0]*:0} 的错误信息。

PS. 截图是在 OBProxy 4.2.1 版本截取的，因为在 OBProxy 4.2.3 中是 INFO 而不是 WARN 等级的日志，并且是 succ 状态。（why?）

image
image1686×831 180 KB

复现路径

  1. 正常安装 OCP 组件和 OB 集群，使用自签名证书给 OCP 站点添加 HTTPS 反代；
  2. 将 OCP 的配置项 ocp.site.url 配置为 https://ocp.xxx.com 后重启 OCP 组件；
  3. 将自签名 CA 证书添加到 OBProxy 主机的系统级证书库中；（建议重启服务器）
  4. 创建 OBProxy 集群并连接到 OB 集群；（必须重新创建）
  5. 使用 mysql -hobproxy.xxx.com -P2883 -uroot@sys#obcluster -p 连接集群；

临时解决方案

将 OCP 的配置项 ocp.site.url 配置为 http://xxx.xxx.xxx.xxx:8080 后重启 OCP 组件，再重新创建 OBProxy 集群即可。

发帖主要是想咨询一下还有没有其他解决方案。

这不是最佳解决方案，希望官方能排查并解决这个自签名证书的异常

附：

不单独开贴了，部署 OCP 组件时填写的 OCP 配置，租户名字包含横杠会导致安装失败，比如 ocp-meta 这种。

2 个赞
#3

问题1: 通过 OCP 部署 OB 集群时，如果在 OCP 配置项 ocp.site.url 配置了 HTTPS 协议的内网域名，且使用的是自签名证书，那么集群就无法通过 OBProxy 访问了，大概是因为 OBProxy 无法获取到集群相关信息。

ocp.site.url配置为https://HTTPS 协议的内网域名 后外部访问ocp是否正常？

image
image1720×386 75.1 KB

问题2: 截图是在 OBProxy 4.2.1 版本截取的，因为在 OBProxy 4.2.3 中是 INFO 而不是 WARN 等级的日志，并且是 succ 状态。（why?）

可能不同版本日志告警级别做了调整，能否提供完整的obproxy.log，ocp-server.log

问题3: 部署 OCP 组件时填写的 OCP 配置，租户名字包含横杠会导致安装失败，比如 ocp-meta 这种。
应该是租户命名规范
https://www.oceanbase.com/docs/common-oceanbase-database-cn-1000000001050388

#4

补充一个解决方案：

  1. 将配置项 ocp.config-url.site.url 配置为 http://xxx.xxx.xxx.xxx:8080
  2. 将配置项 ocp.site.url 配置为 https://ocp.xxx.com （需重启 OCP 组件）
  3. 重新创建 OBProxy 集群

还是希望能统一配置为内网域名

#5

复现路径

是说 第4步正常，第5步异常 对吗？

#6

对的，使用 mysql 命令通过 OBProxy 连接集群会提示 ERROR 4669 (HY000): cluster not exist 错误。

#7

问题1：将 ocp.site.url 配置为 HTTPS 协议的内网域名后，外部可以正常访问 OCP 组件
问题2：日志我这边只导出了一份，仅供参考，见后
问题3：那么看来是部署 OCP 的白屏那边没做好租户名称的检测

  • OBPROXY_LOG diagnosis
[2024-08-03 22:02:08.394374] [13938][Y0-0000000000000000] [REBOOT](info="obproxy version: ObProxy-OceanBase 4.2.3.0-3, revision: 1-local-301841698608231dcec7a7a8b033ee9600ad874e, sysname: Linux, os release 3.10.0-1160.119.1.el7.x86_64, machine: x86_64, pid: 13938, ppid: 1, result: reboot success")
[2024-08-03 22:02:19.980722] [13938][Y0-00007FB4BCD38710] [LOGIN](trace_type="PROXY_INTERNAL_TRACE", connection_diagnosis={cs_id:1, ss_id:0, proxy_session_id:0, server_session_id:0, client_addr:"172.31.8.89:35072", server_addr:"*Not IP address [0]*:0", cluster_name:"obcloud", tenant_name:"proxysys", user_name:"root", error_code:0, error_msg:"Internal error", request_cmd:"OB_MYSQL_COM_LOGIN", sql_cmd:"OB_MYSQL_COM_LOGIN", req_total_time(us):363}{user_sql:""})
[2024-08-03 22:02:19.985960] [13938][Y0-00007FB4BC8D6710] [LOGIN](trace_type="PROXY_INTERNAL_TRACE", connection_diagnosis={cs_id:2, ss_id:0, proxy_session_id:0, server_session_id:0, client_addr:"172.31.8.89:35074", server_addr:"*Not IP address [0]*:0", cluster_name:"obcloud", tenant_name:"proxysys", user_name:"root", error_code:0, error_msg:"Internal error", request_cmd:"OB_MYSQL_COM_LOGIN", sql_cmd:"OB_MYSQL_COM_LOGIN", req_total_time(us):122}{user_sql:""})
[2024-08-03 22:02:34.489725] [13938][Y0-00007FB4BC8D6710] [LOGIN](trace_type="PROXY_INTERNAL_TRACE", connection_diagnosis={cs_id:16, ss_id:0, proxy_session_id:0, server_session_id:0, client_addr:"172.31.8.89:35150", server_addr:"*Not IP address [0]*:0", cluster_name:"obcloud", tenant_name:"proxysys", user_name:"root", error_code:0, error_msg:"Internal error", request_cmd:"OB_MYSQL_COM_LOGIN", sql_cmd:"OB_MYSQL_COM_LOGIN", req_total_time(us):176}{user_sql:""})
[2024-08-03 22:02:34.490922] [13938][Y0-00007FB4BCD38710] [LOGIN](trace_type="PROXY_INTERNAL_TRACE", connection_diagnosis={cs_id:17, ss_id:0, proxy_session_id:0, server_session_id:0, client_addr:"172.31.8.89:35152", server_addr:"*Not IP address [0]*:0", cluster_name:"obcloud", tenant_name:"proxysys", user_name:"root", error_code:0, error_msg:"Internal error", request_cmd:"OB_MYSQL_COM_LOGIN", sql_cmd:"OB_MYSQL_COM_LOGIN", req_total_time(us):167}{user_sql:""})
[2024-08-03 22:02:45.340567] [13938][Y0-00007FB4BC8D6710] [CONNECTION](trace_type="PROXY_INTERNAL_TRACE", connection_diagnosis={cs_id:524294, ss_id:0, proxy_session_id:0, server_session_id:0, client_addr:"127.0.0.1:53568", server_addr:"*Not IP address [0]*:0", cluster_name:"", tenant_name:"", user_name:"", error_code:0, error_msg:"Connect error", request_cmd:"OB_MYSQL_COM_SLEEP", sql_cmd:"OB_MYSQL_COM_HANDSHAKE", req_total_time(us):36858}{user_sql:""})
[2024-08-03 22:02:46.332802] [13938][Y0-00007FB4BC8D6710] [CONNECTION](trace_type="PROXY_INTERNAL_TRACE", connection_diagnosis={cs_id:524295, ss_id:0, proxy_session_id:0, server_session_id:0, client_addr:"127.0.0.1:53572", server_addr:"*Not IP address [0]*:0", cluster_name:"", tenant_name:"", user_name:"", error_code:0, error_msg:"Connect error", request_cmd:"OB_MYSQL_COM_SLEEP", sql_cmd:"OB_MYSQL_COM_HANDSHAKE", req_total_time(us):102}{user_sql:""})
[2024-08-03 22:09:02.084893] [13938][Y0-00007FB4BCD3A650] [CONNECTION](trace_type="PROXY_INTERNAL_TRACE", connection_diagnosis={cs_id:524737, ss_id:0, proxy_session_id:0, server_session_id:0, client_addr:"127.0.0.1:54404", server_addr:"*Not IP address [0]*:0", cluster_name:"", tenant_name:"", user_name:"", error_code:0, error_msg:"Connect error", request_cmd:"OB_MYSQL_COM_SLEEP", sql_cmd:"OB_MYSQL_COM_HANDSHAKE", req_total_time(us):171}{user_sql:""})
[2024-08-03 22:10:23.174783] [13938][Y0-00007FB4BCD3A650] [CONNECTION](trace_type="PROXY_INTERNAL_TRACE", connection_diagnosis={cs_id:524834, ss_id:0, proxy_session_id:0, server_session_id:0, client_addr:"127.0.0.1:54582", server_addr:"*Not IP address [0]*:0", cluster_name:"", tenant_name:"", user_name:"", error_code:0, error_msg:"Connect error", request_cmd:"OB_MYSQL_COM_SLEEP", sql_cmd:"OB_MYSQL_COM_HANDSHAKE", req_total_time(us):73}{user_sql:""})
[2024-08-03 22:11:23.733915] [13938][Y0-00007FB4BC8D8650] [CONNECTION](trace_type="PROXY_INTERNAL_TRACE", connection_diagnosis={cs_id:524905, ss_id:0, proxy_session_id:0, server_session_id:0, client_addr:"127.0.0.1:54716", server_addr:"*Not IP address [0]*:0", cluster_name:"", tenant_name:"", user_name:"", error_code:0, error_msg:"Connect error", request_cmd:"OB_MYSQL_COM_SLEEP", sql_cmd:"OB_MYSQL_COM_HANDSHAKE", req_total_time(us):159}{user_sql:""})
[2024-08-03 22:16:42.990986] [13938][Y0-00007FB4BCD38710] [CONNECTION](trace_type="PROXY_INTERNAL_TRACE", connection_diagnosis={cs_id:525282, ss_id:0, proxy_session_id:0, server_session_id:0, client_addr:"127.0.0.1:55422", server_addr:"*Not IP address [0]*:0", cluster_name:"", tenant_name:"", user_name:"", error_code:0, error_msg:"Connect error", request_cmd:"OB_MYSQL_COM_SLEEP", sql_cmd:"OB_MYSQL_COM_HANDSHAKE", req_total_time(us):269}{user_sql:""})
[2024-08-03 22:17:08.653876] [13938][Y0-00007FB4BC8D6710] [CONNECTION](trace_type="PROXY_INTERNAL_TRACE", connection_diagnosis={cs_id:525309, ss_id:0, proxy_session_id:0, server_session_id:0, client_addr:"127.0.0.1:55480", server_addr:"*Not IP address [0]*:0", cluster_name:"", tenant_name:"", user_name:"", error_code:0, error_msg:"Connect error", request_cmd:"OB_MYSQL_COM_SLEEP", sql_cmd:"OB_MYSQL_COM_HANDSHAKE", req_total_time(us):176}{user_sql:""})
[2024-08-03 22:19:27.277802] [13938][Y0-00007FB4BC8D8650] [CONNECTION](trace_type="PROXY_INTERNAL_TRACE", connection_diagnosis={cs_id:525482, ss_id:0, proxy_session_id:0, server_session_id:0, client_addr:"127.0.0.1:55786", server_addr:"*Not IP address [0]*:0", cluster_name:"", tenant_name:"", user_name:"", error_code:0, error_msg:"Connect error", request_cmd:"OB_MYSQL_COM_SLEEP", sql_cmd:"OB_MYSQL_COM_HANDSHAKE", req_total_time(us):87}{user_sql:""})
[2024-08-03 22:19:34.053710] [13938][Y0-00007FB4BC8D8650] [LOGIN](trace_type="LOGIN_TRACE", connection_diagnosis={cs_id:525490, ss_id:0, proxy_session_id:0, server_session_id:0, client_addr:"172.31.8.89:38480", server_addr:"*Not IP address [0]*:0", cluster_name:"TestCluster", tenant_name:"TestTenant", user_name:"root", error_code:-4669, error_msg:"all cluster info is empty, please check the config server", request_cmd:"OB_MYSQL_COM_LOGIN", sql_cmd:"OB_MYSQL_COM_LOGIN", req_total_time(us):689}{internal_sql:"", login_result:"failed"})
[2024-08-03 22:19:45.034369] [13938][Y0-00007FB4BC8D8650] [LOGIN](trace_type="LOGIN_TRACE", connection_diagnosis={cs_id:525506, ss_id:0, proxy_session_id:0, server_session_id:0, client_addr:"172.31.8.89:38538", server_addr:"*Not IP address [0]*:0", cluster_name:"TestCluster", tenant_name:"TestTenant", user_name:"root", error_code:-4669, error_msg:"all cluster info is empty, please check the config server", request_cmd:"OB_MYSQL_COM_LOGIN", sql_cmd:"OB_MYSQL_COM_LOGIN", req_total_time(us):187}{internal_sql:"", login_result:"failed"})
  • OBPROXY_LOG error
2024-08-03 22:19:34.053737,TestProxy,,,,TestCluster:TestTenant:,OB_MYSQL,,,OB_MYSQL_COM_LOGIN,,failed,-4669,,686us,0us,0us,0us,Y0-00007FB4BC8D8650,,172.31.8.89:38480,,0,,cluster not exist,
2024-08-03 22:19:45.034410,TestProxy,,,,TestCluster:TestTenant:,OB_MYSQL,,,OB_MYSQL_COM_LOGIN,,failed,-4669,,184us,0us,0us,0us,Y0-00007FB4BC8D8650,,172.31.8.89:38538,,0,,cluster not exist,
  • OBPROXY_LOG stat
2024-08-03 22:19:59.500339,TestProxy,,TestCluster:TestTenant:,OB_MYSQL,LOGIN,failed,-4669,2,0,0,0,870us,0us,0us

补充：如有需要可重新复现再导出一份

#8

将 OCP 的配置项 ocp.site.url 配置为 http://xxx.xxx.xxx.xxx:8080 后重启 OCP 组件，再重新创建 OBProxy 集群即可。

或者

  1. 将配置项 ocp.config-url.site.url 配置为 http://xxx.xxx.xxx.xxx:8080
  2. 将配置项 ocp.site.url 配置为 https://ocp.xxx.com （需重启 OCP 组件）
  3. 重新创建 OBProxy 集群

后，第5步正常吗？

#9

这两种方法都能够正常连接的。如果官方不认为这是异常，那么建议在文档上补充一下说明。

#10

ocp.site.url 默认配置就是 http://IP:8080， 和你的方案一是一致的
使用 mysql -hobproxy.xxx.com -P2883 -uroot@sys#obcluster -p 连接集群 正常，是符合预期的