ocp版本为4.4.0 备份无法连接到minio中，请问是版本不支持吗
发下observer详细版本
show variables like '%version_comment%';
你再重试一遍，然后发下ocp-server.log
2 个赞
obclient(root@sys)[oceanbase]> show variables like '%version_comment%';
+-----------------+------------------------------------------------------------------------------------------------------------------+
| Variable_name | Value |
+-----------------+------------------------------------------------------------------------------------------------------------------+
| version_comment | OceanBase_CE 4.4.1.0 (r100000032025101610-70824e90816ed333c56f3dd1af34ffec43d0fa88) (Built Oct 16 2025 10:20:44) |
+-----------------+------------------------------------------------------------------------------------------------------------------+
ocp-server.log 日志
cat /ocp-server/ocp/log/ocp-server.log | grep -C 50 'BACKUP_CONFIG_STORAGE_TEST_FAIL'
2025-12-22 17:59:19.405 ERROR 234776 --- [http-nio-0.0.0.0-8080-exec-1,6893241de63cfe32,47d7c1c71e898d96] c.o.o.b.internal.utils.ObBackupChecker : test storage config failed, directory not exists: s3://servicecenter-database-backup
2025-12-22 17:59:19.414 ERROR 234776 --- [http-nio-0.0.0.0-8080-exec-1,6893241de63cfe32,47d7c1c71e898d96] c.o.o.s.c.trace.RequestTracingAspect : Unexpected API Error: [POST /api/v2/ob/clusters/2/tenants/8/backup/config/storages/test client=10.137.44.215, traceId=6893241de63cfe32, args=2,8,TestStorageParam(hostIdList=null, clusterId=null, obBackupStorageBaseInfo=ObBackupStorageBaseInfo(backupStorageType=BACKUP_STORAGE_S3, storageUrl=servicecenter-database-backup, ossAccessKey=null, cosAccessKey=null, obsAccessKey=null, s3AccessKey=S3AccessKey(region=null, endpoint=https://k8s-minio-api-jw.crgdpharm.com, accessKeyId=xxx), customBackupRangeList=[], advancedConfigList=[BackupAdvancedConfigInfo(name=delete_mode, values=[delete]), BackupAdvancedConfigInfo(name=checksum_type, values=[md5]), BackupAdvancedConfigInfo(name=addressing_model, values=[virtual_hosted_style])]), backupPurposeType=BACKUP, storageId=null),, duration=6811 ms]
com.oceanbase.ocp.core.exception.UnexpectedException: [OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=BACKUP_CONFIG_STORAGE_TEST_FAIL, args=hostIpList:[10.36.40.188, 10.36.40.3],s3://servicecenter-database-backup
at com.oceanbase.ocp.backup.internal.utils.ObBackupChecker.checkStorageAccessible(ObBackupChecker.java:478)
at com.oceanbase.ocp.backup.internal.facadeimpl.common.BackupCommonServiceImpl.testStorageConfig(BackupCommonServiceImpl.java:648)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:359)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:724)
at com.oceanbase.ocp.backup.internal.facadeimpl.common.BackupCommonServiceImpl$$SpringCGLIB$$0.testStorageConfig(<generated>)
at com.oceanbase.ocp.server.common.controller.ObTenantBackupController.testStorageConfig(ObTenantBackupController.java:167)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:359)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:64)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:57)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:58)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89)
at com.oceanbase.ocp.server.common.aspect.OperationEventAspect.aroundAuditEvent(OperationEventAspect.java:132)
at jdk.internal.reflect.GeneratedMethodAccessor497.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:642)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:632)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:71)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.springframework.security.authorization.method.AuthorizationManagerBeforeMethodInterceptor.proceed(AuthorizationManagerBeforeMethodInterceptor.java:268)
at org.springframework.security.authorization.method.AuthorizationManagerBeforeMethodInterceptor.attemptAuthorization(AuthorizationManagerBeforeMethodInterceptor.java:263)
at org.springframework.security.authorization.method.AuthorizationManagerBeforeMethodInterceptor.invoke(AuthorizationManagerBeforeMethodInterceptor.java:196)
at org.springframework.security.config.annotation.method.configuration.DeferringMethodInterceptor.invoke(DeferringMethodInterceptor.java:44)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:728)
at com.oceanbase.ocp.server.common.controller.ObTenantBackupController$$SpringCGLIB$$0.testStorageConfig(<generated>)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:258)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:191)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:986)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:891)
1 个赞
存储配置测试失败，你在40.3和40.188 分别用s3客户端 访问这个bucket可以成功吗？
如果可以成功，你用黑屏设置备份看是否可以成功？
如果黑屏设置备份也可以成功 那就是OCP这里的报错不太符合预期
1 个赞
我在40.3和40.188都是过了用 mc 客户端连接的话是可以连上，并获取桶信息的
黑屏备份到本地服务器还没有试过，有参考文档么
[root@hnqy-ob01 ~]# mc ls jw-minio
[2025-11-06 15:14:30 CST] 0B servicecenter-database-backup/
[2025-09-08 09:36:54 CST] 0B settlement/
[root@hnqy-ob01 ~]# mc alias ls
jw-minio
URL : https://k8s-minio-api-jw.crgdpharm.com
AccessKey : *************
SecretKey : ***************
API : s3v4
Path : auto
Src : /root/.mc/config.json
1 个赞
https://www.oceanbase.com/docs/common-oceanbase-database-cn-1000000003465303
https://www.oceanbase.com/docs/common-oceanbase-database-cn-1000000003378784
1 个赞
打卡学习
1 个赞
MinIO 中有创建 oceanbase-backup bucket吗？
1 个赞
另外MinIO 不支持 virtual_hosted_style，应该使用 path_style，黑屏这里加上
&addressing_model=path_style 再试下
ALTER SYSTEM SET LOG_ARCHIVE_DEST='LOCATION=s3://oceanbase-test-bucket/backup/archive?host=obs.****.myhuaweicloud.com&access_id=****&access_key=****&addressing_model=path_style' TENANT = mysql_tenant;
ALTER SYSTEM SET DATA_BACKUP_DEST='s3://oceanbase-test-bucket/backup/data?host=obs.****.myhuaweicloud.com&access_id=****&access_key=****&addressing_model=path_style' TENANT = mysql_tenant;
OCP上点高级配置 选 path_style
1 个赞
蹲一个后续，遇到了类似问题，OCP上调整path_style也没效果
1 个赞
minio是个啥意思
你OCP4.4.0版本的话应该有ocp.backup.object-storage-sdk.invoke.https参数，默认是true，改为false（等5分钟生效），再试试，目前我遇到类似问是这样解决的。 OCP默认开启https协议连s3，但可能对端不是，导致访问被拒绝，也可以F12调试看下response看看报错码
这个9116报错，应该是权限问题，你在复现一遍取下 observer.log和rootservice.log 我看看缺少什么权限，或者你直接使用管理员账号试下
我试了下host前缀如果不加https就是报9116,错误然后observer.log看不到日志输出，
但是如果加上了https会出现 4012 的超时错误，
然后observer.log是可以看到日志的
[2025-12-25 14:14:30.037018] WDIAG Log (ob_storage_s3_base.cpp:116) [55122][T1_L0_G0][T1006][YB420A242803-000646880E74A325-0-0] [lt=7][errcode=-9129] [S3] module=CurlHttpClient, msg=Curl returned error code 60 - Error
[2025-12-25 14:14:30.038360] WDIAG Log (ob_storage_s3_base.cpp:116) [55122][T1_L0_G0][T1006][YB420A242803-000646880E74A325-0-0] [lt=29][errcode=-9129] [S3] module=AWSXmlClient, msg=HTTP response code: -1
[2025-12-25 14:14:30.038624] WDIAG Log (ob_storage_s3_base.cpp:116) [55122][T1_L0_G0][T1006][YB420A242803-000646880E74A325-0-0] [lt=25][errcode=-9129] [S3] module=AWSClient, msg=If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
[2025-12-25 14:14:30.038705] WDIAG log_error (ob_storage_s3_base.h:323) [55122][T1_L0_G0][T1006][YB420A242803-000646880E74A325-0-0] [lt=21][errcode=0] S3 log error(start_time_us_=1766643270011294, timeout_us_=20000000, attempted_retries=0, request_id="", code=-1, exception="", err_msg="curlCode: 60, Error")
[2025-12-25 14:14:30.090555] WDIAG Log (ob_storage_s3_base.cpp:116) [55122][T1_L0_G0][T1006][YB420A242803-000646880E74A325-0-0] [lt=20][errcode=-9129] [S3] module=CurlHttpClient, msg=Curl returned error code 60 - Error
[2025-12-25 14:14:30.091954] WDIAG Log (ob_storage_s3_base.cpp:116) [55122][T1_L0_G0][T1006][YB420A242803-000646880E74A325-0-0] [lt=55][errcode=-9129] [S3] module=AWSXmlClient, msg=HTTP response code: -1
[2025-12-25 14:14:30.092031] WDIAG Log (ob_storage_s3_base.cpp:116) [55122][T1_L0_G0][T1006][YB420A242803-000646880E74A325-0-0] [lt=24][errcode=-9129] [S3] module=AWSClient, msg=If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
[2025-12-25 14:14:30.092103] WDIAG log_error (ob_storage_s3_base.h:323) [55122][T1_L0_G0][T1006][YB420A242803-000646880E74A325-0-0] [lt=13][errcode=0] S3 log error(start_time_us_=1766643270011294, timeout_us_=20000000, attempted_retries=1, request_id="", code=-1, exception="", err_msg="curlCode: 60, Error")
[2025-12-25 14:14:30.195520] WDIAG Log (ob_storage_s3_base.cpp:116) [55122][T1_L0_G0][T1006][YB420A242803-000646880E74A325-0-0] [lt=18][errcode=-9129] [S3] module=CurlHttpClient, msg=Curl returned error code 60 - Error
[2025-12-25 14:14:30.196737] WDIAG Log (ob_storage_s3_base.cpp:116) [55122][T1_L0_G0][T1006][YB420A242803-000646880E74A325-0-0] [lt=35][errcode=-9129] [S3] module=AWSXmlClient, msg=HTTP response code: -1
[2025-12-25 14:14:30.196813] WDIAG Log (ob_storage_s3_base.cpp:116) [55122][T1_L0_G0][T1006][YB420A242803-000646880E74A325-0-0] [lt=27][errcode=-9129] [S3] module=AWSClient, msg=If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
[2025-12-25 14:14:30.196895] WDIAG log_error (ob_storage_s3_base.h:323) [55122][T1_L0_G0][T1006][YB420A242803-000646880E74A325-0-0] [lt=14][errcode=0] S3 log error(start_time_us_=1766643270011294, timeout_us_=20000000, attempted_retries=2, request_id="", code=-1, exception="", err_msg="curlCode: 60, Error")
[2025-12-25 14:14:30.450078] WDIAG Log (ob_storage_s3_base.cpp:116) [55122][T1_L0_G0][T1006][YB420A242803-000646880E74A325-0-0] [lt=24][errcode=-9129] [S3] module=CurlHttpClient, msg=Curl returned error code 60 - Error
[2025-12-25 14:14:30.451523] WDIAG Log (ob_storage_s3_base.cpp:116) [55122][T1_L0_G0][T1006][YB420A242803-000646880E74A325-0-0] [lt=32][errcode=-9129] [S3] module=AWSXmlClient, msg=HTTP response code: -1
[2025-12-25 14:14:30.451603] WDIAG Log (ob_storage_s3_base.cpp:116) [55122][T1_L0_G0][T1006][YB420A242803-000646880E74A325-0-0] [lt=28][errcode=-9129] [S3] module=AWSClient, msg=If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
[2025-12-25 14:14:30.451679] WDIAG log_error (ob_storage_s3_base.h:323) [55122][T1_L0_G0][T1006][YB420A242803-000646880E74A325-0-0] [lt=13][errcode=0] S3 log error(start_time_us_=1766643270011294, timeout_us_=20000000, attempted_retries=3, request_id="", code=-1, exception="", err_msg="curlCode: 60, Error")
[2025-12-25 14:14:31.154170] WDIAG Log (ob_storage_s3_base.cpp:116) [55122][T1_L0_G0][T1006][YB420A242803-000646880E74A325-0-0] [lt=52][errcode=-9129] [S3] module=CurlHttpClient, msg=Curl returned error code 60 - Error
[2025-12-25 14:14:31.155534] WDIAG Log (ob_storage_s3_base.cpp:116) [55122][T1_L0_G0][T1006][YB420A242803-000646880E74A325-0-0] [lt=37][errcode=-9129] [S3] module=AWSXmlClient, msg=HTTP response code: -1
[2025-12-25 14:14:31.155632] WDIAG Log (ob_storage_s3_base.cpp:116) [55122][T1_L0_G0][T1006][YB420A242803-000646880E74A325-0-0] [lt=29][errcode=-9129] [S3] module=AWSClient, msg=If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
[2025-12-25 14:14:31.155713] WDIAG log_error (ob_storage_s3_base.h:323) [55122][T1_L0_G0][T1006][YB420A242803-000646880E74A325-0-0] [lt=16][errcode=0] S3 log error(start_time_us_=1766643270011294, timeout_us_=20000000, attempted_retries=4, request_id="", code=-1, exception="", err_msg="curlCode: 60, Error")
[2025-12-25 14:14:33.206612] WDIAG Log (ob_storage_s3_base.cpp:116) [55122][T1_L0_G0][T1006][YB420A242803-000646880E74A325-0-0] [lt=20][errcode=-9129] [S3] module=CurlHttpClient, msg=Curl returned error code 60 - Error
[2025-12-25 14:14:33.207937] WDIAG Log (ob_storage_s3_base.cpp:116) [55122][T1_L0_G0][T1006][YB420A242803-000646880E74A325-0-0] [lt=33][errcode=-9129] [S3] module=AWSXmlClient, msg=HTTP response code: -1
[2025-12-25 14:14:33.208020] WDIAG Log (ob_storage_s3_base.cpp:116) [55122][T1_L0_G0][T1006][YB420A242803-000646880E74A325-0-0] [lt=30][errcode=-9129] [S3] module=AWSClient, msg=If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
[2025-12-25 14:14:33.208100] WDIAG log_error (ob_storage_s3_base.h:323) [55122][T1_L0_G0][T1006][YB420A242803-000646880E74A325-0-0] [lt=14][errcode=0] S3 log error(start_time_us_=1766643270011294, timeout_us_=20000000, attempted_retries=5, request_id="", code=-1, exception="", err_msg="curlCode: 60, Error")
[2025-12-25 14:14:53.322198] WDIAG get_file_length (ob_backup_io_adapter.cpp:288) [55122][T1_L0_G0][T1006][YB420A242803-000646880E74A325-0-0] [lt=13][errcode=-9129] fail to get file length!(ret=-9129, ret="OB_OBJECT_STORAGE_IO_ERROR", uri=s3://oceanbase-backup/check_file/1006_nfs_connectivity_check_1766643269915604.obbak, storage_info={endpoint:"host=https://k8s-minio-api-jw.crgdpharm.com", access_id:"access_id=*******", extension:"addressing_model=path_style&checksum_type=md5", type:"S3", checksum_type:1, max_iops:0, max_bandwidth:0, role_arn:0xfffbb55eaa38, external_id:0xfffbb55eab38, enable_worm:false}, device_guard={device_handle_:0xffff325b9870, uri_cstr_:s})
[2025-12-25 14:14:53.322228] WDIAG [SERVER] process (ob_rpc_processor_simple.cpp:4715) [55122][T1_L0_G0][T1006][YB420A242803-000646880E74A325-0-0] [lt=24][errcode=-9129] fail to get file length(ret=-9129, arg={tenant_id:1006, backup_dest_str:"s3://oceanbase-backup/check_file/1006_nfs_connectivity_check_1766643269915604.obbak?host=https://k8s-minio-api-jw.crgdpharm.com&access_id=*******_key=*******&addressing_model=path_style&checksum_type=md5", data_checksum:1576598062, file_len:314})
测试过也是不行，而且我也手动把这个内部域名证书导入进系统了，正常的话也能走应该https协议
操作后立刻将最新的observer.log和rootservice.log 压缩发下，需要所有节点的，可能日志归档比较快，报9116会有日志的
2025-12-25 08:45:54.736479] INFO [SHARE] parse_from (ob_backup_config.cpp:426) [307888][T1_L0_G0][T1][YB420A2428BC-0006468813BFF5CF-0-0] [lt=19] parse from(value=s3://oceanbase-backup?host=k8s-minio-api-jw.crgdpharm.com&access_id=xxxx&access_key=xxxx&addressing_model=path_style, backup_dest={root_path:"s3://oceanbase-backup", storage_info:{ObObjectStorageInfo:{endpoint:"host=k8s-minio-api-jw.crgdpharm.com", access_id:"access_id=VbkwvPRKzay3wS90Ac5y", extension:"addressing_model=path_style", type:"S3", checksum_type:0, max_iops:0, max_bandwidth:0, role_arn:0xfff596ddaa38, external_id:0xfff596ddab38, enable_worm:false}, dest_id:-1}}, path=s3://oceanbase-backup?host=k8s-minio-api-jw.crgdpharm.com&access_id=VbkwvPRKzay3wS90Ac5y&access_key=Kbx1lJM8r7hjH3heNWdWWIta1k4rae6I7V4QEimb&addressing_model=path_style, config_items_=[{key:data_backup_dest, value:s3://oceanbase-backup?host=k8s-minio-api-jw.crgdpharm.com&access_id=VbkwvPRKzay3wS90Ac5y&access_key=Kbx1lJM8r7hjH3heNWdWWIta1k4rae6I7V4QEimb&addressing_model=path_style}])
[2025-12-25 08:45:54.738442] WDIAG begin (ob_hashtable.h:963) [307888][T1_L0_G0][T1007][YB420A2428BC-0006468813BFF5CF-0-0] [lt=26][errcode=-4006] hashtable not init, backtrace=0x14b557f4 0x124b9d3c 0x14b52b44 0x14b4f264 0x14b4e624 0x127d3f9c 0x14bac6b0 0x14b95068 0x14b93354 0x146c2044 0x14a48918 0x14a48294 0x14a46da0 0x14a4522c 0x14a4437c 0x1f30f1c8 0x190ed624 0x19167508 0x19166290 0x19176790 0xd4c1640 0xf0b6d44 0x14c4e930 0x146966c0 0x14687094 0xf59c078 0x1f0c78d0 0xfffc348c87c0 0xfffc3480bb9c
[2025-12-25 08:45:54.738625] WDIAG begin (ob_hashtable.h:963) [307888][T1_L0_G0][T1007][YB420A2428BC-0006468813BFF5CF-0-0] [lt=145][errcode=-4006] hashtable not init, backtrace=0x14b557f4 0x124b9d3c 0x14b52b44 0x14b4f28c 0x14b4e624 0x127d3f9c 0x14bac6b0 0x14b95068 0x14b93354 0x146c2044 0x14a48918 0x14a48294 0x14a46da0 0x14a4522c 0x14a4437c 0x1f30f1c8 0x190ed624 0x19167508 0x19166290 0x19176790 0xd4c1640 0xf0b6d44 0x14c4e930 0x146966c0 0x14687094 0xf59c078 0x1f0c78d0 0xfffc348c87c0 0xfffc3480bb9c
[2025-12-25 08:45:54.739148] INFO [STORAGE.TRANS] get_gts (ob_gts_source.cpp:257) [307888][T1_L0_G0][T1007][YB420A2428BC-0006468813BFF5CF-0-0] [lt=13] push gts task success(*task={:""})
[2025-12-25 08:45:54.745261] INFO [SHARE] get_jobs (ob_backup_data_table_operator.cpp:833) [307888][T1_L0_G0][T1][YB420A2428BC-0006468813BFF5CF-0-0] [lt=22] [DATA_BACKUP]success get jobs(job_attrs=[])
[2025-12-25 08:45:54.746469] INFO [STORAGE] check_read_snapshot_for_normal_or_split_dst (ob_tablet_create_delete_helper.cpp:286) [307888][T1_L0_G0][T1007][YB420A2428BC-0006468813BFF5CF-0-0] [lt=22] tablet create transaction is committed, currently in finish transfer in transacation(ret=0, ls_id={id:1}, tablet_id={id:354}, snapshot_version=1766623554738337000, trans_state=3, user_data={tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1766543211904621000, v:0}, create_commit_version:1766543211904621000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, start_transfer_commit_version:-1, start_split_commit_version:-1, start_transfer_commit_scn:{val:18446744073709551615, v:3}})
[2025-12-25 08:45:54.746511] INFO [STORAGE] check_status_for_new_mds (ob_tablet_create_delete_helper.cpp:206) [307888][T1_L0_G0][T1007][YB420A2428BC-0006468813BFF5CF-0-0] [lt=40] refresh tablet status cache(ret=0, ls_id={id:1}, tablet_id={id:354}, tablet_status_cache={tablet_status:{val:1, str:"NORMAL"}, create_commit_version:1766543211904621000, delete_commit_version:-1}, snapshot_version=1766623554738337000)
[2025-12-25 08:45:54.747340] WDIAG begin (ob_hashtable.h:963) [307888][T1_L0_G0][T1007][YB420A2428BC-0006468813BFF5CF-0-0] [lt=13][errcode=-4006] hashtable not init, backtrace=0x14b557f4 0x124b9d3c 0x14b52b44 0x14b4f264 0x14b4e624 0x127d3f9c 0x14bac6b0 0x14b95068 0x14b93354 0x146c2044 0x14a48918 0x14a48294 0x14a46da0 0x14a4522c 0x14a4437c 0x1f30f1c8 0x19075bb8 0x191675f8 0x19166290 0x19176790 0xd4c1640 0xf0b6d44 0x14c4e930 0x146966c0 0x14687094 0xf59c078 0x1f0c78d0 0xfffc348c87c0 0xfffc3480bb9c
[2025-12-25 08:45:54.747396] WDIAG begin (ob_hashtable.h:963) [307888][T1_L0_G0][T1007][YB420A2428BC-0006468813BFF5CF-0-0] [lt=25][errcode=-4006] hashtable not init, backtrace=0x14b557f4 0x124b9d3c 0x14b52b44 0x14b4f28c 0x14b4e624 0x127d3f9c 0x14bac6b0 0x14b95068 0x14b93354 0x146c2044 0x14a48918 0x14a48294 0x14a46da0 0x14a4522c 0x14a4437c 0x1f30f1c8 0x19075bb8 0x191675f8 0x19166290 0x19176790 0xd4c1640 0xf0b6d44 0x14c4e930 0x146966c0 0x14687094 0xf59c078 0x1f0c78d0 0xfffc348c87c0 0xfffc3480bb9c
[2025-12-25 08:45:54.753913] INFO [RS] skip_check_member_list_change_ (ob_ls_recovery_stat_handler.cpp:109) [203946][T1008_TeRec][T1008][YB420A242803-00064687EB055FBD-0-0] [lt=23] is primary tenant, no need check readable_scn
[2025-12-25 08:45:54.754457] INFO [RS] skip_check_member_list_change_ (ob_ls_recovery_stat_handler.cpp:109) [203946][T1008_TeRec][T1008][YB420A242803-00064687EB055FBD-0-0] [lt=21] is primary tenant, no need check readable_scn
[2025-12-25 08:45:54.756467] INFO [RS] skip_check_member_list_change_ (ob_ls_recovery_stat_handler.cpp:109) [203946][T1008_TeRec][T1008][YB420A242803-00064687EB055FBD-0-0] [lt=14] is primary tenant, no need check readable_scn
[2025-12-25 08:45:54.758370] INFO [SHARE] get_backup_dest_status (ob_backup_connectivity.cpp:1402) [307888][T1_L0_G0][T1][YB420A2428BC-0006468813BFF5CF-0-0] [lt=13] path is not exist, it's not cleaning(ret=-4018, tenant_id=1008)
[2025-12-25 08:45:54.758527] INFO [SHARE] check_before_update_inner_config (ob_backup_config.cpp:461) [307888][T1_L0_G0][T1][YB420A2428BC-0006468813BFF5CF-0-0] [lt=82] backup dest is not exist, it's a new backup dest and it is not cleaning(ret=0, tenant_id=1008, config_items=[{key:data_backup_dest, value:s3://oceanbase-backup?host=k8s-minio-api-jw.crgdpharm.com&access_id=VbkwvPRKzay3wS90Ac5y&access_key=Kbx1lJM8r7hjH3heNWdWWIta1k4rae6I7V4QEimb&addressing_model=path_style}])
[2025-12-25 08:45:54.764209] INFO [SHARE] get_server_locality_info_ (ob_backup_connectivity.cpp:2078) [307888][T1_L0_G0][T1][YB420A2428BC-0006468813BFF5CF-0-0] [lt=25] succeed to get region and idc(addr="10.36.40.188:2882", region=deault_region, idc=default_idc, zone=zone1)
[2025-12-25 08:45:54.764241] INFO [SHARE] get_server_locality_info_ (ob_backup_connectivity.cpp:2078) [307888][T1_L0_G0][T1][YB420A2428BC-0006468813BFF5CF-0-0] [lt=30] succeed to get region and idc(addr="10.36.40.3:2882", region=deault_region, idc=default_idc, zone=zone2)
[2025-12-25 08:45:54.831875] WDIAG Log (ob_storage_s3_base.cpp:116) [307888][T1_L0_G0][T1][YB420A2428BC-0006468813BFF5CF-0-0] [lt=6][errcode=-9129] [S3] module=AWSErrorMarshaller, msg=Encountered AWSError 'AccessDenied': Access Denied.
[2025-12-25 08:45:54.831937] WDIAG Log (ob_storage_s3_base.cpp:116) [307888][T1_L0_G0][T1][YB420A2428BC-0006468813BFF5CF-0-0] [lt=31][errcode=-9129] [S3] module=AWSXmlClient, msg=HTTP response code: 403
Resolved remote host IP address: 10.152.181.197
Request ID: 18844EC35D211C4B
Exception name: AccessDenied
Error message: Access Denied.
14 response headers:
accept-ranges : bytes
connection : keep-alive
content-length : 164
content-type : application/xml
date : Thu, 25 Dec 2025 00:45:54 GMT
location : https://k8s-minio-api-jw.crgdpharm.com/oceanbase-backup?max-keys=1000
strict-transport-security : max-age=15724800; includeSubDomains
vary : Origin
x-amz-id-2 : 83564099173bbbe56aa9a19b41e8124fecfa2e9875abc9daf7e6b05709a9c9a8
x-amz-request-id : 18844EC35D211C4B
x-content-type-options : nosniff
x-ratelimit-limit : 37498
x-ratelimit-remaining : 37498
x-xss-protection : 1; mode=block
[2025-12-25 08:45:54.832050] WDIAG Log (ob_storage_s3_base.cpp:116) [307888][T1_L0_G0][T1][YB420A2428BC-0006468813BFF5CF-0-0] [lt=6][errcode=-9129] [S3] module=AWSClient, msg=If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
[2025-12-25 08:45:54.832228] WDIAG log_s3_status (ob_storage_s3_base.cpp:841) [307888][T1_L0_G0][T1][YB420A2428BC-0006468813BFF5CF-0-0] [lt=40][errcode=-9116] S3 info(request_id="18844EC35D211C4B", code=403, exception="AccessDenied", err_msg="Access Denied.")
[2025-12-25 08:45:54.832251] WDIAG do_list_ (ob_storage_s3_base.cpp:1308) [307888][T1_L0_G0][T1][YB420A2428BC-0006468813BFF5CF-0-0] [lt=21][errcode=-9116] failed to list s3 objects(ret=-9116, bucket=oceanbase-backup, object=, max_list_num=1000, delimiter=NULL)
[2025-12-25 08:45:54.832262] WDIAG list_files_ (ob_storage_s3_base.cpp:1877) [307888][T1_L0_G0][T1][YB420A2428BC-0006468813BFF5CF-0-0] [lt=9][errcode=-9116] fail to list s3 objects(ret=-9116, uri=s3://oceanbase-backup/)
[2025-12-25 08:45:54.832286] WDIAG [STORAGE] list_files (ob_storage.cpp:1040) [307888][T1_L0_G0][T1][YB420A2428BC-0006468813BFF5CF-0-0] [lt=13][errcode=-9116] fail to list files(ret=-9116, uri=s3://oceanbase-backup)
[2025-12-25 08:45:54.832293] WDIAG [STORAGE] print_access_storage_log_ (ob_i_storage.cpp:936) [307888][T1_L0_G0][T1][YB420A2428BC-0006468813BFF5CF-0-0] [lt=7][errcode=-9116] access object storage cost too much time: list_files (ob_storage.cpp:1023), uri=s3://oceanbase-backup, size=0 byte, start_time=1766623554764275, cost_ts=68017 us, speed=0.00 MB/s, is_slow=1
[2025-12-25 08:45:54.832308] WDIAG inner_scan_dir_ (ob_object_device.cpp:799) [307888][T1_L0_G0][T1][YB420A2428BC-0006468813BFF5CF-0-0] [lt=8][errcode=-9116] fail to do list/dir scan!(ret=-9116, dir_name=s3://oceanbase-backup, is_dir_scan=false, is_marker_scan=false)
[2025-12-25 08:45:54.832327] WDIAG is_empty_directory (ob_backup_io_adapter.cpp:1371) [307888][T1_L0_G0][T1][YB420A2428BC-0006468813BFF5CF-0-0] [lt=17][errcode=-9116] fail to scan dir!(ret=-9116, ret="OB_OBJECT_STORAGE_PERMISSION_DENIED", uri=s3://oceanbase-backup, storage_info={endpoint:"host=k8s-minio-api-jw.crgdpharm.com", access_id:"access_id=xxxx", extension:"addressing_model=path_style&checksum_type=md5", type:"S3", checksum_type:1, max_iops:0, max_bandwidth:0, role_arn:0xfff596dcea38, external_id:0xfff596dceb38, enable_worm:false}, device_guard={device_handle_:0xfffc209b9820, uri_cstr_:s})
[2025-12-25 08:45:54.832352] WDIAG [SHARE] dest_is_empty_directory (ob_backup_store.cpp:274) [307888][T1_L0_G0][T1][YB420A2428BC-0006468813BFF5CF-0-0] [lt=20][errcode=-9116] fail to init store(ret=-9116, backup_dest={root_path:"s3://oceanbase-backup", storage_info:{ObObjectStorageInfo:{endpoint:"host=k8s-minio-api-jw.crgdpharm.com", access_id:"access_id=VbkwvPRKzay3wS90Ac5y", extension:"addressing_model=path_style&checksum_type=md5", type:"S3", checksum_type:1, max_iops:0, max_bandwidth:0, role_arn:0xfff596dcea38, external_id:0xfff596dceb38, enable_worm:false}, dest_id:-1}})
[2025-12-25 08:45:54.832368] WDIAG [SHARE] check_dest_validity (ob_backup_store.cpp:521) [307888][T1_L0_G0][T1][YB420A2428BC-0006468813BFF5CF-0-0] [lt=15][errcode=-9116] fail to check dest is empty dirctory(ret=-9116, backup_dest={root_path:"s3://oceanbase-backup", storage_info:{ObObjectStorageInfo:{endpoint:"host=k8s-minio-api-jw.crgdpharm.com", access_id:"access_id=xxxx", extension:"addressing_model=path_style&checksum_type=md5", type:"S3", checksum_type:1, max_iops:0, max_bandwidth:0, role_arn:0xfff596de2a38, external_id:0xfff596de2b38, enable_worm:false}, dest_id:-1}})
[2025-12-25 08:45:54.832386] WDIAG [SHARE] check_before_update_inner_config (ob_backup_config.cpp:494) [307888][T1_L0_G0][T1][YB420A2428BC-0006468813BFF5CF-0-0] [lt=16][errcode=-9116] fail to check dest validity(ret=-9116, tenant_id=1008, backup_dest=s3://oceanbase-backup?host=k8s-minio-api-jw.crgdpharm.com&access_id=VbkwvPRKzay3wS90Ac5y&access_key=Kbx1lJM8r7hjH3heNWdWWIta1k4rae6I7V4QEimb&addressing_model=path_style&checksum_type=md5)
[2025-12-25 08:45:54.832398] WDIAG [SHARE] update_inner_config_table (ob_backup_config.cpp:374) [307888][T1_L0_G0][T1][YB420A2428BC-0006468813BFF5CF-0-0] [lt=10][errcode=-9116] fail to check before update inner config(ret=-9116)
[2025-12-25 08:45:54.832405] WDIAG [SHARE] admin_set_backup_config (ob_backup_config.cpp:1311) [307888][T1_L0_G0][T1][YB420A2428BC-0006468813BFF5CF-0-0] [lt=7][errcode=-9116] fail to update inner config table(ret=-9116)
[2025-12-25 08:45:54.832855] INFO [STORAGE.TRANS] rollback_tx (ob_tx_api.cpp:406) [307888][T1_L0_G0][T1007][YB420A2428BC-0006468813BFF5CF-0-0] [lt=6] rollback tx(ret=0, *this={is_inited_:true, tenant_id_:1007, this:0xfff517a04030}, tx={this:0xfffb56bbaa90, tx_id:{txid:952847}, state:7, addr:"10.36.40.3:2882", tenant_id:1007, session_id:3221793571, assoc_session_id:3221793571, client_sid:3221793571, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:{branch:0, seq:1}, op_sn:6, alloc_ts:1766623554668907, active_ts:1766623554668907, commit_ts:-1, finish_ts:1766623554831869, timeout_us:9996656, lock_timeout_us:-1, expire_ts:1766623564665563, coord_id:{id:-1}, parts:[{id:{id:1}, addr:"10.36.40.188:2882", epoch:368075815503810, first_scn:{branch:0, seq:7006}, last_scn:{branch:0, seq:58913}, last_touch_ts:1766623554675430, flag_.flag_bit_:{is_dup_ls:0, is_clean:false}}], exec_info_reap_ts:1766623554675429, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1765268698, cluster_version:17180131584, seq_base:1766623554666697, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:-6002, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:1766623554746974000, v:0}, ref:2})
[2025-12-25 08:45:54.832937] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:328) [307888][T1_L0_G0][T1007][YB420A2428BC-0006468813BFF5CF-0-0] [lt=57][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS")
[2025-12-25 08:45:54.833071] WDIAG [RS] admin_set_config (ob_root_service.cpp:8938) [307888][T1_L0_G0][T1][YB420A2428BC-0006468813BFF5CF-0-0] [lt=16][errcode=-9116] fail to set backup config(ret=-9116, arg={items:[{name:"data_backup_dest", value:"s3://oceanbase-backup?host=k8s-minio-api-jw.crgdpharm.com&access_id=xxxx&access_key=xxxx&addressing_model=path_style", comment:"", zone:"", server:"0.0.0.0:0", tenant_name:"test", exec_tenant_id:1, tenant_ids:[], want_to_set_tenant_config:false}], is_inner:false})
[2025-12-25 08:45:54.833417] INFO [SHARE] add_event (ob_event_history_table_operator.h:271) [307888][T1_L0_G0][T1][YB420A2428BC-0006468813BFF5CF-0-0] [lt=217] event table add task(ret=0, event_table_name="__all_rootservice_event_history", sql=INSERT INTO __all_rootservice_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, value4, value5, value6, rs_svr_ip, rs_svr_port) VALUES (usec_to_time(1766623554833289), 'root_service', 'admin_set_config', 'ret', -9116, 'arg', '{name:"data_backup_dest", value:"s3://oceanbase-backup?host=k8s-minio-api-jw.crgdpharm.com&access_id=xxxx&access_key=xxxx&addressing_model=path_style", comment:"", zone:"", server:"0.0.0.0:0", tenant_name:"test", exec_tenant_id:1, tenant_ids:[], want_to_set_tenant_config:false}', 'is_inner', 0, '', '', '', '10.36.40.3', 2882))
[2025-12-25 08:45:54.833435] WDIAG [RS] process_ (ob_rs_rpc_processor.h:339) [307888][T1_L0_G0][T1][YB420A2428BC-0006468813BFF5CF-0-0] [lt=18][errcode=-9116] process failed(ret=-9116)