Binlog服务运行一段时间就没有继续采集日志了

AntTech_4D0BKE · 2024 年11 月 26 日 17:13

Binlog创建好一段时间是有采集日志的，过一段时间就没了

淇铭 · 2024 年11 月 26 日 17:14

你发一下ob的版本和binlog的版本日志信息发一下

AntTech_4D0BKE · 2024 年11 月 26 日 17:15

用的oceanbase-all-in-one-4.2.1_bp10_20241122.el7.x86_64.tar.gz安装部署的，日志没发现有什么问题

淇铭 · 2024 年11 月 26 日 17:16

binlog的版本呢

AntTech_4D0BKE · 2024 年11 月 26 日 17:18

obbinlog-ce-4.0.1-1.el7.x86_64.rpm这个包

淇铭 · 2024 年11 月 26 日 17:31

查看一下下面的信息
SHOW BINLOG SERVER FOR TENANT cluster.tenant;
SHOW BINLOG INSTANCE instance_name;

AntTech_4D0BKE · 2024 年11 月 26 日 18:11

目前看到了一些错误日志，正好也是时间点发生的
[2024-11-26 17:07:55.472643] ERROR issue_dba_error (ob_log.cpp:1875) [3787520][][T0][YCA230A1A2DBC-0000000001B01705-0-0] [lt=15][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file=“ob_cdc_tablet_to_table_info.cpp”, line_no=136, info=“ObBatchCreateTabletArg is invalid”)
[2024-11-26 17:07:55.472677] EDIAG [TLOG.PARSER] parse_create_tablet_op_ (ob_cdc_tablet_to_table_info.cpp:136) [3787520][][T0][YCA230A1A2DBC-0000000001B01705-0-0] [lt=34][errcode=-4016] ObBatchCreateTabletArg is invalid(ret=-4016, ret=“OB_ERR_UNEXPECTED”, tls_id={tenant_id:1004, ls_id:{id:1001}}, create_tablet_arg=id:{id:1001}, major_frozen_scn:{val:1, v:0}, need_check_tablet_cnt:true, is_old_mds:false, tablets:[tablet_ids:[{id:200009}], data_tablet_id:{id:200009}, table_schema_index:[0], compat_mode:0, is_create_bind_hidden_tablets:false]) BACKTRACE:0x17f12620 0x96909b5 0x9597979 0x95974af 0x95973f8 0x9597221 0xbce5b1f 0xbce4eff 0xbce4237 0xbfd44bb 0xbfd40fa 0xc150ed5 0xc150995 0xc02712d 0xc0264fc 0xc0e0ff9 0x7fd5e52bbea5 0x7fd5e29a996d
[2024-11-26 17:07:55.473596] ERROR issue_dba_error (ob_log.cpp:1875) [3787520][][T0][YCA230A1A2DBC-0000000001B01705-0-0] [lt=899][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file=“ob_cdc_tablet_to_table_info.cpp”, line_no=98, info=“parse_create_tablet_op_ failed”)
[2024-11-26 17:07:55.473617] EDIAG [TLOG.PARSER] parse_from_multi_data_source_buf (ob_cdc_tablet_to_table_info.cpp:98) [3787520][][T0][YCA230A1A2DBC-0000000001B01705-0-0] [lt=20][errcode=-4016] parse_create_tablet_op_ failed(ret=-4016, ret=“OB_ERR_UNEXPECTED”, tls_id={tenant_id:1004, ls_id:{id:1001}}, multi_data_source_node={register_no_:0, has_submitted_:false, has_synced_:false, type:3, data_.length():229}, create_tablet_arg=id:{id:1001}, major_frozen_scn:{val:1, v:0}, need_check_tablet_cnt:true, is_old_mds:false, tablets:[tablet_ids:[{id:200009}], data_tablet_id:{id:200009}, table_schema_index:[0], compat_mode:0, is_create_bind_hidden_tablets:false], this={cmd:1, create_tablet_cnt:0, delete_tablet_cnt:0}) BACKTRACE:0x17f12620 0x96909b5 0x9825eba 0xbd689af 0x9797ff2 0x95a97f7 0xbce50cb 0xbce445c 0xbfd44bb 0xbfd40fa 0xc150ed5 0xc150995 0xc02712d 0xc0264fc 0xc0e0ff9 0x7fd5e52bbea5 0x7fd5e29a996d
[2024-11-26 17:07:55.473780] ERROR issue_dba_error (ob_log.cpp:1875) [3787520][][T0][YCA230A1A2DBC-0000000001B01705-0-0] [lt=150][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file=“ob_log_part_trans_task.cpp”, line_no=3402, info=“parse_from_multi_data_source_buf to tablet_change_info failed”)
[2024-11-26 17:07:55.473789] EDIAG [TLOG] parse_tablet_change_mds_ (ob_log_part_trans_task.cpp:3402) [3787520][][T0][YCA230A1A2DBC-0000000001B01705-0-0] [lt=9][errcode=-4016] parse_from_multi_data_source_buf to tablet_change_info failed(ret=-4016, ret=“OB_ERR_UNEXPECTED”, tls_id={tenant_id:1004, ls_id:{id:1001}}, trans_id={txid:20727}, multi_data_source_node={lsn:{lsn:44977643}, tx_buf_node:{register_no_:0, has_submitted_:false, has_synced_:false, type:3, data_.length():229}}, tx_buf_node={register_no_:0, has_submitted_:false, has_synced_:false, type:3, data_.length():229}) BACKTRACE:0x17f12620 0x96909b5 0x9825eba 0xbd689af 0x9797ff2 0x95a97f7 0xbfde07e 0xbfd4687 0xbfd40fa 0xc150ed5 0xc150995 0xc02712d 0xc0264fc 0xc0e0ff9 0x7fd5e52bbea5 0x7fd5e29a996d
[2024-11-26 17:07:55.473839] ERROR issue_dba_error (ob_log.cpp:1875) [3787520][][T0][YCA230A1A2DBC-0000000001B01705-0-0] [lt=48][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file=“ob_log_part_trans_task.cpp”, line_no=2598, info=“parse_tablet_change_mds_ failed”)
[2024-11-26 17:07:55.473846] EDIAG [TLOG] parse_multi_data_source_data (ob_log_part_trans_task.cpp:2598) [3787520][][T0][YCA230A1A2DBC-0000000001B01705-0-0] [lt=6][errcode=-4016] parse_tablet_change_mds_ failed(ret=-4016, ret=“OB_ERR_UNEXPECTED”, tls_id={tenant_id:1004, ls_id:{id:1001}}, trans_id={txid:20727}, mds_node={lsn:{lsn:44977643}, tx_buf_node:{register_no_:0, has_submitted_:false, has_synced_:false, type:3, data_.length():229}}, idx=0, this={state:1, type:“DML_TRANS”, type:1, cluster_id:1732607596, exec_tenant_id:0, tls_id:{tenant_id:1004, ls_id:{id:1001}}, trans_id:{txid:20727}, trans_type:“DIST”, is_xa_or_dup:false, is_trans_committed:true, trans_commit_version:1732612075430591290, prepare_ts:1732612075428000745, prepare_log_lsn:{lsn:44978041}, commit_ts:1732612075430591290, commit_log_lsn:{lsn:44978252}, participant_count:2, participants:[{id_:{id:1}, offset_:{lsn:18621066}}, {id_:{id:1001}, offset_:{lsn:44978041}}], trace_id:"", trace_info:"", sorted_log_entry_info:{fetched_log_entry_count:1, recorded_lsn_count:1, fetched_log_entry_arr:{sorted_treeify_list:{count:1}, is_tree_mode:false, auto_treeify_mode:true, is_tree_empty:true, list:{sorted_linked_list:{count:1}, is_list_empty:false, l_head:{val:{lsn:{lsn:44977643}, next:null}}, l_tail:{val:{lsn:{lsn:44977643}, next:null}}, l_tail:{val:{lsn:{lsn:44977643}, next:null}}, is_unique:true}, auto_treeify_threshold:32, auto_untreeify_threshold:30}, recorded_lsn_arr:[{lsn:44977643}]}, sorted_redo_list:{ready_node_num:0, redo_sorted_progress:{dispatched_redo_count:0, sorted_redo_count:0, sorted_row_seq_no:0}, cur_dispatch_redo:{list:{sorted_treeify_list:{count:0}, is_tree_mode:false, auto_treeify_mode:true, is_tree_empty:true, list:{sorted_linked_list:{count:0}, is_list_empty:true, l_head:null, l_tail:null, l_tail:NULL, is_unique:true}, auto_treeify_threshold:32, auto_untreeify_threshold:30}, node_ptr:NULL}, cur_sort_redo:{list:{sorted_treeify_list:{count:0}, is_tree_mode:false, auto_treeify_mode:true, is_tree_empty:true, list:{sorted_linked_list:{count:0}, is_list_empty:true, l_head:null, l_tail:null, l_tail:NULL, is_unique:true}, auto_treeify_threshold:32, auto_untreeify_threshold:30}, node_ptr:NULL}, cur_sort_stmt:null, is_dml_stmt_iter_end:false}, rollback_list:{head:null, tail:null, num:0}, part_tx_fetch_state:6, ref_cnt:1, multi_data_source_node_arr:[{lsn:{lsn:44977643}, tx_buf_node:{register_no_:0, has_submitted_:false, has_synced_:false, type:3, data_.length():229}}], multi_data_source_info:has_ls_table_op: false, is_ddl_trans: 0, tablet_change_info: None}, checkpoint_seq:17680, global_trans_seq:0, global_schema_version:-1, local_schema_version:-1, stmt_list:{head:null, tail:null, num:0}, is_data_ready:true, wait_formatted_cond:null, output_br_count_by_turn:0, next_task:null}) BACKTRACE:0x17f12620 0x96909b5 0x9835e34 0x9835b76 0xbce026e 0x961579a 0xbfd48b5 0xbfd4236 0xc150ed5 0xc150995 0xc02712d 0xc0264fc 0xc0e0ff9 0x7fd5e52bbea5 0x7fd5e29a996d
[2024-11-26 17:07:55.473986] ERROR issue_dba_error (ob_log.cpp:1875) [3787520][][T0][YCA230A1A2DBC-0000000001B01705-0-0] [lt=137][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file=“ob_log_trans_ctx.cpp”, line_no=504, info=“parse_multi_data_source_data failed”)
[2024-11-26 17:07:55.473994] EDIAG [TLOG] add_ready_participant_ (ob_log_trans_ctx.cpp:504) [3787520][][T0][YCA230A1A2DBC-0000000001B01705-0-0] [lt=7][errcode=-4016] parse_multi_data_source_data failed(ret=-4016, ret=“OB_ERR_UNEXPECTED”, part_trans_task={state:1, type:“DML_TRANS”, type:1, cluster_id:1732607596, exec_tenant_id:0, tls_id:{tenant_id:1004, ls_id:{id:1001}}, trans_id:{txid:20727}, trans_type:“DIST”, is_xa_or_dup:false, is_trans_committed:true, trans_commit_version:1732612075430591290, prepare_ts:1732612075428000745, prepare_log_lsn:{lsn:44978041}, commit_ts:1732612075430591290, commit_log_lsn:{lsn:44978252}, participant_count:2, participants:[{id_:{id:1}, offset_:{lsn:18621066}}, {id_:{id:1001}, offset_:{lsn:44978041}}], trace_id:"", trace_info:"", sorted_log_entry_info:{fetched_log_entry_count:1, recorded_lsn_count:1, fetched_log_entry_arr:{sorted_treeify_list:{count:1}, is_tree_mode:false, auto_treeify_mode:true, is_tree_empty:true, list:{sorted_linked_list:{count:1}, is_list_empty:false, l_head:{val:{lsn:{lsn:44977643}, next:null}}, l_tail:{val:{lsn:{lsn:44977643}, next:null}}, l_tail:{val:{lsn:{lsn:44977643}, next:null}}, is_unique:true}, auto_treeify_threshold:32, auto_untreeify_threshold:30}, recorded_lsn_arr:[{lsn:44977643}]}, sorted_redo_list:{ready_node_num:0, redo_sorted_progress:{dispatched_redo_count:0, sorted_redo_count:0, sorted_row_seq_no:0}, cur_dispatch_redo:{list:{sorted_treeify_list:{count:0}, is_tree_mode:false, auto_treeify_mode:true, is_tree_empty:true, list:{sorted_linked_list:{count:0}, is_list_empty:true, l_head:null, l_tail:null, l_tail:NULL, is_unique:true}, auto_treeify_threshold:32, auto_untreeify_threshold:30}, node_ptr:NULL}, cur_sort_redo:{list:{sorted_treeify_list:{count:0}, is_tree_mode:false, auto_treeify_mode:true, is_tree_empty:true, list:{sorted_linked_list:{count:0}, is_list_empty:true, l_head:null, l_tail:null, l_tail:NULL, is_unique:true}, auto_treeify_threshold:32, auto_untreeify_threshold:30}, node_ptr:NULL}, cur_sort_stmt:null, is_dml_stmt_iter_end:false}, rollback_list:{head:null, tail:null, num:0}, part_tx_fetch_state:6, ref_cnt:1, multi_data_source_node_arr:[{lsn:{lsn:44977643}, tx_buf_node:{register_no_:0, has_submitted_:false, has_synced_:false, type:3, data_.length():229}}], multi_data_source_info:has_ls_table_op: false, is_ddl_trans: 0, tablet_change_info: None}, checkpoint_seq:17680, global_trans_seq:0, global_schema_version:-1, local_schema_version:-1, stmt_list:{head:null, tail:null, num:0}, is_data_ready:true, wait_formatted_cond:null, output_br_count_by_turn:0, next_task:null}) BACKTRACE:0x17f12620 0x96909b5 0x97cf108 0x97cec1b 0x97ceb5e 0x97ce986 0xc153082 0xc151041 0xc150995 0xc02712d 0xc0264fc 0xc0e0ff9 0x7fd5e52bbea5 0x7fd5e29a996d
[2024-11-26 17:07:55.474136] ERROR issue_dba_error (ob_log.cpp:1875) [3787520][][T0][YCA230A1A2DBC-0000000001B01705-0-0] [lt=138][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file=“ob_log_trans_ctx.cpp”, line_no=286, info=“add_ready_participant_ fail”)
[2024-11-26 17:07:55.474155] EDIAG [TLOG] add_participant (ob_log_trans_ctx.cpp:286) [3787520][][T0][YCA230A1A2DBC-0000000001B01705-0-0] [lt=17][errcode=-4016] add_ready_participant_ fail(ret=-4016, ret=“OB_ERR_UNEXPECTED”, part_trans_task={state:1, type:“DML_TRANS”, type:1, cluster_id:1732607596, exec_tenant_id:0, tls_id:{tenant_id:1004, ls_id:{id:1001}}, trans_id:{txid:20727}, trans_type:“DIST”, is_xa_or_dup:false, is_trans_committed:true, trans_commit_version:1732612075430591290, prepare_ts:1732612075428000745, prepare_log_lsn:{lsn:44978041}, commit_ts:1732612075430591290, commit_log_lsn:{lsn:44978252}, participant_count:2, participants:[{id_:{id:1}, offset_:{lsn:18621066}}, {id_:{id:1001}, offset_:{lsn:44978041}}], trace_id:"", trace_info:"", sorted_log_entry_info:{fetched_log_entry_count:1, recorded_lsn_count:1, fetched_log_entry_arr:{sorted_treeify_list:{count:1}, is_tree_mode:false, auto_treeify_mode:true, is_tree_empty:true, list:{sorted_linked_list:{count:1}, is_list_empty:false, l_head:{val:{lsn:{lsn:44977643}, next:null}}, l_tail:{val:{lsn:{lsn:44977643}, next:null}}, l_tail:{val:{lsn:{lsn:44977643}, next:null}}, is_unique:true}, auto_treeify_threshold:32, auto_untreeify_threshold:30}, recorded_lsn_arr:[{lsn:44977643}]}, sorted_redo_list:{ready_node_num:0, redo_sorted_progress:{dispatched_redo_count:0, sorted_redo_count:0, sorted_row_seq_no:0}, cur_dispatch_redo:{list:{sorted_treeify_list:{count:0}, is_tree_mode:false, auto_treeify_mode:true, is_tree_empty:true, list:{sorted_linked_list:{count:0}, is_list_empty:true, l_head:null, l_tail:null, l_tail:NULL, is_unique:true}, auto_treeify_threshold:32, auto_untreeify_threshold:30}, node_ptr:NULL}, cur_sort_redo:{list:{sorted_treeify_list:{count:0}, is_tree_mode:false, auto_treeify_mode:true, is_tree_empty:true, list:{sorted_linked_list:{count:0}, is_list_empty:true, l_head:null, l_tail:null, l_tail:NULL, is_unique:true}, auto_treeify_threshold:32, auto_untreeify_threshold:30}, node_ptr:NULL}, cur_sort_stmt:null, is_dml_stmt_iter_end:false}, rollback_list:{head:null, tail:null, num:0}, part_tx_fetch_state:6, ref_cnt:1, multi_data_source_node_arr:[{lsn:{lsn:44977643}, tx_buf_node:{register_no_:0, has_submitted_:false, has_synced_:false, type:3, data_.length():229}}], multi_data_source_info:has_ls_table_op: false, is_ddl_trans: 0, tablet_change_info: None}, checkpoint_seq:17680, global_trans_seq:0, global_schema_version:-1, local_schema_version:-1, stmt_list:{head:null, tail:null, num:0}, is_data_ready:true, wait_formatted_cond:null, output_br_count_by_turn:0, next_task:null}, is_part_trans_served=true, is_all_participants_ready=false) BACKTRACE:0x17f12620 0x96909b5 0x9614ac5 0x961467f 0x96145c2 0x961444c 0xc151299 0xc1509bd 0xc02712d 0xc0264fc 0xc0e0ff9 0x7fd5e52bbea5 0x7fd5e29a996d
[2024-11-26 17:07:55.474271] ERROR issue_dba_error (ob_log.cpp:1875) [3787520][][T0][YCA230A1A2DBC-0000000001B01705-0-0] [lt=112][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file=“ob_log_sequencer1.cpp”, line_no=605, info=“add participant fail”)
[2024-11-26 17:07:55.474289] EDIAG [TLOG.SEQUENCER] handle_part_trans_task_ (ob_log_sequencer1.cpp:605) [3787520][][T0][YCA230A1A2DBC-0000000001B01705-0-0] [lt=17][errcode=-4016] add participant fail(ret=-4016, ret=“OB_ERR_UNEXPECTED”, part_trans_task={state:1, type:“DML_TRANS”, type:1, cluster_id:1732607596, exec_tenant_id:0, tls_id:{tenant_id:1004, ls_id:{id:1001}}, trans_id:{txid:20727}, trans_type:“DIST”, is_xa_or_dup:false, is_trans_committed:true, trans_commit_version:1732612075430591290, prepare_ts:1732612075428000745, prepare_log_lsn:{lsn:44978041}, commit_ts:1732612075430591290, commit_log_lsn:{lsn:44978252}, participant_count:2, participants:[{id_:{id:1}, offset_:{lsn:18621066}}, {id_:{id:1001}, offset_:{lsn:44978041}}], trace_id:"", trace_info:"", sorted_log_entry_info:{fetched_log_entry_count:1, recorded_lsn_count:1, fetched_log_entry_arr:{sorted_treeify_list:{count:1}, is_tree_mode:false, auto_treeify_mode:true, is_tree_empty:true, list:{sorted_linked_list:{count:1}, is_list_empty:false, l_head:{val:{lsn:{lsn:44977643}, next:null}}, l_tail:{val:{lsn:{lsn:44977643}, next:null}}, l_tail:{val:{lsn:{lsn:44977643}, next:null}}, is_unique:true}, auto_treeify_threshold:32, auto_untreeify_threshold:30}, recorded_lsn_arr:[{lsn:44977643}]}, sorted_redo_list:{ready_node_num:0, redo_sorted_progress:{dispatched_redo_count:0, sorted_redo_count:0, sorted_row_seq_no:0}, cur_dispatch_redo:{list:{sorted_treeify_list:{count:0}, is_tree_mode:false, auto_treeify_mode:true, is_tree_empty:true, list:{sorted_linked_list:{count:0}, is_list_empty:true, l_head:null, l_tail:null, l_tail:NULL, is_unique:true}, auto_treeify_threshold:32, auto_untreeify_threshold:30}, node_ptr:NULL}, cur_sort_redo:{list:{sorted_treeify_list:{count:0}, is_tree_mode:false, auto_treeify_mode:true, is_tree_empty:true, list:{sorted_linked_list:{count:0}, is_list_empty:true, l_head:null, l_tail:null, l_tail:NULL, is_unique:true}, auto_treeify_threshold:32, auto_untreeify_threshold:30}, node_ptr:NULL}, cur_sort_stmt:null, is_dml_stmt_iter_end:false}, rollback_list:{head:null, tail:null, num:0}, part_tx_fetch_state:6, ref_cnt:1, multi_data_source_node_arr:[{lsn:{lsn:44977643}, tx_buf_node:{register_no_:0, has_submitted_:false, has_synced_:false, type:3, data_.length():229}}], multi_data_source_info:has_ls_table_op: false, is_ddl_trans: 0, tablet_change_info: None}, checkpoint_seq:17680, global_trans_seq:0, global_schema_version:-1, local_schema_version:-1, stmt_list:{head:null, tail:null, num:0}, is_data_ready:true, wait_formatted_cond:null, output_br_count_by_turn:0, next_task:null}, *trans_ctx={state:“PREPARED”, trx_sort_elem:{trans_id:{txid:20727}, trans_commit_version:1732612075430591290}, seq:0, participants:{obj:NULL, tls_id:{tenant_id:1004, ls_id:{id:1}}}, participant_count:2, ready_participant_objs:null, ready_participant_count:0, total_br_count:0, committed_br_count:0, revertable_participant_count:0, is_trans_redo_dispatched:false, is_trans_sorted:false}) BACKTRACE:0x17f12620 0x96909b5 0x9597979 0x95974af 0x95973f8 0x9597221 0xc0285fa 0xc02729c 0xc0264fc 0xc0e0ff9 0x7fd5e52bbea5 0x7fd5e29a996d
[2024-11-26 17:07:55.474384] ERROR issue_dba_error (ob_log.cpp:1875) [3787520][][T0][YCA230A1A2DBC-0000000001B01705-0-0] [lt=90][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file=“ob_log_sequencer1.cpp”, line_no=505, info=“handle_part_trans_task_ fail”)
[2024-11-26 17:07:55.474397] EDIAG [TLOG.SEQUENCER] handle (ob_log_sequencer1.cpp:505) [3787520][][T0][YCA230A1A2DBC-0000000001B01705-0-0] [lt=12][errcode=-4016] handle_part_trans_task_ fail(ret=-4016, ret=“OB_ERR_UNEXPECTED”, thread_index=0, part_trans_task={state:1, type:“DML_TRANS”, type:1, cluster_id:1732607596, exec_tenant_id:0, tls_id:{tenant_id:1004, ls_id:{id:1001}}, trans_id:{txid:20727}, trans_type:“DIST”, is_xa_or_dup:false, is_trans_committed:true, trans_commit_version:1732612075430591290, prepare_ts:1732612075428000745, prepare_log_lsn:{lsn:44978041}, commit_ts:1732612075430591290, commit_log_lsn:{lsn:44978252}, participant_count:2, participants:[{id_:{id:1}, offset_:{lsn:18621066}}, {id_:{id:1001}, offset_:{lsn:44978041}}], trace_id:"", trace_info:"", sorted_log_entry_info:{fetched_log_entry_count:1, recorded_lsn_count:1, fetched_log_entry_arr:{sorted_treeify_list:{count:1}, is_tree_mode:false, auto_treeify_mode:true, is_tree_empty:true, list:{sorted_linked_list:{count:1}, is_list_empty:false, l_head:{val:{lsn:{lsn:44977643}, next:null}}, l_tail:{val:{lsn:{lsn:44977643}, next:null}}, l_tail:{val:{lsn:{lsn:44977643}, next:null}}, is_unique:true}, auto_treeify_threshold:32, auto_untreeify_threshold:30}, recorded_lsn_arr:[{lsn:44977643}]}, sorted_redo_list:{ready_node_num:0, redo_sorted_progress:{dispatched_redo_count:0, sorted_redo_count:0, sorted_row_seq_no:0}, cur_dispatch_redo:{list:{sorted_treeify_list:{count:0}, is_tree_mode:false, auto_treeify_mode:true, is_tree_empty:true, list:{sorted_linked_list:{count:0}, is_list_empty:true, l_head:null, l_tail:null, l_tail:NULL, is_unique:true}, auto_treeify_threshold:32, auto_untreeify_threshold:30}, node_ptr:NULL}, cur_sort_redo:{list:{sorted_treeify_list:{count:0}, is_tree_mode:false, auto_treeify_mode:true, is_tree_empty:true, list:{sorted_linked_list:{count:0}, is_list_empty:true, l_head:null, l_tail:null, l_tail:NULL, is_unique:true}, auto_treeify_threshold:32, auto_untreeify_threshold:30}, node_ptr:NULL}, cur_sort_stmt:null, is_dml_stmt_iter_end:false}, rollback_list:{head:null, tail:null, num:0}, part_tx_fetch_state:6, ref_cnt:1, multi_data_source_node_arr:[{lsn:{lsn:44977643}, tx_buf_node:{register_no_:0, has_submitted_:false, has_synced_:false, type:3, data_.length():229}}], multi_data_source_info:has_ls_table_op: false, is_ddl_trans: 0, tablet_change_info: None}, checkpoint_seq:17680, global_trans_seq:0, global_schema_version:-1, local_schema_version:-1, stmt_list:{head:null, tail:null, num:0}, is_data_ready:true, wait_formatted_cond:null, output_br_count_by_turn:0, next_task:null}) BACKTRACE:0x17f12620 0x96909b5 0x9597979 0x95974af 0x95973f8 0x9597221 0xc0273ed 0xc026533 0xc0e0ff9 0x7fd5e52bbea5 0x7fd5e29a996d
[2024-11-26 17:07:55.474468] INFO [TLOG] handle_error (ob_log_instance.cpp:1998) [3787520][][T0][YCA230A1A2DBC-0000000001B01705-0-0] [lt=67] HANDLE_ERROR: err_cb=0x7fd5d5729070, errno=-4016, errmsg=“Sequencer thread exits, thread_index=0, err=-4016”
[2024-11-26 17:07:55.474479] INFO [TLOG] handle_error (ob_log_instance.cpp:2006) [3787520][][T0][YCA230A1A2DBC-0000000001B01705-0-0] [lt=9] ERROR_CALLBACK begin(err_cb_=0x7fd5d5729070)
[2024-11-26 17:07:55.474587] INFO [TLOG] handle_error (ob_log_instance.cpp:2008) [3787520][][T0][YCA230A1A2DBC-0000000001B01705-0-0] [lt=11] ERROR_CALLBACK end(err_cb_=0x7fd5d5729070)
[2024-11-26 17:07:55.474602] INFO [TLOG] mark_stop_flag (ob_log_instance.cpp:1577) [3787520][][T0][YCA230A1A2DBC-0000000001B01705-0-0] [lt=13] mark_stop_flag begin(global_errno_=-4016, stop_reason=ERROR_CALLBACK)
[2024-11-26 17:07:55.474613] INFO [TLOG.FETCHER] mark_stop_flag (ob_log_fetcher.cpp:403) [3787520][][T0][YCA230A1A2DBC-0000000001B01705-0-0] [lt=9] mark fetcher stop begin(is_loading_data_dict_baseline_data=false)
[2024-11-26 17:07:55.474624] INFO [TLOG.FETCHER] mark_stop_flag (ob_ls_worker.cpp:147) [3787520][][T0][YCA230A1A2DBC-0000000001B01705-0-0] [lt=9] stream worker mark_stop_flag begin
[2024-11-26 17:07:55.474633] INFO [TLOG.FETCHER] mark_stop_flag (ob_ls_worker.cpp:150) [3787520][][T0][YCA230A1A2DBC-0000000001B01705-0-0] [lt=8] stream worker mark_stop_flag end
[2024-11-26 17:07:55.474640] INFO [TLOG.FETCHER] mark_stop_flag (ob_log_fetcher_dead_pool.cpp:139) [3787520][][T0][YCA230A1A2DBC-0000000001B01705-0-0] [lt=6] mark fetcher dead pool stop
[2024-11-26 17:07:55.474651] INFO [TLOG.FETCHER] mark_stop_flag (ob_log_fetcher_idle_pool.cpp:142) [3787520][][T0][YCA230A1A2DBC-0000000001B01705-0-0] [lt=9] mark fetcher idle pool stop
[2024-11-26 17:07:55.474662] INFO [TLOG.FETCHER] mark_stop_flag (ob_log_fetcher.cpp:413) [3787520][][T0][YCA230A1A2DBC-0000000001B01705-0-0] [lt=7] mark fetcher stop succ(is_loading_data_dict_baseline_data=false)
[2024-11-26 17:07:55.474693] INFO [TLOG.SORTER] mark_stop_flag (ob_log_trans_msg_sorter.cpp:185) [3787520][][T0][YCA230A1A2DBC-0000000001B01705-0-0] [lt=11] mark TransMsgSorter stop succ
[2024-11-26 17:07:55.474707] INFO [TLOG] mark_stop_flag (ob_log_resource_collector.cpp:161) [3787520][][T0][YCA230A1A2DBC-0000000001B01705-0-0] [lt=11] resource_collector mark_stop_flag
[2024-11-26 17:07:55.474722] INFO [TLOG] mark_stop_flag (ob_log_instance.cpp:1595) [3787520][][T0][YCA230A1A2DBC-0000000001B01705-0-0] [lt=8] mark_stop_flag end(global_errno_=-4016, stop_reason=ERROR_CALLBACK)
[2024-11-26 17:07:55.474739] ERROR issue_dba_error (ob_log.cpp:1875) [3787520][][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4016, file=“ob_multi_fixed_queue_thread.h”, line_no=228, info=“handle task fail”)

AntTech_4D0BKE · 2024 年11 月 26 日 18:26

重新创建binlog服务，执行一段时间之后创建表，log服务就不采集日志了，详细错误日志如下：
详细日志.txt (58.7 KB)

Giant · 2024 年11 月 27 日 09:50

登录到对应租户使用 2883端口执行 show master status ; 看下

您的租户id 应该是 1004 。使用obproxy 的2883端口登录

执行几个dml 语句后再次执行 show master status 看下

川粉 · 2024 年11 月 27 日 11:24

ob 是也换到 4.2.1.10 了吗？ obbinlog 自带的 obcdc 只适配到了 4.2.1.9，如果用 4.2.1.10，需要手动装一下 obcdc 4.2.1.10 然后覆盖一下 4.2.1.x-access 目录。

https://www.oceanbase.com/docs/community-oblogproxy-doc-1000000001493746

川粉 · 2024 年11 月 27 日 11:26

日志看起来是 obcdc 在报错，如果版本适配没问题的话，就得让 cdc 的同学看一下了。等你确认了 observer 的版本我们再进行下一步。

AntTech_4D0BKE · 2024 年11 月 27 日 14:41

老师，obcdc 4.2.1.10好像没有这个版本的包

AntTech_4D0BKE · 2024 年11 月 27 日 14:41

ob我用的 4.2.1.10

AntTech_4D0BKE · 2024 年11 月 28 日 09:11

这是更新完之后的包：

更新完之后报错
[2024-11-27 21:52:41.869913] [info] status_thread.cpp(43): COUNTS:[NREADER:0][NCHANNEL:0]
[2024-11-27 21:52:44.446215] [error] fs_util.cpp(455): Failed to calculate disk size No such file or directory for /home/ds/oblogproxy/run/u8iv01rhnj (deleted)
[2024-11-27 21:52:44.453036] [error] fs_util.cpp(455): Failed to calculate disk size No such file or directory for /home/ds/oblogproxy/run/u5wwudtmqe (deleted)
[2024-11-27 21:52:45.455997] [info] status_thread.cpp(94): METRICS:[MEM:8810/50152MB,17.57%][UDISK:45072/510640MB,8.827%][CPU:12,36.12%][LOAD1,5:428%,423%][NETIO:24KB/s,13KB/s]
[2024-11-27 21:52:45.456142] [info] status_thread.cpp(77): METRICS:[CLIENT_ID:/home/ds/oblogproxy][PID:3572530][MEM:16/50152MB,0.0319%][UDISK:11857/510640MB,2.322%][CPU:12,0%][NETIO:27KB/s,13KB/s]

川粉 · 2024 年11 月 28 日 14:07

u8iv01rhnj 和 u5wwudtmqe 这两个实例是啥时候创建的？现在是啥状态？新建 Binlog 任务有异常吗？

AntTech_4D0BKE · 2024 年11 月 28 日 15:59

启动的时候创建失败，新建不了binlog任务

AntTech_4D0BKE · 2024 年11 月 28 日 18:00

昨天应该是我配置的有问题，现在binlog服务启动了，看看这次会不会断掉