Invalid request sent from clickhouse-cpp

[IyeOnline]

I am sending data to a local clickhouse instance using clickhouse-cpp. The
program successfully send some data, but at some point fails. The point where it fails seems to be random.

My local client then throws a std::system_error:

fail to send 8192 bytes of data: Connection reset by peer

which originates from socket.cpp:462

My clickhouse sever is a local instance simply started via ./clickhouse server.

Looking into the server's console output/log I can see

server log "Unrecognized token: Syntax error"

2025.05.23 14:28:23.925116 [ 46881 ] {40ff087d-cda1-4b93-9c57-e5fbfabfb96a} <Debug> executeQuery: (from 127.0.0.1:50400) INSERT INTO ocsf_dns ( `activity_id`,`activity_name`,`category_uid`,`category_name`,`class_uid`,`class_name`,`severity_id`,`severity`,`type_uid`,`time`,`start_time`,`metadata_log_name`,`metadata_logged_time`,`metadata_product_name`,`metadata_product_vendor_name`,`metadata_product_cpe_name`,`metadata_uid`,`metadata_version`,`answers_rdata`,`answers_ttl`,`query_class`,`query_hostname`,`query_type`,`query_time`,`response_time`,`rcode`,`rcode_id`,`src_endpoint_ip`,`src_endpoint_port`,`dst_endpoint_ip`,`dst_endpoint_port`,`connection_info_direction`,`connection_info_direction_id`,`connection_info_protocol_name`,`connection_info_protocol_num`,`connection_info_protocol_ver_id`,`status`,`status_id`,`unmapped` ) VALUES (stage: Complete)
2025.05.23 14:28:23.931055 [ 46881 ] {40ff087d-cda1-4b93-9c57-e5fbfabfb96a} <Debug> TCPHandler: Processed in 0.006340198 sec.
2025.05.23 14:28:23.931178 [ 46881 ] {40ff087d-cda1-4b93-9c57-e5fbfabfb96a} <Error> executeQuery: Code: 62. DB::Exception: Unrecognized token: Syntax error (data type): failed at position 1 (): JJJJJJJJJ. . (SYNTAX_ERROR) (version 25.6.1.781 (official build)) (from 127.0.0.1:50400) (in query: INSERT INTO ocsf_dns ( `activity_id`,`activity_name`,`category_uid`,`category_name`,`class_uid`,`class_name`,`severity_id`,`severity`,`type_uid`,`time`,`start_time`,`metadata_log_name`,`metadata_logged_time`,`metadata_product_name`,`metadata_product_vendor_name`,`metadata_product_cpe_name`,`metadata_uid`,`metadata_version`,`answers_rdata`,`answers_ttl`,`query_class`,`query_hostname`,`query_type`,`query_time`,`response_time`,`rcode`,`rcode_id`,`src_endpoint_ip`,`src_endpoint_port`,`dst_endpoint_ip`,`dst_endpoint_port`,`connection_info_direction`,`connection_info_direction_id`,`connection_info_protocol_name`,`connection_info_protocol_num`,`connection_info_protocol_ver_id`,`status`,`status_id`,`unmapped` ) VALUES), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fc7139b
1. DB::Exception::createDeprecated(String const&, int, bool) @ 0x000000000fd3a94d
2. DB::parseQueryAndMovePosition(DB::IParser&, char const*&, char const*, String const&, bool, unsigned long, unsigned long, unsigned long) @ 0x0000000015ee7b52
3. std::shared_ptr<DB::IDataType const> DB::DataTypeFactory::getImpl<false>(String const&) const @ 0x0000000012e4edf6
4. DB::NativeReader::read() @ 0x0000000015433b67
5. DB::TCPHandler::receivePacketsExpectData(DB::QueryState&) @ 0x00000000153e4fe0
6. DB::TCPHandler::processInsertQuery(DB::QueryState&) @ 0x00000000153daea3
7. DB::TCPHandler::runImpl() @ 0x00000000153cb748
8. DB::TCPHandler::run() @ 0x00000000153ea719
9. Poco::Net::TCPServerConnection::start() @ 0x0000000018afea67
10. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018afeeb9
11. Poco::PooledThread::run() @ 0x0000000018aca19b
12. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018ac867d
13. start_thread @ 0x000000000009caa4
14. __clone3 @ 0x0000000000129c3c

2025.05.23 14:28:23.931449 [ 46881 ] {40ff087d-cda1-4b93-9c57-e5fbfabfb96a} <Error> TCPHandler: Can't send logs or exception to client. Close connection.: Code: 101. DB::Exception: Unexpected packet Hello received from client. (UNEXPECTED_PACKET_FROM_CLIENT), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fc7139b
1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009e8968c
2. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x0000000009e95a0b
3. DB::TCPHandler::processUnexpectedHello() @ 0x00000000153decbf
4. DB::TCPHandler::receivePacketsExpectData(DB::QueryState&) @ 0x00000000153e5bd0
5. DB::TCPHandler::skipData(DB::QueryState&) @ 0x00000000153dc670
6. DB::TCPHandler::runImpl() @ 0x00000000153cc7fe
7. DB::TCPHandler::run() @ 0x00000000153ea719
8. Poco::Net::TCPServerConnection::start() @ 0x0000000018afea67
9. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018afeeb9
10. Poco::PooledThread::run() @ 0x0000000018aca19b
11. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018ac867d
12. start_thread @ 0x000000000009caa4
13. __clone3 @ 0x0000000000129c3c
 (version 25.6.1.781 (official build))
2025.05.23 14:28:23.931557 [ 46881 ] {} <Error> TCPHandler: Code: 62. DB::Exception: Unrecognized token: Syntax error (data type): failed at position 1 (): JJJJJJJJJ. . (SYNTAX_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fc7139b
1. DB::Exception::createDeprecated(String const&, int, bool) @ 0x000000000fd3a94d
2. DB::parseQueryAndMovePosition(DB::IParser&, char const*&, char const*, String const&, bool, unsigned long, unsigned long, unsigned long) @ 0x0000000015ee7b52
3. std::shared_ptr<DB::IDataType const> DB::DataTypeFactory::getImpl<false>(String const&) const @ 0x0000000012e4edf6
4. DB::NativeReader::read() @ 0x0000000015433b67
5. DB::TCPHandler::receivePacketsExpectData(DB::QueryState&) @ 0x00000000153e4fe0
6. DB::TCPHandler::processInsertQuery(DB::QueryState&) @ 0x00000000153daea3
7. DB::TCPHandler::runImpl() @ 0x00000000153cb748
8. DB::TCPHandler::run() @ 0x00000000153ea719
9. Poco::Net::TCPServerConnection::start() @ 0x0000000018afea67
10. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018afeeb9
11. Poco::PooledThread::run() @ 0x0000000018aca19b
12. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018ac867d
13. start_thread @ 0x000000000009caa4
14. __clone3 @ 0x0000000000129c3c

I have also seen a variation of this where the error was Empty Query:, but i got the same failure in the client:

alternative server log "Empty query. (SYNTAX_ERROR)"

2025.05.23 14:55:32.203191 [ 46881 ] {18a4db8a-80fb-4234-bff7-e4f5fa4526d9} <Debug> executeQuery: (from 127.0.0.1:46478) INSERT INTO ocsf_dns ( `activity_id`,`activity_name`,`category_uid`,`category_name`,`class_uid`,`class_name`,`severity_id`,`severity`,`type_uid`,`time`,`start_time`,`metadata_log_name`,`metadata_logged_time`,`metadata_product_name`,`metadata_product_vendor_name`,`metadata_product_cpe_name`,`metadata_uid`,`metadata_version`,`answers_rdata`,`answers_ttl`,`query_class`,`query_hostname`,`query_type`,`query_time`,`response_time`,`rcode`,`rcode_id`,`src_endpoint_ip`,`src_endpoint_port`,`dst_endpoint_ip`,`dst_endpoint_port`,`connection_info_direction`,`connection_info_direction_id`,`connection_info_protocol_name`,`connection_info_protocol_num`,`connection_info_protocol_ver_id`,`status`,`status_id`,`unmapped` ) VALUES (stage: Complete)
2025.05.23 14:55:32.211155 [ 46881 ] {18a4db8a-80fb-4234-bff7-e4f5fa4526d9} <Debug> TCPHandler: Processed in 0.008380116 sec.
2025.05.23 14:55:32.211270 [ 46881 ] {18a4db8a-80fb-4234-bff7-e4f5fa4526d9} <Error> executeQuery: Code: 62. DB::Exception: Empty query. (SYNTAX_ERROR) (version 25.6.1.781 (official build)) (from 127.0.0.1:46478) (in query: INSERT INTO ocsf_dns ( `activity_id`,`activity_name`,`category_uid`,`category_name`,`class_uid`,`class_name`,`severity_id`,`severity`,`type_uid`,`time`,`start_time`,`metadata_log_name`,`metadata_logged_time`,`metadata_product_name`,`metadata_product_vendor_name`,`metadata_product_cpe_name`,`metadata_uid`,`metadata_version`,`answers_rdata`,`answers_ttl`,`query_class`,`query_hostname`,`query_type`,`query_time`,`response_time`,`rcode`,`rcode_id`,`src_endpoint_ip`,`src_endpoint_port`,`dst_endpoint_ip`,`dst_endpoint_port`,`connection_info_direction`,`connection_info_direction_id`,`connection_info_protocol_name`,`connection_info_protocol_num`,`connection_info_protocol_ver_id`,`status`,`status_id`,`unmapped` ) VALUES), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fc7139b
1. DB::Exception::createDeprecated(String const&, int, bool) @ 0x000000000fd3a94d
2. DB::parseQueryAndMovePosition(DB::IParser&, char const*&, char const*, String const&, bool, unsigned long, unsigned long, unsigned long) @ 0x0000000015ee7b52
3. std::shared_ptr<DB::IDataType const> DB::DataTypeFactory::getImpl<false>(String const&) const @ 0x0000000012e4edf6
4. DB::NativeReader::read() @ 0x0000000015433b67
5. DB::TCPHandler::receivePacketsExpectData(DB::QueryState&) @ 0x00000000153e4fe0
6. DB::TCPHandler::processInsertQuery(DB::QueryState&) @ 0x00000000153daea3
7. DB::TCPHandler::runImpl() @ 0x00000000153cb748
8. DB::TCPHandler::run() @ 0x00000000153ea719
9. Poco::Net::TCPServerConnection::start() @ 0x0000000018afea67
10. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018afeeb9
11. Poco::PooledThread::run() @ 0x0000000018aca19b
12. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018ac867d
13. start_thread @ 0x000000000009caa4
14. __clone3 @ 0x0000000000129c3c

2025.05.23 14:55:32.211535 [ 46881 ] {18a4db8a-80fb-4234-bff7-e4f5fa4526d9} <Error> TCPHandler: Can't send logs or exception to client. Close connection.: Code: 101. DB::Exception: Unexpected packet Hello received from client. (UNEXPECTED_PACKET_FROM_CLIENT), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fc7139b
1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009e8968c
2. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x0000000009e95a0b
3. DB::TCPHandler::processUnexpectedHello() @ 0x00000000153decbf
4. DB::TCPHandler::receivePacketsExpectData(DB::QueryState&) @ 0x00000000153e5bd0
5. DB::TCPHandler::skipData(DB::QueryState&) @ 0x00000000153dc670
6. DB::TCPHandler::runImpl() @ 0x00000000153cc7fe
7. DB::TCPHandler::run() @ 0x00000000153ea719
8. Poco::Net::TCPServerConnection::start() @ 0x0000000018afea67
9. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018afeeb9
10. Poco::PooledThread::run() @ 0x0000000018aca19b
11. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018ac867d
12. start_thread @ 0x000000000009caa4
13. __clone3 @ 0x0000000000129c3c
 (version 25.6.1.781 (official build))
2025.05.23 14:55:32.211631 [ 46881 ] {} <Error> TCPHandler: Code: 62. DB::Exception: Empty query. (SYNTAX_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fc7139b
1. DB::Exception::createDeprecated(String const&, int, bool) @ 0x000000000fd3a94d
2. DB::parseQueryAndMovePosition(DB::IParser&, char const*&, char const*, String const&, bool, unsigned long, unsigned long, unsigned long) @ 0x0000000015ee7b52
3. std::shared_ptr<DB::IDataType const> DB::DataTypeFactory::getImpl<false>(String const&) const @ 0x0000000012e4edf6
4. DB::NativeReader::read() @ 0x0000000015433b67
5. DB::TCPHandler::receivePacketsExpectData(DB::QueryState&) @ 0x00000000153e4fe0
6. DB::TCPHandler::processInsertQuery(DB::QueryState&) @ 0x00000000153daea3
7. DB::TCPHandler::runImpl() @ 0x00000000153cb748
8. DB::TCPHandler::run() @ 0x00000000153ea719
9. Poco::Net::TCPServerConnection::start() @ 0x0000000018afea67
10. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018afeeb9
11. Poco::PooledThread::run() @ 0x0000000018aca19b
12. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018ac867d
13. start_thread @ 0x000000000009caa4
14. __clone3 @ 0x0000000000129c3c

I also got back the exception from the server a few times:

server log where I get back the exception in my client

2025.05.27 14:56:24.999297 [ 27578 ] {3fefab3e-45b5-4be3-a9a6-dbba9ea53f02} <Debug> executeQuery: (from 127.0.0.1:53512) INSERT INTO ocsf_dns ( `activity_id`,`activity_name`,`category_uid`,`category_name`,`class_uid`,`class_name`,`severity_id`,`severity`,`type_uid`,`time`,`start_time`,`metadata`,`answers`,`query`,`query_time`,`response_time`,`rcode`,`rcode_id`,`src_endpoint`,`dst_endpoint`,`connection_info`,`status`,`status_id`,`unmapped` ) VALUES (stage: Complete)
2025.05.27 14:56:25.006626 [ 27578 ] {3fefab3e-45b5-4be3-a9a6-dbba9ea53f02} <Debug> TCPHandler: Processed in 0.007714557 sec.
2025.05.27 14:56:25.006968 [ 27578 ] {3fefab3e-45b5-4be3-a9a6-dbba9ea53f02} <Error> executeQuery: Code: 62. DB::Exception: Empty query. (SYNTAX_ERROR) (version 25.6.1.781 (official build)) (from 127.0.0.1:53512) (in query: INSERT INTO ocsf_dns ( `activity_id`,`activity_name`,`category_uid`,`category_name`,`class_uid`,`class_name`,`severity_id`,`severity`,`type_uid`,`time`,`start_time`,`metadata`,`answers`,`query`,`query_time`,`response_time`,`rcode`,`rcode_id`,`src_endpoint`,`dst_endpoint`,`connection_info`,`status`,`status_id`,`unmapped` ) VALUES), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fc7139b
1. DB::Exception::createDeprecated(String const&, int, bool) @ 0x000000000fd3a94d
2. DB::parseQueryAndMovePosition(DB::IParser&, char const*&, char const*, String const&, bool, unsigned long, unsigned long, unsigned long) @ 0x0000000015ee7b52
3. std::shared_ptr<DB::IDataType const> DB::DataTypeFactory::getImpl<false>(String const&) const @ 0x0000000012e4edf6
4. DB::NativeReader::read() @ 0x0000000015433b67
5. DB::TCPHandler::receivePacketsExpectData(DB::QueryState&) @ 0x00000000153e4fe0
6. DB::TCPHandler::processInsertQuery(DB::QueryState&) @ 0x00000000153daea3
7. DB::TCPHandler::runImpl() @ 0x00000000153cb748
8. DB::TCPHandler::run() @ 0x00000000153ea719
9. Poco::Net::TCPServerConnection::start() @ 0x0000000018afea67
10. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018afeeb9
11. Poco::PooledThread::run() @ 0x0000000018aca19b
12. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018ac867d
13. start_thread @ 0x000000000009caa4
14. __clone3 @ 0x0000000000129c3c

2025.05.27 14:56:25.015245 [ 27578 ] {3fefab3e-45b5-4be3-a9a6-dbba9ea53f02} <Error> TCPHandler: Can't send logs or exception to client. Close connection.: Code: 101. DB::Exception: Unexpected packet Hello received from client. (UNEXPECTED_PACKET_FROM_CLIENT), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fc7139b
1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009e8968c
2. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x0000000009e95a0b
3. DB::TCPHandler::processUnexpectedHello() @ 0x00000000153decbf
4. DB::TCPHandler::receivePacketsExpectData(DB::QueryState&) @ 0x00000000153e5bd0
5. DB::TCPHandler::skipData(DB::QueryState&) @ 0x00000000153dc670
6. DB::TCPHandler::runImpl() @ 0x00000000153cc7fe
7. DB::TCPHandler::run() @ 0x00000000153ea719
8. Poco::Net::TCPServerConnection::start() @ 0x0000000018afea67
9. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018afeeb9
10. Poco::PooledThread::run() @ 0x0000000018aca19b
11. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018ac867d
12. start_thread @ 0x000000000009caa4
13. __clone3 @ 0x0000000000129c3c
 (version 25.6.1.781 (official build))
2025.05.27 14:56:25.015351 [ 27578 ] {} <Error> TCPHandler: Code: 62. DB::Exception: Empty query. (SYNTAX_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fc7139b
1. DB::Exception::createDeprecated(String const&, int, bool) @ 0x000000000fd3a94d
2. DB::parseQueryAndMovePosition(DB::IParser&, char const*&, char const*, String const&, bool, unsigned long, unsigned long, unsigned long) @ 0x0000000015ee7b52
3. std::shared_ptr<DB::IDataType const> DB::DataTypeFactory::getImpl<false>(String const&) const @ 0x0000000012e4edf6
4. DB::NativeReader::read() @ 0x0000000015433b67
5. DB::TCPHandler::receivePacketsExpectData(DB::QueryState&) @ 0x00000000153e4fe0
6. DB::TCPHandler::processInsertQuery(DB::QueryState&) @ 0x00000000153daea3
7. DB::TCPHandler::runImpl() @ 0x00000000153cb748
8. DB::TCPHandler::run() @ 0x00000000153ea719
9. Poco::Net::TCPServerConnection::start() @ 0x0000000018afea67
10. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018afeeb9
11. Poco::PooledThread::run() @ 0x0000000018aca19b
12. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018ac867d
13. start_thread @ 0x000000000009caa4
14. __clone3 @ 0x0000000000129c3c

---

As far as I can tell, the Block I am sending is well-formed.

I can provide a full end-to-end setup/demo if you are interested. Our source
code is open-source here: https://github.com/tenzir/tenzir/tree/main/plugins/clickhouse
and I can also give you the pipeline & data used for testing. Unfortunately I
was not yet able to craft a smaller scale test case. For smaller handcrafted
tests, I do not have any issues.

Let me know if I can assist in tracking this down in any way, I'd be happy to
help!

[mshustov]

I can provide a full end-to-end setup/demo if you are interested. Our source

yes, a minimal repro scenario would be helpful.

[IyeOnline]

I have mostly figured it out now.

It is caused by trying to send invalid Arrays with missmatching offsets + data arrays. While the invalid arrays are clearly created by my code, there seem to be some basic sanity checks for this missing in clickhouse-cpp .

More importantly, the Clickhouse sever itself also does not properly guard against this/validate the request it gets, causing it to essentially "go out of bounds", leading to almost random errors.

Once I have my own thing fixed, I will create a simple repro and file proper issues for both of these points.

[IyeOnline]

I opened up a PR that fixes my issue and added a reproduction to that:

#428