Skip to content

Clickhouse migration is broken  #7836

Open
Bug
Open
Clickhouse migration is broken #7836
Bug
@sentry

Description

@sentry
sentry
bot
opened on Jun 19, 2026

Sentry Issue: FLAGSMITH-API-5Q7

ServerException: DB::Exception: Looks like this replica doesn't catchup with latest ALTER query updates: metadata version on replica is 553, while common metadata is 555. Please retry this query. Stack trace:

0. ./ci/tmp/build/./src/Common/Exception.cpp:141:1: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001692bdaa
1. ./src/Common/Exception.h:171:100: DB::Exception::Exception(String&&, int, String, bool) @ 0x000000000e7a484e
2. ./src/Common/Exception.h:57:54: DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000e7a4189
3. ./src/Common/Exception.h:189:77: DB::Exception::Exception<int, int&>(int, FormatStringHelperImpl<std::type_identity<int>::type, std::type_identity<int&>::type>, int&&, int&) @ 0x000000001cdbd576
4. ./ci/tmp/build/./src/Storages/StorageSharedMergeTree.cpp:6861:27: DB::StorageSharedMergeTree::alter(DB::AlterCommands const&, std::shared_ptr<DB::Context const>, std::unique_lock<std::timed_mutex>&)::$_0::operator()() const @ 0x000000001cd99a17
5. ./ci/tmp/build/./src/Storages/StorageSharedMergeTree.cpp:6989:9: DB::StorageSharedMergeTree::alter(DB::AlterCommands const&, std::shared_ptr<DB::Context const>, std::unique_lock<std::timed_mutex>&) @ 0x000000001cd90fb9
6. ./ci/tmp/build/./src/Interpreters/InterpreterAlterQuery.cpp:346:16: DB::InterpreterAlterQuery::executeToTable(DB::ASTAlterQuery const&) @ 0x000000001bacddd2
7. ./ci/tmp/build/./src/Interpreters/InterpreterAlterQuery.cpp:89:16: DB::InterpreterAlterQuery::execute() @ 0x000000001bac9493
8. ./ci/tmp/build/./src/Interpreters/executeQuery.cpp:1776:40: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, std::unique_ptr<DB::ReadBuffer, std::default_delete<DB::ReadBuffer>>&, boost::intrusive_ptr<DB::IAST>&, std::shared_ptr<DB::ImplicitTransactionControlExecutor>, std::function<void ()>, DB::QueryResultDetails&) @ 0x000000001c0e38be
9. ./ci/tmp/build/./src/Interpreters/executeQuery.cpp:2320:19: DB::executeQuery(std::unique_ptr<DB::ReadBuffer, std::default_delete<DB::ReadBuffer>>, DB::WriteBuffer&, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&, String const&, std::shared_ptr<DB::Context const> const&, std::optional<DB::FormatSettings> const&)>, std::function<void ()>, std::function<void ()>) @ 0x000000001c0ebc04
10. ./ci/tmp/build/./src/Interpreters/executeQuery.cpp:2182:5: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&, String const&, std::shared_ptr<DB::Context const> const&, std::optional<DB::FormatSettings> const&)>, std::function<void ()>, std::function<void ()>) @ 0x000000001c0eb2e4
11. ./ci/tmp/build/./src/Interpreters/SharedDatabaseCatalog.cpp:5543:9: DB::SharedDatabaseCatalog::tryExecuteDDLQuery(boost::intrusive_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const>, DB::QueryFlags) @ 0x000000001bd4fd94
12. ./ci/tmp/build/./src/Interpreters/InterpreterAlterQuery.cpp:149:50: DB::InterpreterAlterQuery::executeToTable(DB::ASTAlterQuery const&) @ 0x000000001bacb537
13. ./ci/tmp/build/./src/Interpreters/InterpreterAlterQuery.cpp:89:16: DB::InterpreterAlterQuery::execute() @ 0x000000001bac9493
14. ./ci/tmp/build/./src/Interpreters/executeQuery.cpp:1776:40: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, std::unique_ptr<DB::ReadBuffer, std::default_delete<DB::ReadBuffer>>&, boost::intrusive_ptr<DB::IAST>&, std::shared_ptr<DB::ImplicitTransactionControlExecutor>, std::function<void ()>, DB::QueryResultDetails&) @ 0x000000001c0e38be
15. ./ci/tmp/build/./src/Interpreters/executeQuery.cpp:2101:11: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001c0dde42
16. ./ci/tmp/build/./src/Server/TCPHandler.cpp:796:68: DB::TCPHandler::runImpl() @ 0x000000001dcd24ec
17. ./ci/tmp/build/./src/Server/TCPHandler.cpp:2954:9: DB::TCPHandler::run() @ 0x000000001dcf5164
18. ./src/Server/TCPProtocolStackHandler.h:42:25: DB::TCPProtocolStackHandler::run() @ 0x0000000016d7b394
19. ./ci/tmp/build/./base/poco/Net/src/TCPServerConnection.cpp:40:3: Poco::Net::TCPServerConnection::start() @ 0x000000002339fc07
20. ./ci/tmp/build/./base/poco/Net/src/TCPServerDispatcher.cpp:115:42: Poco::Net::TCPServerDispatcher::run() @ 0x00000000233a0182
21. ./ci/tmp/build/./base/poco/Foundation/src/ThreadPool.cpp:205:14: Poco::PooledThread::run() @ 0x0000000023357345
22. ./base/poco/Foundation/src/Thread_POSIX.cpp:341:27: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000233558cf
23. ? @ 0x0000000000094ac3
24. ? @ 0x00000000001268d0

(2 additional frame(s) were not displayed)
...
  File "clickhouse_driver/client.py", line 580, in process_ordinary_query
    return self.receive_result(with_column_types=with_column_types,
  File "clickhouse_driver/client.py", line 212, in receive_result
    return result.get_result()
  File "clickhouse_driver/result.py", line 50, in get_result
    for packet in self.packet_generator:
  File "clickhouse_driver/client.py", line 228, in packet_generator
    packet = self.receive_packet()
  File "clickhouse_driver/client.py", line 245, in receive_packet
    raise packet.exception

OperationalError: Code: 517.
DB::Exception: Looks like this replica doesn't catchup with latest ALTER query updates: metadata version on replica is 553, while common metadata is 555. Please retry this query. Stack trace:

0. ./ci/tmp/build/./src/Common/Exception.cpp:141:1: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001692bdaa
1. ./src/Common/Exception.h:171:100: DB::Exception::Exception(String&&, int, String, bool) @ 0x000000000e7a484e
2. ./src/Common/Exception.h:57:54: DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000e7a4189
3. ./src/Common/Exception.h:189:77: DB::Exception::Exception<int, int&>(int, FormatStringHelperImpl<std::type_identity<int>::type, std::type_identity<int&>::type>, int&&, int&) @ 0x000000001cdbd576
4. ./ci/tmp/build/./src/Storages/StorageSharedMergeTree.cpp:6861:27: DB::StorageSharedMergeTree::alter(DB::AlterCommands const&, std::shared_ptr<DB::Context const>, std::unique_lock<std::timed_mutex>&)::$_0::operator()() const @ 0x000000001cd99a17
5. ./ci/tmp/build/./src/Storages/StorageSharedMergeTree.cpp:6989:9: DB::StorageSharedMergeTree::alter(DB::AlterCommands const&, std::shared_ptr<DB::Context const>, std::unique_lock<std::timed_mutex>&) @ 0x000000001cd90fb9
6. ./ci/tmp/build/./src/Interpreters/InterpreterAlterQuery.cpp:346:16: DB::InterpreterAlterQuery::executeToTable(DB::ASTAlterQuery const&) @ 0x000000001bacddd2
7. ./ci/tmp/build/./src/Interpreters/InterpreterAlterQuery.cpp:89:16: DB::InterpreterAlterQuery::execute() @ 0x000000001bac9493
8. ./ci/tmp/build/./src/Interpreters/executeQuery.cpp:1776:40: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, std::unique_ptr<DB::ReadBuffer, std::default_delete<DB::ReadBuffer>>&, boost::intrusive_ptr<DB::IAST>&, std::shared_ptr<DB::ImplicitTransactionControlExecutor>, std::function<void ()>, DB::QueryResultDetails&) @ 0x000000001c0e38be
9. ./ci/tmp/build/./src/Interpreters/executeQuery.cpp:2320:19: DB::executeQuery(std::unique_ptr<DB::ReadBuffer, std::default_delete<DB::ReadBuffer>>, DB::WriteBuffer&, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&, String const&, std::shared_ptr<DB::Context const> const&, std::optional<DB::FormatSettings> const&)>, std::function<void ()>, std::function<void ()>) @ 0x000000001c0ebc04
10. ./ci/tmp/build/./src/Interpreters/executeQuery.cpp:2182:5: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&, String const&, std::shared_ptr<DB::Context const> const&, std::optional<DB::FormatSettings> const&)>, std::function<void ()>, std::function<void ()>) @ 0x000000001c0eb2e4
11. ./ci/tmp/build/./src/Interpreters/SharedDatabaseCatalog.cpp:5543:9: DB::SharedDatabaseCatalog::tryExecuteDDLQuery(boost::intrusive_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const>, DB::QueryFlags) @ 0x000000001bd4fd94
12. ./ci/tmp/build/./src/Interpreters/InterpreterAlterQuery.cpp:149:50: DB::InterpreterAlterQuery::executeToTable(DB::ASTAlterQuery const&) @ 0x000000001bacb537
13. ./ci/tmp/build/./src/Interpreters/InterpreterAlterQuery.cpp:89:16: DB::InterpreterAlterQuery::execute() @ 0x000000001bac9493
14. ./ci/tmp/build/./src/Interpreters/executeQuery.cpp:1776:40: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, std::unique_ptr<DB::ReadBuffer, std::default_delete<DB::ReadBuffer>>&, boost::intrusive_ptr<DB::IAST>&, std::shared_ptr<DB::ImplicitTransactionControlExecutor>, std::function<void ()>, DB::QueryResultDetails&) @ 0x000000001c0e38be
15. ./ci/tmp/build/./src/Interpreters/executeQuery.cpp:2101:11: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001c0dde42
16. ./ci/tmp/build/./src/Server/TCPHandler.cpp:796:68: DB::TCPHandler::runImpl() @ 0x000000001dcd24ec
17. ./ci/tmp/build/./src/Server/TCPHandler.cpp:2954:9: DB::TCPHandler::run() @ 0x000000001dcf5164
18. ./src/Server/TCPProtocolStackHandler.h:42:25: DB::TCPProtocolStackHandler::run() @ 0x0000000016d7b394
19. ./ci/tmp/build/./base/poco/Net/src/TCPServerConnection.cpp:40:3: Poco::Net::TCPServerConnection::start() @ 0x000000002339fc07
20. ./ci/tmp/build/./base/poco/Net/src/TCPServerDispatcher.cpp:115:42: Poco::Net::TCPServerDispatcher::run() @ 0x00000000233a0182
21. ./ci/tmp/build/./base/poco/Foundation/src/ThreadPool.cpp:205:14: Poco::PooledThread::run() @ 0x0000000023357345
22. ./base/poco/Foundation/src/Thread_POSIX.cpp:341:27: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000233558cf
23. ? @ 0x0000000000094ac3
24. ? @ 0x00000000001268d0

  File "django/db/backends/utils.py", line 103, in _execute
    return self.cursor.execute(sql)
  File "clickhouse_backend/driver/connection.py", line 171, in execute
    super().execute(operation, parameters)
  File "clickhouse_driver/dbapi/cursor.py", line 117, in execute
    raise OperationalError(orig)

OperationalError: Code: 517.
DB::Exception: Looks like this replica doesn't catchup with latest ALTER query updates: metadata version on replica is 553, while common metadata is 555. Please retry this query. Stack trace:

0. ./ci/tmp/build/./src/Common/Exception.cpp:141:1: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001692bdaa
1. ./src/Common/Exception.h:171:100: DB::Exception::Exception(String&&, int, String, bool) @ 0x000000000e7a484e
2. ./src/Common/Exception.h:57:54: DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000e7a4189
3. ./src/Common/Exception.h:189:77: DB::Exception::Exception<int, int&>(int, FormatStringHelperImpl<std::type_identity<int>::type, std::type_identity<int&>::type>, int&&, int&) @ 0x000000001cdbd576
4. ./ci/tmp/build/./src/Storages/StorageSharedMergeTree.cpp:6861:27: DB::StorageSharedMergeTree::alter(DB::AlterCommands const&, std::shared_ptr<DB::Context const>, std::unique_lock<std::timed_mutex>&)::$_0::operator()() const @ 0x000000001cd99a17
5. ./ci/tmp/build/./src/Storages/StorageSharedMergeTree.cpp:6989:9: DB::StorageSharedMergeTree::alter(DB::AlterCommands const&, std::shared_ptr<DB::Context const>, std::unique_lock<std::timed_mutex>&) @ 0x000000001cd90fb9
6. ./ci/tmp/build/./src/Interpreters/InterpreterAlterQuery.cpp:346:16: DB::InterpreterAlterQuery::executeToTable(DB::ASTAlterQuery const&) @ 0x000000001bacddd2
7. ./ci/tmp/build/./src/Interpreters/InterpreterAlterQuery.cpp:89:16: DB::InterpreterAlterQuery::execute() @ 0x000000001bac9493
8. ./ci/tmp/build/./src/Interpreters/executeQuery.cpp:1776:40: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, std::unique_ptr<DB::ReadBuffer, std::default_delete<DB::ReadBuffer>>&, boost::intrusive_ptr<DB::IAST>&, std::shared_ptr<DB::ImplicitTransactionControlExecutor>, std::function<void ()>, DB::QueryResultDetails&) @ 0x000000001c0e38be
9. ./ci/tmp/build/./src/Interpreters/executeQuery.cpp:2320:19: DB::executeQuery(std::unique_ptr<DB::ReadBuffer, std::default_delete<DB::ReadBuffer>>, DB::WriteBuffer&, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&, String const&, std::shared_ptr<DB::Context const> const&, std::optional<DB::FormatSettings> const&)>, std::function<void ()>, std::function<void ()>) @ 0x000000001c0ebc04
10. ./ci/tmp/build/./src/Interpreters/executeQuery.cpp:2182:5: DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&, String const&, std::shared_ptr<DB::Context const> const&, std::optional<DB::FormatSettings> const&)>, std::function<void ()>, std::function<void ()>) @ 0x000000001c0eb2e4
11. ./ci/tmp/build/./src/Interpreters/SharedDatabaseCatalog.cpp:5543:9: DB::SharedDatabaseCatalog::tryExecuteDDLQuery(boost::intrusive_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const>, DB::QueryFlags) @ 0x000000001bd4fd94
12. ./ci/tmp/build/./src/Interpreters/InterpreterAlterQuery.cpp:149:50: DB::InterpreterAlterQuery::executeToTable(DB::ASTAlterQuery const&) @ 0x000000001bacb537
13. ./ci/tmp/build/./src/Interpreters/InterpreterAlterQuery.cpp:89:16: DB::InterpreterAlterQuery::execute() @ 0x000000001bac9493
14. ./ci/tmp/build/./src/Interpreters/executeQuery.cpp:1776:40: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, std::unique_ptr<DB::ReadBuffer, std::default_delete<DB::ReadBuffer>>&, boost::intrusive_ptr<DB::IAST>&, std::shared_ptr<DB::ImplicitTransactionControlExecutor>, std::function<void ()>, DB::QueryResultDetails&) @ 0x000000001c0e38be
15. ./ci/tmp/build/./src/Interpreters/executeQuery.cpp:2101:11: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001c0dde42
16. ./ci/tmp/build/./src/Server/TCPHandler.cpp:796:68: DB::TCPHandler::runImpl() @ 0x000000001dcd24ec
17. ./ci/tmp/build/./src/Server/TCPHandler.cpp:2954:9: DB::TCPHandler::run() @ 0x000000001dcf5164
18. ./src/Server/TCPProtocolStackHandler.h:42:25: DB::TCPProtocolStackHandler::run() @ 0x0000000016d7b394
19. ./ci/tmp/build/./base/poco/Net/src/TCPServerConnection.cpp:40:3: Poco::Net::TCPServerConnection::start() @ 0x000000002339fc07
20. ./ci/tmp/build/./base/poco/Net/src/TCPServerDispatcher.cpp:115:42: Poco::Net::TCPServerDispatcher::run() @ 0x00000000233a0182
21. ./ci/tmp/build/./base/poco/Foundation/src/ThreadPool.cpp:205:14: Poco::PooledThread::run() @ 0x0000000023357345
22. ./base/poco/Foundation/src/Thread_POSIX.cpp:341:27: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000233558cf
23. ? @ 0x0000000000094ac3
24. ? @ 0x00000000001268d0

(13 additional frame(s) were not displayed)
...
  File "manage.py", line 9, in <module>
    main()

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Bug

    Fields

    No fields configured for Bug.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions