ClickHouse/ClickHouse

Request to allow storage policy with MaterializedPostgreSQL

Open

#81,878 opened on 2025年6月15日

GitHub で見る
 (0 comments) (0 reactions) (0 assignees)C++ (8,400 forks)batch import
featurehelp wanted

Repository metrics

Stars
 (47,419 stars)
PR merge metrics
 (平均マージ 2d 2h) (30d で 1,000 merged PRs)

説明

Company or project name

No response

Use case

My AWS RDS postgres has terrabytes of data and it would keep growing with time. I would like to be able to sync that to Clickhouse, but use S3 as storage. But MaterializedPostgreSQL doesn't support storage_policy and thus I am unable to do that at the moment. Storing everything locally on disk means I will have to manually increase EBS size regularly which I would like to avoid. Also, starting with a large EBS would mean higher than needed cost

2025.06.15 16:01:32.731335 [ 788 ] {} TCPHandler: Code: 115. DB::Exception: Unknown setting 'storage_policy': for storage MaterializedPostgreSQL. (UNKNOWN_SETTING), Stack trace (when copying this message, always include the lines below):

  1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000d759a88
  2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x00000000090c437c
  3. DB::Exception::Exception(int, FormatStringHelperImpl<std::type_identity::type>, String&&) @ 0x00000000090c40d8
  4. DB::BaseSettingsHelpers::throwSettingNotFound(std::basic_string_view<char, std::char_traits>) @ 0x00000000101998e8
  5. DB::BaseSettingsDB::MaterializedPostgreSQLSettingsTraits::set(std::basic_string_view<char, std::char_traits>, DB::Field const&) @ 0x000000001069e4a4
  6. DB::MaterializedPostgreSQLSettings::loadFromQuery(DB::ASTStorage&) @ 0x000000001069f784
  7. std::shared_ptrDB::IDatabase std::__function::__policy_invoker<std::shared_ptrDB::IDatabase (DB::DatabaseFactory::Arguments const&)>::__call_impl[abi:ne190107]<std::__function::__default_alloc_func<DB::registerDatabaseMaterializedPostgreSQL(DB::DatabaseFactory&)::$_0, std::shared_ptrDB::IDatabase (DB::DatabaseFactory::Arguments const&)>>(std::__function::__policy_storage const*, DB::DatabaseFactory::Arguments const&) (.llvm.8839956939764748957) @ 0x00000000106826a4
  8. DB::DatabaseFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context const>) @ 0x00000000105c8778
  9. DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x00000000110420fc
  10. DB::InterpreterCreateQuery::execute() @ 0x0000000011062974
  11. DB::executeQueryImpl(char const*, char const*, std::shared_ptrDB::Context, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*, std::shared_ptrDB::IAST&) @ 0x00000000113ab24c
  12. DB::executeQuery(String const&, std::shared_ptrDB::Context, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x00000000113a5fc4
  13. DB::TCPHandler::runImpl() @ 0x00000000124e4020
  14. DB::TCPHandler::run() @ 0x00000000124ff948
  15. Poco::Net::TCPServerConnection::start() @ 0x000000001550abb8
  16. Poco::Net::TCPServerDispatcher::run() @ 0x000000001550b0d4
  17. Poco::PooledThread::run() @ 0x00000000154d545c
  18. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000154d3830
  19. ? @ 0x000000000007d5b8
  20. ? @ 0x00000000000e5edc

Describe the solution you'd like

Ideally, I would like be able to use s3 backed storage with MaterializedPostgreSQL.

Describe alternatives you've considered

No response

Additional context

No response

コントリビューターガイド