Request to allow storage policy with MaterializedPostgreSQL
#81,878 opened on 2025年6月15日
Repository metrics
- Stars
- (47,419 stars)
- PR merge metrics
- (平均マージ 2d 2h) (30d で 1,000 merged PRs)
説明
Company or project name
No response
Use case
My AWS RDS postgres has terrabytes of data and it would keep growing with time. I would like to be able to sync that to Clickhouse, but use S3 as storage. But MaterializedPostgreSQL doesn't support storage_policy and thus I am unable to do that at the moment. Storing everything locally on disk means I will have to manually increase EBS size regularly which I would like to avoid. Also, starting with a large EBS would mean higher than needed cost
2025.06.15 16:01:32.731335 [ 788 ] {} TCPHandler: Code: 115. DB::Exception: Unknown setting 'storage_policy': for storage MaterializedPostgreSQL. (UNKNOWN_SETTING), Stack trace (when copying this message, always include the lines below):
- DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000d759a88
- DB::Exception::Exception(PreformattedMessage&&, int) @ 0x00000000090c437c
- DB::Exception::Exception(int, FormatStringHelperImpl<std::type_identity::type>, String&&) @ 0x00000000090c40d8
- DB::BaseSettingsHelpers::throwSettingNotFound(std::basic_string_view<char, std::char_traits>) @ 0x00000000101998e8
- DB::BaseSettingsDB::MaterializedPostgreSQLSettingsTraits::set(std::basic_string_view<char, std::char_traits>, DB::Field const&) @ 0x000000001069e4a4
- DB::MaterializedPostgreSQLSettings::loadFromQuery(DB::ASTStorage&) @ 0x000000001069f784
- std::shared_ptrDB::IDatabase std::__function::__policy_invoker<std::shared_ptrDB::IDatabase (DB::DatabaseFactory::Arguments const&)>::__call_impl[abi:ne190107]<std::__function::__default_alloc_func<DB::registerDatabaseMaterializedPostgreSQL(DB::DatabaseFactory&)::$_0, std::shared_ptrDB::IDatabase (DB::DatabaseFactory::Arguments const&)>>(std::__function::__policy_storage const*, DB::DatabaseFactory::Arguments const&) (.llvm.8839956939764748957) @ 0x00000000106826a4
- DB::DatabaseFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context const>) @ 0x00000000105c8778
- DB::InterpreterCreateQuery::createDatabase(DB::ASTCreateQuery&) @ 0x00000000110420fc
- DB::InterpreterCreateQuery::execute() @ 0x0000000011062974
- DB::executeQueryImpl(char const*, char const*, std::shared_ptrDB::Context, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*, std::shared_ptrDB::IAST&) @ 0x00000000113ab24c
- DB::executeQuery(String const&, std::shared_ptrDB::Context, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x00000000113a5fc4
- DB::TCPHandler::runImpl() @ 0x00000000124e4020
- DB::TCPHandler::run() @ 0x00000000124ff948
- Poco::Net::TCPServerConnection::start() @ 0x000000001550abb8
- Poco::Net::TCPServerDispatcher::run() @ 0x000000001550b0d4
- Poco::PooledThread::run() @ 0x00000000154d545c
- Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000154d3830
- ? @ 0x000000000007d5b8
- ? @ 0x00000000000e5edc
Describe the solution you'd like
Ideally, I would like be able to use s3 backed storage with MaterializedPostgreSQL.
Describe alternatives you've considered
No response
Additional context
No response