[Feature] Add an extensible file-format provider contract to PyPaimon

Inizia leggendo paimon-python/pypaimon/read/split_read.py e paimon-python/pypaimon/write/writer/data_writer.py per comprendere gli attuali rami di dispatch. Poi esamina le issue #6739 e #8049 e la PR #8292 per le relative restrizioni di formato e packaging. La issue sarà pronta per l’implementazione solo dopo che i maintainer avranno deciso i confini dell’estensione, il modello di registrazione e se sia necessario un PIP.

Search before asking

• I searched in the issues and found nothing similar.

Motivation

PyPaimon currently selects readers and writers with built-in if/elif branches. New formats such
as Lance and Mosaic have therefore been integrated directly into the PyPaimon package. This works
well for formats that the project wants to support and release as built-ins, but there is no stable
extension boundary for a deployment-specific format or for an embedding runtime that supplies its
own implementation.

Java Paimon already discovers FileFormatFactory implementations. I also noticed that Java PR
#8292 proposed another runtime-side FileFormatProvider and was closed because an execution engine
could provide that SPI in its own integration. PyPaimon is itself the SDK/runtime that performs the
format dispatch, so an embedding application cannot add a format without patching that dispatch.

Would the community consider a small PyPaimon extension contract, or is built-in-only format
support the intended boundary?

Solution

A possible contract would have these properties:

1. Table metadata stores only an immutable, language-neutral format identifier, never a Python
   module or class name.
2. A provider declares read and write capabilities independently and receives public context
   objects rather than PyPaimon writer implementation details.
3. Registration is explicit and deterministic. Python package entry points may be an optional
   discovery mechanism, but duplicate identifiers fail rather than depend on import order.
4. A missing provider fails during planning with a clear diagnostic. It never falls back to a
   different physical format.
5. Packaging and distribution to remote workers remain the responsibility of the embedding engine;
   the table does not persist package locations.
6. The extension API has a small version/capability handshake so a provider developed against one
   PyPaimon release does not silently run with incompatible context objects.

The minimum API could be an explicit registry such as
register_file_format(identifier, provider). Entry-point discovery can be considered separately
after the packaging work in #8049 is settled.

Questions:

• Is third-party format discovery in scope for PyPaimon?
• If so, would maintainers prefer an explicit registry, Python entry points, or an embedding-runtime
  hook?
• Should the first version expose read-only providers before defining writer and abort semantics?
• Would this public extension contract require a PIP before a prototype PR?

Anything else?

Relevant code and discussions:

• PyPaimon reader dispatch:
  https://github.com/apache/paimon/blob/master/paimon-python/pypaimon/read/split_read.py
• PyPaimon writer dispatch:
  https://github.com/apache/paimon/blob/master/paimon-python/pypaimon/write/writer/data_writer.py
• Lance format discussion: https://github.com/apache/paimon/issues/6739
• PyPaimon packaging discussion: https://github.com/apache/paimon/issues/8049
• Java runtime-side provider discussion: https://github.com/apache/paimon/pull/8292

Are you willing to submit a PR?

• I'm willing to submit a PR!

I can contribute a minimal read-only prototype with an out-of-tree example provider and tests for
duplicate registration, missing providers, capability checks, and distributed-worker loading after
the preferred boundary is clear.