Add structured and configurable logging support to Kubeflow SDK

What you would like to be added?

I would like the Kubeflow SDK to include structured, configurable logging across its modules. Specifically:

Consistent use of Python’s logging library instead of print statements.

Support for different logging levels (DEBUG, INFO, WARNING, ERROR).

Ability for users to configure log formatting and destinations (stdout, file, JSON, etc.).

Clear and actionable log messages for key SDK operations (e.g., pipeline compilation, run submission, experiment creation).

Why is this needed?

Currently, the Kubeflow SDK provides limited visibility into its execution flow, making it difficult to debug and trace issues during development and runtime interactions.

Proper logging would:

• Improve developer experience by making failures and warnings more transparent.
• Help users understand the sequence of SDK operations and their outcomes.
• Enable integration with observability tools (e.g., ELK, ...etc) by supporting structured log output.
• Align the SDK with standard Python practices for enterprise-grade software, making it easier to maintain and troubleshoot.

Love this feature?

Give it a 👍 We prioritize the features with most 👍