:memo: Consolidate a documentation format to a single approach

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

What type of changes will we make to the documentation?

Refactoring

Affects Version(s)

master/1.11

Improving the documentation

Description

Currently, the Apache Kyuubi documentation uses a hybrid approach, mixing reStructuredText (.rst) and Markdown (.md) files, while being processed by Sphinx.

While this hybrid approach provides flexibility (using myst-parser for Markdown), it introduces several challenges:

1. Contributors need to be familiar with both syntaxes.
2. Cross-referencing between RST and MD files can be error-prone and hard to detect.
3. The build pipeline relies on multiple Sphinx extensions (like myst-parser, sphinx-markdown-tables) to bridge the gap.
4. Formatting features available in RST might not be easily replicable in Markdown (or vice-versa).

I propose migrating to a single technology to streamline the contribution process and improve documentation quality.

---

Technology Comparison

Below is an analysis of the two primary paths for consolidation:

Option 1: Migrate to Markdown (using MyST-Parser)

• Pros:
  • Markdown is the de facto standard for most developers, so it is easier for new contributors.
  • Better preview support in GitHub, IDEs, and other tools.
  • MyST (Markedly Structured Text) provides almost all the power of RST (directives, roles) while remaining valid Markdown.
• Cons:
  • Once we start using advanced features, the Markdown "cleanliness" can get cluttered with curly braces and colons.
  • Some advanced Sphinx features or specialized extensions may still favor RST's native structure.

Option 2: Migrate to reStructuredText (RST)

• Pros:
  • RST is the native format for Sphinx, offering the most robust support for technical documentation features.
  • Eliminates the need for "bridge" plugins like myst-parser.
• Cons:
  • RST syntax is more verbose and less intuitive for casual contributors compared to Markdown.
  • Limited rendering outside of Sphinx-based environments.

---

Action Plan

1. Decide on the target format (Markdown/MyST vs. RST).
2. Convert a few effortless directories to the target format using automated tools or AI agents to verify the approach.
3. Deprecate a non-target format in contributing guidelines.
4. Convert other pages, including those requiring fixing broken cross-references and formatting edge cases.
5. Remove unnecessary Sphinx dependencies from docs/requirements.txt and configuration from conf.py after migration.

---

What do you think? Should we standardize on Markdown for ease of contribution or stick with RST for its native power?

Are you willing to submit PR?

• Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
• No. I cannot submit a PR at this time.