[TASK][CHALLENGE] Support Spark Connect Frontend/Backend

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before creating

• I have searched in the task list and found no similar tasks.

Mentor

• I have sufficient knowledge and experience of this task, and I volunteer to be the mentor of this task to guide contributors to complete the task.

Skill requirements

• Knowledge about Spark Connect
• Knowledge about Kyuubi architecture
• Knowledge about protobuf
• Knowledge about grpc
• Knowledge about thrift

Background and Goals

Make Kyuubi server compatible with Spark Connect protocol, so that people can use Spark Connect client to connect to Kyuubi Server.

Implementation steps

1. Add a new Spark Connect frontend 1.1 Add basic gRpc server as frontend 1.2 Compatible with Spark Connect protocol, see https://github.com/apache/spark/blob/master/connector/connect/common/src/main/protobuf/spark/connect/base.proto 1.3 Support ExecutePlan 1.4 Support AnalyzePlan 1.5 Support Config 1.6 Support AddArtifacts 1.7 Support ArtifactsStatus 1.8 Support Interrupt 1.9 Support ReattachExecute 1.10 Support ReleaseExecute 1.11 Serialize the protobuf based request

2. Add a new Spark Connect backend 2.1 Imprort Sprak-Connect-Server and rewrite SparkConnectService https://github.com/apache/spark/blob/master/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectServer.scala 2.2 Deserialize response to protobuf based

3. Add IT

4. Add docs

Additional context

Introduction of https://github.com/apache/kyuubi/issues/6232