apache/kyuubi

[TASK][CHALLENGE] Offline GPT backend for Chat engine

Open

#4,555 opened on Mar 18, 2023

View on GitHub
 (3 comments) (0 reactions) (0 assignees)Scala (2,332 stars) (996 forks)batch import
help wanted

Description

Code of Conduct

Search before asking

  • I have searched in the issues and found no similar issues.

Describe the subtask

Currently, the Kyuubi supports Chat engine by invoking the ChatGPT online open API

Client => Kyuubi Server => Kyuubi Chat engine (invoke ChatGPT REST API)

But, is there any chance we can add a built-in offline ChatGPT engine? The answer is YES.

Client => Kyuubi Server => Kyuubi Chat engine (do prediction using local GPT model)

There is a project https://github.com/karpathy/nanoGPT which can train the GPT-2 model.

... currently the file train.py reproduces GPT-2 (124M) on OpenWebText, running on a single 8XA100 40GB node in about 4 days of training.

So, the basic idea is, training a GPT-like model, and Kyuubi Chat engine invoking this model to answer the question.

There are some specific questions:

  • Given that Kyuubi is a kind of SQL gateway, we may want it to be smart in SQL area, then how to choose the training DataSet?
  • karpathy/nanoGPT written in PyTorch, how can Kyuubi Chat engine invoke it? The options may be
    • use PyTorch Java SDK(does such thing exist?) to load model and Kyuubi Chat engine simply calls function to do prediction
    • launch a PyTorch serving service which expose RESTful/gRPC/thrift api, and Kyuubi Chat engine using RPC call for prediction

Also, there is another interesting project https://github.com/ggerganov/llama.cpp

Parent issue

https://github.com/apache/kyuubi/issues/4549

Are you willing to submit PR?

  • Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
  • No. I cannot submit a PR at this time.

Contributor guide

Tech stack
pythonpytorchscala
Domain
backendmachine learningai
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
5
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
PyTorchScalaMachine learning basicsKyuubi concepts
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
10
Research direction
Investigate how to train a GPT 2 model using nanoGPT (karpathy/nanoGPT) on a SQL related dataset. Explore options for integrating the model into Kyuubi's Chat engine: either using PyTorch Java SDK to load the model directly, or deploying a PyTorch serving service with RESTful/gRPC API. Also consider llama.cpp as an alternative for C++ based inference. Refer to parent issue #4549 for context. There are 3 comments on the issue that may provide additional guidance.