apache/kyuubi

[TASK][CHALLENGE] Offline GPT backend for Chat engine

Open

#4.555 aberto em 18 de mar. de 2023

Ver no GitHub
 (3 comments) (0 reactions) (0 assignees)Scala (2.332 stars) (996 forks)batch import
help wanted

Description

Code of Conduct

Search before asking

  • I have searched in the issues and found no similar issues.

Describe the subtask

Currently, the Kyuubi supports Chat engine by invoking the ChatGPT online open API

Client => Kyuubi Server => Kyuubi Chat engine (invoke ChatGPT REST API)

But, is there any chance we can add a built-in offline ChatGPT engine? The answer is YES.

Client => Kyuubi Server => Kyuubi Chat engine (do prediction using local GPT model)

There is a project https://github.com/karpathy/nanoGPT which can train the GPT-2 model.

... currently the file train.py reproduces GPT-2 (124M) on OpenWebText, running on a single 8XA100 40GB node in about 4 days of training.

So, the basic idea is, training a GPT-like model, and Kyuubi Chat engine invoking this model to answer the question.

There are some specific questions:

  • Given that Kyuubi is a kind of SQL gateway, we may want it to be smart in SQL area, then how to choose the training DataSet?
  • karpathy/nanoGPT written in PyTorch, how can Kyuubi Chat engine invoke it? The options may be
    • use PyTorch Java SDK(does such thing exist?) to load model and Kyuubi Chat engine simply calls function to do prediction
    • launch a PyTorch serving service which expose RESTful/gRPC/thrift api, and Kyuubi Chat engine using RPC call for prediction

Also, there is another interesting project https://github.com/ggerganov/llama.cpp

Parent issue

https://github.com/apache/kyuubi/issues/4549

Are you willing to submit PR?

  • Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
  • No. I cannot submit a PR at this time.

Guia do colaborador

Pilha de tecnologia
pythonpytorchscala
Domain
backendmachine learningai
Tipo Issue
feature
DifficultyDificuldade de implementação estimada para um novo contribuidor, de 1 para alterações muito pequenas a 5 para trabalho de nível especializado.
5
Tempo estimadoUm intervalo de tempo aproximado para um colaborador experiente investigar, implementar, testar e preparar um pull request.
over 1 week
Status da atividadeQuão disponível o issue aparece agora: novo, ativo, obsoleto, bloqueado ou aguardando entrada do mantenedor.
fresh
ClarityCom que clareza o issue explica a mudança esperada, os critérios de aceitação e a próxima etapa.
mostly clear
Prerequisites
PyTorchScalaMachine learning basicsKyuubi concepts
Simpatia para novatosUma pontuação de 1 a 100 que estima o quão acessível este issue é para colaboradores iniciantes.
10
Direção de pesquisa
Investigate how to train a GPT 2 model using nanoGPT (karpathy/nanoGPT) on a SQL related dataset. Explore options for integrating the model into Kyuubi's Chat engine: either using PyTorch Java SDK to load the model directly, or deploying a PyTorch serving service with RESTful/gRPC API. Also consider llama.cpp as an alternative for C++ based inference. Refer to parent issue #4549 for context. There are 3 comments on the issue that may provide additional guidance.