Question to attention computation

Open
#944 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Documentation
Clarity
Needs clarification
Activity status
Stale
Tech stack
python
Domain
machine-learning

Research direction

Start with inference/huggingface/zero_inference/model-support.md, especially section 5, and investigate the implementation details of the CPU attention computation referenced there. Determine whether a GPU-versus-CPU latency or throughput comparison and a detailed explanation can be added to answer the question.

Written by the indexing model from the issue text.

Description

Hi, thank you for the amazing demo and doc! I have a question regarding this section in zero-inference. It is mentioned that "Thus, our current implementation computes attention scores on CPU". May I ask if there is a detailed comparison of the latency or throughput between GPU-attention and CPU-attention to support this desicion? I am also serious about the implementation detail of the CPU-attention computation. Thank you!

Dominant language
Python
Stars
6.8k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
1

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from deepspeedai/DeepSpeedExamples

All issues in deepspeedai/DeepSpeedExamples

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.