Benchmarking against PyTorch & jit.compile

Open
#1,136 4 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Stale
Tech stack
csharp, pytorch
Domain
machine-learning, performance

Research direction

Start with the benchmark results in this issue and the linked discussion at dotnet/TorchSharp/discussions/1126. Compare the reported TorchSharp and PyTorch CUDA timings, including tf32 and bf16, and investigate the mentioned jit.compile and TorchScript behavior. Done should be defined as a concrete, reproducible performance or feature-support improvement, but the issue does not identify files or tests.

Written by the indexing model from the issue text.

Description

Discussed in https://github.com/dotnet/TorchSharp/discussions/1126

Originally posted by pkese October 28, 2023
If anyone is interested...

I made a small language model inspired by https://github.com/karpathy/nanoGPT in both PyTorch and TorchSharp.
The model has 2 layers of transformers totalling 150k parameters and is trained on Shakespeare's text.

I found out that going to smaller data types, improves training time, as does PyTorch's jit.compile, which is not available in TorchSharp.

Here are some benchmarks of model training times (minutes and seconds) with CUDA on a small GPU (RTX 3070).

default tf32 bf16
TorchSharp 0.100.7 6:46 5:20 N/A
PyTorch 2.0.1 5:31 5:27 4:28
PyTorch+jit.compile 4:04 3:57 2:26

For bf16 I used:

from torch.cuda.amp import autocast
with autocast(dtype=torch.bfloat16):
    <train code>

I couldn't achieve the same bf16 functionality with TorchSharp.

I don't quite understand why default TorchSharp code is slower than default PyTorch code.
After I set torch.backends.cuda.matmul.allow_tf32 = true in both Python and TorchSharp, I get comparable performance (see first vs second column of results).

If someone is interested I can publish the code.
(I was trying to also get TorchScript models to work on both sides which messed up the code quite a bit ... and I might wish to reverse that.)
BTW, TorchScript model was 1% slower to train on PyTorch and crashed in TorchSharp.

Dominant language
C#
Stars
1.9k
Forks
228
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from dotnet/TorchSharp

All issues in dotnet/TorchSharp

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.