Inconsistent gdr read latency
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- c
- Domain
- performance
Research direction
No source file or test is named. Start by reproducing the reported 3 KB, 100-read benchmark with the stated gdr mapping, page-locked host memory, alignment, and CU_POINTER_ATTRIBUTE_SYNC_MEMOPS settings; then inspect the gdr_copy_from_bar SSE4_1 implementation mentioned in the report. Done means identifying the cause of the alternating latency and documenting or correcting the behavior.
Written by the indexing model from the issue text.
Description
I have a memory allocator based on gdr which initially allocates a large chunk of gdr memory (e.g. 16 MB) and then allocates pieces of this chunk to the subsequent memory requests. During performance benchmarking, I noticed that the read latency of the same memory size fluctuates quite significantly and I can't understand why. For example, if I allocate 3KB memory read 100 times and do the same thing again and again, the average read time fluctuates between 4.5 us and 70 us (i.e. 4.5 -> 70 -> 4.5 -> 70 ...) even though the same piece of memory is allocated for every 100 reads.
Here are some details regarding my settings:
- I have set CU_POINTER_ATTRIBUTE_SYNC_MEMOPS to 1 for the entire chunk using cuPointerSetAttribute.
- The api reports "using SSE4_1 implementation of gdr_copy_from_bar" for read operation.
- The data is read into page-locked host memory allocated using cudaMallocHost.
- Both the gdr-mapped source pointer and the destination pointer are 128-bit aligned.
- The write latency is quite consistent.
- Dominant language
- C
- Stars
- 1.4k
- Forks
- 198
- PR merge metrics
- No merged PRs in 30d
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from NVIDIA/gdrcopy
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Proposal: honor `TMPDIR` in gdrcopy package build scriptsPossibly taken @efegokdemir claimed this 53 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
Difficulty 3/5 1-2 days Newbie friendliness 58/100
Similar issues
-
SPIDevice selects an active-high device as soon as it is createdPossibly taken @mikeysklar claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
adafruit/circuitpython#11869 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 78/100
gnif/LookingGlass#1352 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
bazel-contrib/rules_go#4784 · 1 comment ·
Maintainers usually reply within 1 day