NVIDIA/DALI

ExternalSource and fn.readers.file

Open

#3,811 创建于 2022年4月11日

在 GitHub 查看
 (9 评论) (0 反应) (1 负责人)C++ (670 fork)auto 404
external sourcehelp wanted

仓库指标

Star
 (5,722 star)
PR 合并指标
 (PR 指标待抓取)

描述

  1. CodeBase1: I need to use the weighted random sampler for ImageNet1000, so, i checke this issue. link. Then, I followed this link to get my own codes, including Sampler, ExternalSourcePipeline and DALIGenericIterator.

  2. CodeBase2: Also, I check this ImageNet1000-resnet link.

However, the difference in efficiency of loading data is surprisingly large!!!

I'm not sure. The possible reason may be the difference between fn.readers.file in link and np.fromfile in link

for _ in range(self.batch_size):
    jpeg_filename, label = self.files[self.i % self.n].split(' ')
    batch.append(np.fromfile(self.images_dir + jpeg_filename, dtype = np.uint8))  # we can use numpy
    labels.append(torch.tensor([int(label)], dtype = torch.uint8)) # or PyTorch's native tensors
    self.i += 1

How should I achieve an efficiency similar to CodeBase2 with weighted random sampler.

贡献者指南