FedML-AI/FedML

How to change data transform on built-in dataset?

Open

#1 115 ouverte le 15 juin 2023

Voir sur GitHub
 (0 commentaires) (0 réactions) (0 assignés)Python (743 forks)batch import
help wantedquestion

Métriques du dépôt

Stars
 (3 930 stars)
Métriques de merge PR
 (Aucune PR mergée en 30 j)

Description

I've read how you implement cifar10 in

  • fedml/data/data_loader.py : Entry point of loading dataset
  • fedml/data/cifar10/efficient_loader.py : loading cifar10 efficiently

I'm wondering if it is possible to change _data_transforms to a customized one? Can I configure it in fedml_config.yaml?

Since I want to perform federated transfer learning (FTL) from the ImageNet-pretrained model, which have been trained at different resolution (224x224).

Thanks!

def _data_transforms_cifar10():
    CIFAR_MEAN = [0.49139968, 0.48215827, 0.44653124]
    CIFAR_STD = [0.24703233, 0.24348505, 0.26158768]

    train_transform = transforms.Compose(
        [
            transforms.ToPILImage(),
            transforms.RandomCrop(32, padding=4),
            transforms.RandomHorizontalFlip(),
            transforms.ToTensor(),
            transforms.Normalize(CIFAR_MEAN, CIFAR_STD),
        ]
    )

    train_transform.transforms.append(Cutout(16))

    valid_transform = transforms.Compose(
        [
            transforms.ToTensor(),
            transforms.Normalize(CIFAR_MEAN, CIFAR_STD),
        ]
    )

    return train_transform, valid_transform


def load_cifar10_data(datadir):
    train_transform, test_transform = _data_transforms_cifar10()

    cifar10_train_ds = CIFAR10_truncated(
        datadir, train=True, download=True, transform=train_transform
    )
    cifar10_test_ds = CIFAR10_truncated(
        datadir, train=False, download=True, transform=test_transform
    )

    X_train, y_train = cifar10_train_ds.data, cifar10_train_ds.target
    X_test, y_test = cifar10_test_ds.data, cifar10_test_ds.target

    return (X_train, y_train, X_test, y_test)

Guide contributeur