Skip to content

PyTorch DataLoader processed in multiple remote computation machines for heavy data processings

License

Notifications You must be signed in to change notification settings

ildoonet/remote-dataloader

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

6 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

remote-dataloader

DataLoader processed in multiple remote computation machines for heavy data processing.

Architecture

Usage

RemoteDataLoader

total_trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform_train)
loader = RemoteDataLoader(total_trainset, batch_size=32, timeout=5)

Example.py

example.py contains a simple example to process cifar10 images using remote nodes.

$ python example.py     # run server(dataloader)
$ python remote_dataloader/worker.py --server {master_ip}:1958      # run multiple workers
$ python remote_dataloader/worker.py --server {master_ip}:1958
$ ...  

About

PyTorch DataLoader processed in multiple remote computation machines for heavy data processings

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages