New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sharding of data for Horovod distributed training for SSD and YOLOv3 #1610

Open

karan6181 wants to merge 2 commits into dmlc:master from karan6181:fix_training_script

Contributor

karan6181 commented Feb 10, 2021

Data was not sharded across GPUs when running Horovod distributed Training. This PR will fix that issue.
Fixed the issue with the validation condition where it was doing validation at epoch 0 irrespective of --val-interval.
Fixed the calculation of throughput based on --log-interval.
Tested the functionality with and without horovod for first few batches.

karan6181 added 2 commits

February 10, 2021 09:44


          Sharding of data for Horovod distributed training

8be3fd1


          Fixed throughput calculation based on log_interval

github-actions bot commented Feb 10, 2021

Job PR-1610-3018577 is done.
Docs are uploaded to http://gluon-vision-staging.s3-website-us-west-2.amazonaws.com/PR-1610/3018577/index.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment