About volume sampler

osteology · September 1, 2020, 8:12pm

Hi, I have a question about VolumeSampler in fastmri.data.volume_sampler.py.
We can see the following code in VolumeSampler.
‘’’
need to send equal number of samples to each process - take the max
self.num_samples = max([len(l) for l in indices])

add extra samples to match num_samples
indices = indices + indices[: self.num_samples - len(indices)]
‘’’
Is this process for matching num_samples necessary? It seems to me that VolumeSampler is also performed well without this additional code.

mmuckley · September 2, 2020, 4:04pm

Hello @osteology, this code was inserted per this GitHub issue: https://github.com/facebookresearch/fastMRI/issues/48.

Some users have encountered problems here, and I think it’s a well-known issue with NCCL. Note that the update matches the PyTorch DistributedSampler implementation:

https://pytorch.org/docs/stable/_modules/torch/utils/data/distributed.html#DistributedSampler

If you’re using the MriModule PyTorch Lightning module, the extra slices should be handled here:

github.com

facebookresearch/fastMRI/blob/d72390a07e99eab3929c3982738adc0cfde07b98/fastmri/mri_module.py#L208-L211


# need to check for duplicate slices
if slice_ind not in [s for (s, _) in outputs[int(fname)]]:
    outputs[int(fname)].append((int(slice_ind), val_log["output"][i]))
    targets[int(fname)].append((int(slice_ind), val_log["target"][i]))