To add the the PVFS2 Roll to the nodes of your cluster, you'll need to reinstall the nodes.
To reinstall all your nodes, as root execute:
# rocks run host /boot/kickstart/cluster-kickstart