Been there, done that, spent days trying multiple configurations, work arounds, and "fixes" (even re-installing OI from scratch), but nothing worked. And I'm not the only one with that problem of not being able to enable Jumbo Frames in a virtualized OpenIndiana on ESXi, as the posts I kept on finding the configurations, work arounds, and "fixes", were the post that started with the same "How do I enable Jumbo Frames in OI in ESXi?" type of question.
I just created a new VM, installed OI, nappit, vmtools.
vmxnet3s0 is MTU 9000. I can do the following:
ping -s <ESX Host IP> 9000 4
And it pings with no issue.
Something specific you were trying to do that I can test? Do I need to create and run a VM to test the NFS?