There's really a ton of variables here.
First we have network speed. Are they using a VPN client? I know I would never allow people to reach our network without a secure encrypted connection however some companies may have different ideas. (by network speed we have usually multiple NICs, so internal network and external network speeds here.)
Secondly, I/O performance. Is it purely a physical server? Now days virtualization and esxi hosts rule the roost so if it's an older physical server or a vm with little resources then it can also be an issue.
Are they all hammering the server at once? We have had dev's that run 2 -3 million line inserts on database servers that just kills the server. And hardware wise they're brand new.
Then you also have each users home network connection to worry about. I know that when I use my vpn for work ours is pretty darn good. I don't have quite the same responsiveness that I get from physically being there but It's definately useable. A lot of this can be peoples expectations.
The above poster was spot on in the questions to ask, but without more details it's hard to narrow it down.
From a networking perspective, anytime you introduce a vpn and ecryptions your adding overhead to traffic and cutting down on throughput. How many nic's does the server have? what speed? are they using fiber for I/O? how many disks, luns, etc. A lot of stuff here.
For a quick little test you could always open a command prompt and tracert the servers IP just to see what latency your getting from the network via ping.
We did have a few users with random VPN issues that for some reason or another were offset by them vpn connecting to their desktop with remote desktop, and then connecting to the server via the desktop session. I don't recommend this as it adds another hop in the traffic but for some users it helped. If it's an option it's also worth a shot as a last resort.
Also, I wouldn't complain to much and ruin the friday work from home gig! hahahahaha
The best thing to do is open a ticket (or whatever system they use to address problems).
The biggest annoyance with sys admins and tech professionals is the tendency that they always think "It's your end not ours" mentality. It's even more fun to deal with when your working between two businesses. I can't tell you how many times I've reached out to try and mutually help solve a issue and they refuse to work with me, until I provide the cold hard proof to them.
I don't know what monitoring software they use(if any who knows) but in vsphere and with opmanager and a few others you can see the servers resource usage over weeks months, etc and opening a ticket and having them check it out is a good place to start.