You just have to be careful. set your irc client to not automatically allow DCC connections to you. also do not click on links that are spammed as they can be trojans or virii. run an Antivirus and a firewall. basically just be careful and cautious.
I must agree its pretty much safe, but watch out on what IRC script your using. As i say script i mean MIRC, but dressed up with options. Some of them contain backdoors and such. If you use the regular MIRC i see no reason its not safe, but always disable autoget.