first to check, try to turn down the resizing to something like x2 (that's higher than 720p already), keep everything else the same and see if your machine could run (I bet it's ok). I have a feeling it's the 1600x1200 res. that's too much to ask for from your x2 3800 plus most likely, you don't really need to upscale to that res imho. or try the
basic setting from avsforum too. your cpu should be able to handle intermediate setting imho.
as of the mobo, in my eye (for htpc's point of view), there's not a lot of difference between two. but certainly have a couple that will mainly depends on what you going use on your htpc! imho, a good htpc mobo needs, at least eSATA (you'll never know when you'll need more space, and eSATA is fast!), good audio codec if not planning add on audio card (which I don't think is nessasary with nowaday's cpu power), DDLive, DTSConnect support?
the main difference in my eye of the two mobo is the audio function! the AW9DMax support DDLive already (no need for add on audio card) and is Dolby Master Studio certified!! can't go wrong w/ that.
on the other hand, the AB9QuadGT doesn't seem to have any DTSConnect/DDLive enabled codec. so if you plan on using DDLive/DTSConnect, you'll need to go for an add on card. however, it's nice it has HDMI audio header that can connect to you HDMI'ed video card and simplified the cabling to your HDTV/Stereo system. but that will only give your straight pass through audio too (which is fine for HD TV or DVDs, etc.) but you still won't get DDlive, etc., if that's what you'll needed (e.g. gaming!). also, when thinking to use HDMI connection to your HDTV, your HDTV needs to have optical out to pass the audio signal to your receiver too. otherwise, you'll still need to use two cable from your HTPC (1 HDMI for video to TV, 1 optical for audio to receiver).
so, the main difference is the audio part, and since you'll be using it for gaming too, and I assume you dont' have the audio card yet, then for go the AW9Max. but if you already got the audio card (remember you were looking at it), then you can go for the AB9QuadGT to save some $.