You wouldn't compare a PC with a laptop, and say "Wow, that laptop is so overpriced!", why do you compare an all-in-one with a tower PC then?
Have you ever heard of the ASUS W90 gaming series of laptops? They're not selling at newegg anymore but I've seen one with a Q9000 (quadcore), 4870 X2 Mobility graphics card, and 6GB of DDR3 RAM, with 500GB of space. For the same price as the Mac up there ($2,499).
That's an all in one feature right there. What are you going to use as an excuse now?
They're much thinner and designed to produce less heat, just like the laptops, since they're supposed to be placed on the desk and not below.
Right. Apple has to put $500 extra into their shabby products for... heat reduction. You should be able to recognize a poor investment right there (and the heat reduction factor is silly - my computer alone has better specs than that Mac I linked, and it has a very low average case temperature, which can run from 29c to 40c when I'm gaming. I only paid $800 for my computer. What does that Mac have over it? Portability. Nothing else.)
Besides that, does that $349 monitor have LED backlighting and an IPS matrix? These are essential features of it
Those are mere accessories and structural type. Does that Mac have the ability to dual-view screens for daytrading? The only important things, coming from my artistic design experience (take that however you will, I'm not claiming to be a good designer), in a monitor, are contrast ratio, resolution, and display colors. I doubt very heavily people will buy that Apple product simply because it has LED backlighting and an IPS matrix, when there are cheaper and more useful alternatives out there. I'm not saying those features should not make the product more expensive, I'm simply stating that the amount of expense for those features is too high in relation to its benefit.
In a way, it's like selling a computer case for $250 or similar extra, simply because it has 8 eSATA ports.