I've recently test driven a few S3s, a standard 210 and 225, couldn't really tell the difference between the 2 to be honest, then drove an MTM tuned 210 and a superchipped 210, way way better, seriously drive a mapped one, should be around 265 bhp, the difference is really noticeable, tranforms the car. Don't knock it till you've tried it!
However I also used to own an 330bhp EVO 6, you won't match that for shear power and grip, awesome car, same with a Impreza P1 I drove, different league, however with those you lose out with build quality, rattley, plasticey, cheap feeling, don't get that with an Audi, so the impression of speed seems subdued, so much more comfy and quiet, don't rely just on what it feels like, your prob used to that big hit of turbo lag, watch the speedo it'll tell you different.
I felt the same when I was looking but the more you drive them the more you realise there's not much in it, I eventually bought an S4 (my mate bought the superchipped S3! couldn't be out done there!) and now loving it, build quality is 2nd to none, comfortable, quiet, (I must be getting old!)smooth as f@#k! Delivers the power in a totally different way, but it's there, believe me!
Insurance was shed loads cheaper too, £1100 for my evo, £860 on my skyline, £500 on my S4, £400 on an S3!
In short I see your reservations, but drive more of them, different specs, for longer. I wouldn't go back.