I know just what you mean and have had that same dilema. I too have gloss black in my S3 and its a pain to keep in tip top condition, and as I've done that I wanted a change. I like the carbon but in the end decided to stick with the aluminium which I think lightens the dark cockpit and goes really well with the rotor grey I've chosen too.
Although I'm sure the carbon trims will make the car alot quick to drive
, and do differentiate the S4 interior I decided I'd rather spend the £425 on fuel, and if I came to conclusion that if I come to dislike the aluminium that much I'll get it wrapped or have it transfer dipped later.
I did have the Aluminium in my A6 an really liked it, it was very easy to look after. Thats my take on it anyway - Ultimatly the choice is yours