I can speak from experience in saying these are virtually impossible to remove without breaking (at least on the fixed rear seat saloon, as you have no access from the back). The issue is, the end of these inserts is barbed, so once fitted, the base flares out within the cylindrical tube they slide into, which is metal. Audi I think have an official tool to remove them, which might help, if you can find one. Levering up on them from the top is very likely to break the flared top section too, ask me how I know.
The other issue is, at least on the saloon, the inserts that have the button on were £105 from Audi EACH when I last bought them, nearly as much as I paid for the whole leather interior, and because they're near impossible to remove, you won't find them on eBay second hand. I don't think Audi makes them any more, and there's limited numbers left in the UK, and for whatever reason there are none on the aftermarket.
When I upgraded my Special Edition leather seats to the Valcona seats, I actually ended up cutting 3 holes in the rear seat back so I could get access to the base of each tube. This allowed me to compress the barb with big pliers, and remove them without breaking them, which saved me around £300. It's a shame it came to that, and it may be different on the Avant.