I don't see how it could be an issue, provided you do ADD the accounts you want to be vCenter admins to the administrators group before you remove Domain Admins.

I also attempted it as the Local Admin. Neither allowed me to add any domain users to the Remote Desktop access list.<BR><BR><BR>Futher info:<BR>I have tried to add Domain Users through both:<BR>1.