Employment is a mutual arrangement - never be concerned about looking suspicious. Be prepared to explain your choice honestly and stand by the actual work you’re producing. Asses-in-chairs style management is toxic and you should push back against it.