Yeah, the sentiment is fine. “I don’t think I would enjoy that, but thanks for the invite” would go over a lot better with people.