Baldurs Gate 3 had a similar issue that they had to fix. Performance by Act 3 would be super bad and things would load very slowly or crash. Turns out it was due to all the NPCs judging every single crime you’ve committed over your entire playthrough.

I thought it was really neat.