On July 10th an unknown user managed to find an exploit involving unsanitized input strings, meaning the user was able to inject JavaScript directly into emoji text. This was a bug that was part of the Lemmy distribution, not our code, but our instance in particular was targeted due to the fact that we have a lot of custom emojis. Users who viewed affected comments in a web browser (not apps) were able to be hacked, but as far as we can tell none of our admins nor members were affected.

As soon as we learned of the issue, I took the instance offline and discussed the bug with other admins, and a patch was created which was then applied to our instance. This patch had a compatibility issue and broke the site. @[email protected] and I decided to wipe the site and restore from a recent backup.

Unfortunately, the Lemmy-recommended backup system seems to have had an error and resulted in all of the backups being corrupted, aside from a few posts and comments. This means nearly everything was lost, including user accounts.

To make matters worse, even upon restarting with a clean slate, federation is now broken for our domain, and other servers are unable to view our content despite us being able to view theirs.

I have spent countless hours over the last few days, trying to fix the server and recover the data, but it seems like doing so is impossible. We apologize for lost accounts. I’m pretty sure I can recover all comments that were on the instance, however the images (memes) are also lost.

We are trying to figure out where to go from here, likely starting a new Lemmy with a new domain to fix the federation issues. We are open to suggestions and apologize for this series of disasters.