Hello sh.itjust.works community,

Many of you have been eager to get an update about when the sh.itjust.works instance will get it’s upgrade to the latest version of lemmy. Here’s a update along with a tentative timeline.

In December 2023 I purchased a new server for this community. It took me awhile but I eventually made the time to get it racked at the local datacenter. For the sysadmins lingering and those interested here are the specs:

  • Dual Xeon 2.9Ghz CPUs (32 cores total)
  • 256GB ram
  • 4 x 1TB SSD in raid 10 (with room to add 6 more disks)
  • 10gbit networking

While I’m ready to proceed with the upgrade, I’ve decided to first migrate this instance over to the new hardware. Here are two reasons.

  1. Those of you who have been around long enough may remember that I’ve been running this instance on “borrowed” unused resources that were available at the time. There are no more resources available for this instance to grow.
  2. There are reports that the latest version of lemmy may use more resources. Given we are among the bigger instances, should I end up in a situation where I need to increase resources to keep things fast I’ll be restricted.

Here’s the tentative timeline:

Task Date Expected Downtime
Migration to new server Tuesday February 27 2024 @ 8:00PM ET 90 Minutes
Upgrade to V19.3 Thursday February 29 2024 @ 8:00PM ET Up to 120 Minutes
  • If anything major goes wrong on the 27th I will revert back the changes and bring the instance back up on the current server.
  • If anything major goes wrong on the 29th I will revert back using an earlier snapshot. If that fails, I will restore from a backup.

During these two planned events those who want to provide moral support or who want to get periodic updates are more than welcome to join us on our matrix channel

=========================================================
Update February 29 2024
We’ve successfully completed the upgrade to v1.9.3. I’m happy to announce that we did it in an astonishing 27 minutes, a whole 93 minutes under what was expected. The extra leg work that was done over the last few weeks combined with the better hardware definitely played a part. Looking over the processes, it looks like the service responsible for images is still doing some work so it’s possible that you will come across some broken images. I’ll be keeping on eye on that over the next bit and make adjustments if needed. Thank you all for the support and to all of you who kept me company on our matrix channel. Have a good evening.

=========================================================
Update February 27 2024
We’ve successfully completed the migration. I’m happy to announce that this instance is now running on its new hardware dedicated solely to this community! We experienced just under 40 minutes of downtime which is a whole 50 minutes less than expected. Please give this instance a chance to catch up what it missed but we should be good within the next 30 or so minutes. Thank you

  • idefix
    link
    fedilink
    English
    arrow-up
    100
    ·
    9 months ago

    So you bought a server for us? I really need to contribute financially

    • loutr
      link
      fedilink
      English
      arrow-up
      32
      ·
      9 months ago

      Is there a way to donate yet?

      • can
        link
        fedilink
        English
        arrow-up
        11
        arrow-down
        1
        ·
        9 months ago

        Not as of yet. Leaving me no real avenue to support Lemmy. Does anyone know if there any alternative collectives of developers working on the main Lemmy code I could donate to?

  • Vertelleus
    link
    fedilink
    English
    arrow-up
    70
    ·
    9 months ago

    Every time I see SJW, I think “Social Justice Warrior” not “Shit Just Works.”

    Omnissiah’s blessing on the upgrade @TheDude!

    • Classy
      link
      fedilink
      English
      arrow-up
      28
      ·
      9 months ago

      TIL this instance is not “Shhh, it just works!” and it is in fact “Shit just works”

      • threelonmusketeers
        link
        fedilink
        English
        arrow-up
        18
        ·
        9 months ago

        I love your alternate interpretation. There are few things in my life which work but I’m not sure how, and I don’t want to investigate too much for fear that I break them.

      • can
        link
        fedilink
        English
        arrow-up
        11
        ·
        9 months ago

        You’re just too classy

    • p1mrx
      link
      fedilink
      English
      arrow-up
      25
      ·
      9 months ago

      I would suggest shijw as an abbreviation.

    • UNWILLING_PARTICIPANT
      link
      fedilink
      English
      arrow-up
      20
      ·
      9 months ago

      I like it. I’m just waiting for someone to bring it up as a mark against our instance somehow

      • Night Monkey
        link
        fedilink
        English
        arrow-up
        19
        arrow-down
        1
        ·
        9 months ago

        I’ve already been accused of being a troll alt right Nazi for using this instance. Weird

          • SolOrion
            link
            fedilink
            English
            arrow-up
            5
            ·
            9 months ago

            I swear, the average Hexbear user would call anyone right of Marx himself an alt right Nazi.

            • CancerMancer
              link
              fedilink
              English
              arrow-up
              5
              ·
              edit-2
              9 months ago

              So long as you support anything resembling free speech, you will be a Nazi to them. It doesn’t matter that 20 years ago, lefties were being actively suppressed in North America using the same weapons they are using to supress the populists today. It doesn’t matter if you’re on your union local’s executive and advocate for socialist policy and democratic reform. It doesn’t matter if you have a past of creating actual material change for the exploited.

              You let someone say what Hexbear considers bad words, that means you’re a Nazi.

            • MomoTimeToDie
              link
              fedilink
              English
              arrow-up
              5
              arrow-down
              2
              ·
              9 months ago

              Hell, probably even a few people to his left as well

      • Cracks_InTheWalls
        link
        fedilink
        English
        arrow-up
        15
        ·
        edit-2
        9 months ago

        Does anyone else feel strangely old reading this comment, despite the fact that sh.itjust.works has only been around for less than a year?

        “I was there, when the wall came down and the hexbears came”

        (Obligatory “sh.it.heads for life”; “up with sh.it.heads, down with shitheads”; and so on)

        • imaqtpieMA
          link
          fedilink
          English
          arrow-up
          13
          ·
          edit-2
          9 months ago

          Yeah same. Makes me feel warm inside to know that there are some sh.itheads who never got caught up in the various internet slapfights that have unfolded over the past year.

          I mean forget about hexbear, OG sh.itheads will remember the exploding-heads defederation drama. I’ll never forget when these users absolutely eviscerated me regarding our “glacial” response to defederating EH.

          https://sh.itjust.works/comment/512007

          Oof

          Your “oh maybe, I kinda guess it’s possible” attitude here is the problem. You are acting now, great, but this was an incredibly slow response, it doesn’t matter if your vote is tomorrow. It’s already incredibly, mind bogglingly slow, and you still decided to post a day long debate thread before the vote (it seems to show overwhelming support for defederation, which I consider totally unsurprising, and makes me wonder again what you’ve been waiting for).

          I understand a little about why it was so glacial, but you continue to appear to not understand that this was actually a huge issue, that in that time you let the server bias towards the alt-right, showed a general support for alt right topics, and alienated an unknowable number of users in the process. Those are all the sort of users you probably wanted to keep, the ones who aren’t just content to see things go badly and will take action when they see it rather than stare like a deer in headlights. If they’re like me, they were probably posting and trying to find out the delay before leaving in disgust. They’re folks that care about having an instance that reflects their values, and you’ve let them - us - down.

          That can’t be undone, but the way you keep shrugging it off as though you really don’t see it does not inspire confidence.


          Even rereading it now, that one still hurts. Thanks for not turning into a bunch of Nazis, that would have been a lot of guilt for me to work through 😅

          • Cracks_InTheWalls
            link
            fedilink
            English
            arrow-up
            10
            ·
            9 months ago

            Ugh, yep. Right around the time the agora system was getting fleshed out, too, so things just naturally took a bit longer.

            Related - “What’s a Beehaw?” - newer sh.it.heads, probably.

            • imaqtpieMA
              link
              fedilink
              English
              arrow-up
              10
              ·
              edit-2
              9 months ago

              Beehaw was so worried about other instances turning into Nazi bars even as their own server was rapidly bleeding users due to excessively strict moderation and defederation.

              Here’s a fun snapshot of the Lemmy network as of June 15th, 2023. Beehaw was huge in the early days.

              image

              • SolOrion
                link
                fedilink
                English
                arrow-up
                5
                ·
                9 months ago

                Oh, wow. I didn’t realize beehaw dropped off that hard. I thought about going there in the early days, but decided on here instead purely because the signup seemed like less of a hassle. I’m also kinda surprised our user count here isn’t even 5k. I guess that’s federation at work.

              • assaultpotato
                link
                fedilink
                English
                arrow-up
                5
                ·
                9 months ago

                Wait did beehaw fall off? I have two accounts, one here and one on beehaw. I actually made my sh.itjust.works account because beehaw defederated from some instances with communities I wanted to keep and now I use both semi regularly. Mostly lurk though.

                • imaqtpieMA
                  link
                  fedilink
                  English
                  arrow-up
                  6
                  ·
                  9 months ago

                  Yeah they’re down to 600 MAUs. They are apparently planning to move off Lemmy to another platform. I’m not sure why

          • can
            link
            fedilink
            English
            arrow-up
            3
            ·
            9 months ago

            I shudder to think what I might have missed those first few days.

          • CancerMancer
            link
            fedilink
            English
            arrow-up
            2
            ·
            9 months ago

            Man one more day of exposure to a handful of alt-right topics and I would have become a Nazi for sure!

          • MomoTimeToDie
            link
            fedilink
            English
            arrow-up
            4
            arrow-down
            3
            ·
            9 months ago

            Lmao I remember that thread. It was when I realized that this site was hardly better than reddit in terms of people just wanting everything they don’t like censored. I made a dozen or so accounts on other instances to get around defeds as a result. I’m just glad that I can use a proper multi-account app instead of trying to manage everything manually these days

      • can
        link
        fedilink
        English
        arrow-up
        4
        ·
        9 months ago

        The automod actions always make me smirk.

          • [email protected]A
            link
            fedilink
            English
            arrow-up
            7
            ·
            edit-2
            9 months ago

            Sadly, no.
            Like imaqtpie mentioned, I’ve cobbled together something, because otherwise we’d be drowning in garbage/spam.
            Hopefully it doesn’t become sentient enough to rampage on its own.

            There’s the recent spam wave, but there’s a lot of other day-to-day junk.
            Compare, say…
            https://sh.itjust.works/c/[email protected]
            With a random instance’s version:

            Or even with its kbin source: https://kbin.social/m/opensource

            (Unrelated, but federation seems currently broken between that kbin magazine and lemmy in general… but then it’s 99% spam down in there lately)

              • [email protected]A
                link
                fedilink
                English
                arrow-up
                2
                ·
                9 months ago

                I’ve since removed one or two kbin magazine that are just completely overrun with spam.
                [email protected] is one of them.
                It’s gone locally, although I guess one can still browse that mess directly over there: https://kbin.social/m/internet.
                I couldn’t even find a legit post among the garbage so I removed it locally altogether.
                If they ever get it together we can restore it then.

          • imaqtpieMA
            link
            fedilink
            English
            arrow-up
            4
            ·
            9 months ago

            Nah, he’s talking about @SJW_Bot2. Automod functionality isn’t built into Lemmy, InEnduringGrowStrong actually created and maintains that bot to help combat spammers and trolls.

            It would have been impossible to protect this instance from the recent spam wave without that bot, as you can see by scrolling through the modlog to see all the random shit it automatically removes.

    • eestileib
      link
      fedilink
      English
      arrow-up
      9
      ·
      9 months ago

      I’m happy to be affiliated with both so it works for me.

    • Goldmage263
      link
      fedilink
      English
      arrow-up
      7
      ·
      9 months ago

      We should have a Dude Day every year for the anniversary of this instance. I’ve been very happy with their handling of a group of strongly and diversely opinionated interweb users.

      Omnissiah’s blessing indeed.

  • JohnWorks
    link
    fedilink
    English
    arrow-up
    55
    arrow-down
    1
    ·
    9 months ago

    Some reason it never crossed my mind that someone could run their own instance on their own hardware and not like a cloud provider. That’s super cool!

    • brbposting
      link
      fedilink
      English
      arrow-up
      26
      ·
      9 months ago

      Big D Energy

      (Dude)

      (Decentralized)

      (inDependent)

  • merc
    link
    fedilink
    English
    arrow-up
    32
    ·
    9 months ago

    Thanks for running this place, and thanks for the info.

  • DannyBoy
    link
    fedilink
    English
    arrow-up
    28
    ·
    9 months ago

    What do you expect me to do for that 90 and 120 minutes??

  • willya@lemmyf.uk
    link
    fedilink
    English
    arrow-up
    27
    ·
    9 months ago

    Good luck. Yours brought me to Lemmy and then I made my own soon after.

    • can
      link
      fedilink
      English
      arrow-up
      15
      ·
      9 months ago

      With a url like that how could you not?

    • Cracks_InTheWalls
      link
      fedilink
      English
      arrow-up
      15
      ·
      edit-2
      9 months ago

      Godspeed, f.ukwad.

      (Hmm…not as cute as sh.it.head. I still kinda like it, though. Excellent domain name choice in general.)

  • can
    link
    fedilink
    English
    arrow-up
    23
    ·
    9 months ago

    Thank you for all your time and effort.

  • Lucidlethargy
    link
    fedilink
    English
    arrow-up
    22
    ·
    9 months ago

    Thank you for everything you are doing. Take a few days if you have to fit this migration and update. I’ve hosted many servers… I always feel I need to do so much for everyone. Curb that thought, if you can. We genuinely appreciate what you’re doing here.

    • Socsa
      link
      fedilink
      English
      arrow-up
      16
      arrow-down
      1
      ·
      9 months ago

      I love the idea of taking the phrase back. There’s nothing shameful about social justice.

  • minibyte
    link
    fedilink
    English
    arrow-up
    18
    ·
    edit-2
    9 months ago

    I appreciate everything you do for this instance. Also, thanks for posting the specs of the new build. It’s pretty neat to see what’s going on behind the curtain.