With forewarning about a huge influx of users, you know Lemmy.ml will go down. Even if people go to https://join-lemmy.org/instances and disperse among the great instances there, the servers will go down.

Ruqqus had this issue too. Every time there was a mass exodus from Reddit, Ruqqus would go down, and hardly reap the rewards.

Even if it’s not sustainable, just for one month, I’d like to see Lemmy.ml drastically boost their server power. If we can raise money as a community, what kind of server could we get for 100$? 500$? 1,000$?

  • kinther@lemmy.ml
    link
    fedilink
    arrow-up
    1
    ·
    1 year ago

    I’m in the process of setting up a raspberry pi to host an instance, but the documentation is not super helpful. I’ll slog through it and issue a PR once co.plete so others may not have the same issues.

  • RoundSparrow@lemmy.ml
    link
    fedilink
    arrow-up
    1
    ·
    edit-2
    1 year ago

    Based on looking at the code and the relatively small size of the data, I think there may be fundamental scaling issues with the site architecture. Software development may be far more critical than hardware at this point.

    • OsrsNeedsF2P@lemmy.mlOP
      link
      fedilink
      arrow-up
      0
      ·
      1 year ago

      What are you seeing in the code that makes it hard do scale horizontally? I’ve never looked at Lemmy before, but I’ve done the steps of (monolithic app) -> docker -> make app stateless -> Kubernetes before and as a user, I don’t necessarily see the complexity (not saying it’s not there, but wondering what specifically in the site architecture prevents this transition)

      • RoundSparrow@lemmy.ml
        link
        fedilink
        arrow-up
        1
        ·
        edit-2
        1 year ago

        Right now it looks to me like Lemmy is built all around live real-time data queries of the SQL database. This may work when there are 100 postings a day and an active posting gets 80 comments… but it likely doesn’t scale very well. You tend to have to evolve to a queue system where things like comments and votes are merged into the main database in more of a batch process (Reddit does this, you will see on their status page that comments and votes have different uptime tracking than the main website).

        On the output side, it seems ideal to have all data live and up to the very instant, but it can fall over under load surges (which may be a popular topic, not just an influx from the decline of Twitter or Reddit). To scale, you tend to have to make some compromises and reuse output. Some kind of intermediate layer such as every 10 seconds only regenerate the output page if there has been a new write (vote or comment change).

        don’t necessarily see the complexity (not saying it’s not there

        It’s the lack of complexity that’s kind of the problem. Doing direct SQL queries gets you the latest data, but it becomes a big bottleneck. Again, what might have seemed to work fine when there were only 5000 postings and 100,000 total comments in the database can start to seriously fall over when you have reached the point of accumulating 1000 times that.

          • poVoq@slrpnk.net
            link
            fedilink
            arrow-up
            0
            ·
            1 year ago

            The advantage kbin has is that it is build on a pretty well known and tested php Symphony stack. In theory Lemmy is faster due to being built in Rust, but it is much more home-grown and not as optimized yet.

            That said, kbin is also still a pretty new project that hasn’t seen much actual load, so likely some dragons linger in its codebase as well.

            • Sam BOT@slrpnk.netB
              link
              fedilink
              arrow-up
              0
              ·
              1 year ago

              I think it’s probably undesirable to end up with big instances. I think the best situation might be one instance that’s designed to scale. This could be lemmy.ml or another one. It can absorb these waves of new users.

              However it’s also designed to expire accounts after six months.

              After three months it sends users a email explaining it’s time to choose a server, it nags them to do so for a further three months. After that their ability to post is removed. They remain able to migrate their account to a new server.

              After 12 months of not logging in the account is purged.

              • andrew@radiation.party
                link
                fedilink
                arrow-up
                1
                ·
                1 year ago

                Thought on this a bit more, and I’m thinking encouraging users not to silo (and make it easy to discover instances and new communities) will probably be the best bet for scaling the network long-term.

                “Smart” rate limiting of new accounts and activity per-instance might help with this organically. If a user is told that the instance they’re posting on is too active to create a new post, or too active to accept new users, and then is given alternatives, they might not outright leave.

  • nutomic@lemmy.mlM
    link
    fedilink
    arrow-up
    1
    ·
    1 year ago

    The site currently runs on the biggest VPS which is available on OVH. Upgrading further would probably require migrating to a dedicated server, which would mean some downtime. Im not sure if its worth the trouble, anyway the site will go down sooner or later if millions of Reddit users try to join.

    • Divided by Zer0@lemmy.ml
      link
      fedilink
      arrow-up
      1
      ·
      1 year ago

      Do you have the frontend a DB serving in the same VPS? If so it would be a great time to split them. Likewise if you DB is running in a VPS, you’re likely suffering from significant steal from the hypervisor so you would benefit from switching to a dedicated box. My API calls saw a speedup of 10x just from switching from a VPS DB to a Dedicated Box DB.

      I just checked OVH VPS offers and they’re shit! Even at 70 Eur dedicated on hetzner, you would gain more than double those resources without steal. I would recommend switching your DB ASAP for immediate massive gains.

      If you’re wondering why you should listen to me, I built and run https://aihorde.net and are handling about 5K concurrent connections currently.

      • nutomic@lemmy.mlM
        link
        fedilink
        arrow-up
        1
        ·
        edit-2
        1 year ago

        Hetzner is very strict about piracy so thats not an option. And now is almost weekend so I wont have time for a migration. Anyway there are plenty of other instances in case lemmy.ml goes down.

        Edit: I also wouldnt know which size of dedicated server to choose. No matter what I pick, it will get overloaded again after a week or two.

        • Divided by Zer0@lemmy.ml
          link
          fedilink
          arrow-up
          1
          ·
          1 year ago

          Even if you choose Hetzner, it won’t even know it has anything to do with piracy because it will be just hosting the DB, and nobody will know where your DB is. That fear is overblown.

          Likewise believe me a dedicated server is night and day from a VPS.

    • Pisck@lemmy.ml
      link
      fedilink
      arrow-up
      1
      ·
      1 year ago

      There will either be an hour of downtime to migrate and grow or days of downtime to fizzle.

      I love that there’s an influx of volunteers, including SQL experts, to mitigate scaling issues for the entire fediverse but those improvements won’t be ready in time. Things are overloading already and there’s less than a week before things increase 1,000-fold, maybe more.

      • nutomic@lemmy.mlM
        link
        fedilink
        arrow-up
        0
        ·
        1 year ago

        Can we replace Lemmy.ml with Join-lemmy.org when Lemmy.ml is overloaded/down?

        I dont think so, when the site is overloaded then clients cant reach it at all.

        Does LemmyNet have any plans on being Kubernetes (or similar horizontal scaling techniques) compatible?

        It should be compatible if someone sets it up.

        • Leigh@lemmy.ml
          link
          fedilink
          arrow-up
          0
          ·
          1 year ago

          You could configure something like a Cloudflare worker to throw up a page directing users elsewhere whenever healthchecks failed.

          • nutomic@lemmy.mlM
            link
            fedilink
            arrow-up
            1
            arrow-down
            1
            ·
            1 year ago

            Then cloudflare would be able to spy on all the traffic so thats not an option.

            • Leigh@lemmy.ml
              link
              fedilink
              arrow-up
              0
              arrow-down
              2
              ·
              1 year ago

              spy on all the traffic

              That’s…not how things work. Everyone has their philosophical opinions so I won’t attempt to argue the point, but if you want to handle scale and distribution, you’re going to have to start thinking differently, otherwise you’re going to fail when load starts to really increase.

    • Ashwag@lemmy.ca
      link
      fedilink
      arrow-up
      0
      ·
      1 year ago

      So reading this correctly, it’s currently a hosting bill of 30 Euro a month?

      • Milan@discuss.tchncs.de
        link
        fedilink
        arrow-up
        0
        ·
        edit-2
        1 year ago

        No, thats the 8 GB memory option… if its the biggest, it should be around 112 €. Meanwhile i keep wondering if i should let Lemmy stay on the current KVM (which is similarely specked but with dedicated cores and stuff) or if it is better to move it to one of my dedis just in case… well… will see xD

        • nutomic@lemmy.mlM
          link
          fedilink
          arrow-up
          0
          ·
          1 year ago

          Its the one for 30 euros, Im not seeing any vps for 112. Maybe thats a different type of vps?

            • roho@lemmy.ml
              link
              fedilink
              arrow-up
              0
              ·
              edit-2
              1 year ago

              Nowadays doesn’t even make any sense to use servers. … Why not create something better?

              i think you might underestimate the problem.

              Jami.net (a decentralized messaging app) works p2p. it uses a torrent-like distributed-hashmap to locate the peers at any moment. (The main usability issue for nontechnical users, is that devices on an internal ip address aren’t addressable from outside. This requires (a single point of failure and privacy concern), a turn-server)

              They started to incorporate Git for merging chats for the reason that any of set of peers (of a group chat) can be out of reach of another set of peers, i.e. the chat continues on different branches and needs to be merged again later.(this happens in the clients-app, because there is no central server). Jami is aiming at double-digit group sizes… That’s not nearly the size of what Lemmy is handling.

              • Milan@discuss.tchncs.de
                link
                fedilink
                arrow-up
                0
                ·
                1 year ago

                It does not sound like OVHs vServers offer dedicated cores, and it is common to quickly become a bottleneck with VPS offerings across hosters and for example with the initial Mastodon hypes, i had to learn that shared hardware lesson the hard way. For the price you are currently paying, maybe something like a used dedicated (or one of the fancy AMD ones) server at Hetzner is of interest: https://www.hetzner.com/sb

                • nutomic@lemmy.mlM
                  link
                  fedilink
                  arrow-up
                  0
                  ·
                  1 year ago

                  Hetzner is great but they are very strict about piracy, so its not an option for lemmy.ml. For now the load has gone down so I will leave it like this, but a dedicated OVH server might be an option if load increases again.

    • elouboub@kbin.social
      link
      fedilink
      arrow-up
      1
      arrow-down
      1
      ·
      1 year ago

      Is it running in a single docker container or is it spread out across multiple containers? Maybe with docker-machine or kubernetes with horizontal scaling, it could absorb users without issue - well, except maybe cost. OVH has managed kubernetes.

      • Dessalines@lemmy.ml
        link
        fedilink
        arrow-up
        0
        ·
        1 year ago

        SQL. We desperately need SQL experts. It’s been just me for yeRs, and my SQL skills are pretty terrible.

        • Valmond@lemmy.ml
          link
          fedilink
          arrow-up
          1
          ·
          1 year ago

          Put the whole DB in RAM :-)

          Makes me remember optimization, lots of EXPLAIN and JOIN pain, on my old MySQL multiplayer game server lol. A shame I’m not an expert …

  • OsrsNeedsF2P@lemmy.mlOP
    link
    fedilink
    arrow-up
    0
    ·
    1 year ago

    I’m also willing to donate to other instances too - Beehaw, Sopuli, Lemmygrad, Lemmyone - Anything so we can have better shock absorption. If you run one of those instances, please reply and let us know how much you think you need

    • Ruud@lemmy.world
      link
      fedilink
      arrow-up
      0
      ·
      1 year ago

      At the moment, I run lemmy.world using the funding of mastodon.world. If Lemmy.world might grow and need a dedicated server, I’ll try to raise funds for it separately (or create a larger .world fundraiser as I have other instances as well)

      • HKayn@lemmy.ml
        link
        fedilink
        arrow-up
        0
        ·
        1 year ago

        What kind of server and which specs is lemmy.world running on? I’m planning on setting up my own instance for a small community, but I have no idea what to brace for.

        • Ruud@lemmy.world
          link
          fedilink
          arrow-up
          1
          ·
          1 year ago

          Currently a 2 cpu 4GB VPS at Hetzner, costing 5 EUR per month. With a storage volume of also 5 per month. I am monitoring this and will scale when needed. For mastodon.world we scaled it to a dedicated server with 32 cores and 256GB so we can go a long way.

          • molo@beehaw.org
            link
            fedilink
            arrow-up
            0
            ·
            1 year ago

            Can you provide any info about the number of pageviews/month or pageviews/hr that setup can support for lemmy?

            • Ruud@lemmy.world
              link
              fedilink
              arrow-up
              0
              ·
              1 year ago

              No, I have nu clue about that yet. I’ll monitor how Lemmy behaves and try to scale. Maybe after a while I can say something about it.