This is still very much early-days, but I’m thinking about building a distributed link aggregator. Some notes:

  • no servers (all P2P) except to connect peers
  • no “instances” so everything is in one namespace (so like reddit /r/community, not lemmy instance/c/community)
  • everyone is a moderator - moderation based on a web of trust type system
  • desktop first, mobile/web later - web would require servers, mobile may have platform-specific issues
  • no plans to integrate with fediverse, but could be possible
  • initial intent is to be text-only - file storage could be supported later in an opt-in basis (for now, just link to an external img host), but I’m worried about disk storage since there’s no central data lake and everyone needs to chip in some storage space

Some technical details:

  • written in Rust and ReactJS - Tauri-based
  • uses Iroh for data synchronization
  • looking into Appleseed and maybe TrustNet for moderation

Current status:

  • rewrote Appleseed and part of TrustNet to Rust for perf testing last weekend (Rust is waaay faster) - will probably rewrite again once I finish reading the paper (current code is a direct port)
  • have basic Tauri+Iroh app, but it’s a glorified chat app w/ no syncing between peers (a peer can join a chat, but will lose access if the host goes down)
  • reading source code for SimpleX (re-learning Haskell too) - could be interesting for DMs or something

I’m not sure how long I’ll work on this, but I’ll probably release some libraries under a relatively liberal license, so something like LGPL at the most stringent, but probably Apache/MIT (depends on if I need to clean-room the appleseed implementation).

So, if there’s interest here, I’m happy to post updates for discussion periodically.

  • sugar_in_your_teaOPEnglish
    3·
    9 months ago

    I have submitted some patches here and there to Lemmy when something bothers me, this is just an alternative in case it takes off (unlikely) and has issues scaling. There’s a ton of duplication of data, and that adds up, and that’s hard to square without a profit model.

    So that’s the main problem I’m trying to solve here, and I think it’s fundamental to how the fediverse is designed. That, and it’s annoying having to care about instances, which also is a UX issue. I think I can mitigate a lot of the UX issues by hosting a few mirrors to improve availability, and hopefully the community steps up to spread out that cost. It should be a lot cheaper than Lemmy because there’s less duplication, and it should be temporary (hobbyists will hopefully run large nodes, kind of like how Tor operates).

    But you’re right, it’s a pretty big risk because it’s a lot of work for something that might end up being worse. It’s an attractive option with internet speeds getting better, but I know IPFS is kind of a crappy experience and Iroh is unstable and unproven, so it’s a gamble. The theoretical underpinnings look good (better DHT, less bloat, etc), and maybe pieces could be reused in something like Lemmy, like the moderation engine, so I think it’s worth the effort to play with.

    Thanks for the feedback, if you have specific criticisms and potential solutions of p2p apps, I’m interested. I’ve been interested in p2p for years, and this seemed like a good excuse to play around with it.

    • Lung@lemmy.worldEnglish
      1·
      9 months ago

      Thanks - yeah, I think any app that’s not capable of being a simple website is DoA. I’ve seen many attempts at building something in this style, and none of them have really taken off. Social media like diaspora and marketplaces like lighthouse. The closest is BitTorrent. I don’t consider instances to be any more confusing than email domains. Having your phone running a background service would definitely be the worst outcome

      If the final result is that peers need to be online for their data to be accessed, and it’s not replicated around much, then it’s no different than server-client. IPFS is definitely a tragedy when it comes to trying to use it, and it has blacklisting built in. Reality: there is a small cartel of hosts formed, and the effort to run your own instance is way too high when you could just use cloudflare’s. I consider IPFS to be a fully failed experiment that didn’t even take off in crypto

      Again, my suggestion is that you take your passion and obvious brilliance, and apply it to greatly improving products that already have traction. If you have ways to make moderation better and network efficiency higher, then you’d be directly impacting the lives of tens of thousands of Lemmy users, or whatever software you chose to aid. One of these cases where practicality beats ideal design. Unless you really just want to experiment and learn some stuff for your own sake

      • sugar_in_your_teaOPEnglish
        2·
        9 months ago

        simple website

        WebRTC might be sufficient, just like it is for libp2p.

        My app is all in React using Tauri, so I would just need to replace a few calls to a local library with webRTC support, which should be pretty easy.

        phone running a background service

        That would be 100% opt-in. I would even allow disabling sync over data, so you could still post and whatnot offline, and it would sync with peers later.

        But that’s also why I’m doing desktop first, I don’t need you worry about limitations while I’m prototyping it.

        peers need to be online for their data to be accessed

        I intend to have enough duplication that it shouldn’t be an issue.

        To start, I also intend to have always online “clients” that, combined, have “all” (as much as any p2p app can) of the data until enough people use it to have it self-host.

        apply it to greatly improving products that already have traction

        That’s usually what I do anyway. I really don’t want to head a project, but, as you mentioned, other stuff in this space seems to have failed:

        • IPFS - too slow, Iroh improves that
        • Hyphanet (OG freenet) - focused on “web pages” and anonymity, which attracted the wrong people
        • new freenet - isn’t available yet (soon!!) - once it is, I’ll play with it, but it’s libp2p-based, so probably slow
        • scuttlebutt - only small groups, I’m looking for a Reddit alt
        • aether - ephemeral in nature (6 months), and the project seems dead (no activity for 3-4 years)
        • cryptocurrency-related - almost everything else, which it attracts the wrong crowd

        I’ve contributed a bit to Lemmy, but I think lemmy is fundamentally flawed so we’ll need an alternative eventually. I hope to make a bridge to act like an instance eventually, which would at least allow a federation of p2p networks (could be cool).

        ways to make moderation better

        Better is subjective, but yeah, I’ll definitely be pulling from Lemmy datasets for simulations. However, it would probably need to be a browser extension due to the data and interaction I want.

        There are three types of trust:

        • trusted - I have proactively approved something (most web of trust systems)
        • untrusted - no productivity whatsoever
        • distrusted - I have proactively flagged something as harmful

        Lemmy mods handle the third and user votes kind of handle the first two, but if you don’t trust the mods, you’re SOL. It’s also at risk of Sybil attacks (e.g. vote brigading), so it’ll have similar issues as Reddit.

        I think I can all three types of trust with two inputs: flags and votes. Explicit trust of a user could be helpful (e.g. bots or community mods), but hopefully not required. If it significantly improves performance, I might add a “mod queue” so users can train the moderation system (e.g. do you agree with this moderation decision?).

        I think it’s an interesting area to explore, and I don’t see anything using it, which means it either:

        • doesn’t work - so I shouldn’t propose it to lemmy
        • is hard - so experimentation is valuable
        • is too expensive - probably the case (initial naïve simulation with 100k nodes took ~3s to initialize, checking is fast)

        I’m hoping people think it’s harder than it is, which is why it’s overlooked. Votes and blessed moderation are easy to implement, so I’m banking on that. There may also be a way to integrate some limited machine learning and get hardware acceleration from a GPU or AI chip (e.g. Google Pixel’s AI chip), so “expensive” may be less of an issue once I go public.

        Anyway, thanks for the thoughts! I may need to discuss web support with Iroh devs. I know they have a web demo, but I think it routes through a web server, not an actual p2p web app, but I haven’t checked.

        • Lung@lemmy.worldEnglish
          2·
          9 months ago

          Cheers bro, great to meet you and I’m sure we would get along. Hoping for your success and joy