I just spun up Lemmy on my Kubernetes cluster with nginx-unprivileged and ingress-nginx. All is well so far! I’m thinking about posting the Kustomization manifests and continuing to maintain and publish OCI’s per version release of Lemmy.

  • dudeami0@lemmy.dudeami.win
    link
    fedilink
    English
    arrow-up
    7
    ·
    1 year ago

    I currently am running the instance I am responding from on kubernetes. I published a helm chart, and others are working on them too. I feel being able to quickly deploy a kubernetes instance will help a lot of smaller instances pop up, and eventually be a good method of handling larger instances once horizontal scaling is figured out.

      • Andreas@feddit.dk
        link
        fedilink
        English
        arrow-up
        4
        ·
        1 year ago

        Saved this comment. It claims that the Lemmy frontend and backend are stateless and can be scaled arbitrarily, as can the web server. The media server (pict-rs) and Postgres database are the limitations to scaling. I’m working to deploy Lemmy with external object storage to solve media storage scaling and there’s probably some database experts figuring out Postgres optimization and scaling as well. None of the instances are big enough to run into serious issues with vertical scaling yet, so this won’t be a problem for a while.

        • blazarious@mylem.me
          link
          fedilink
          English
          arrow-up
          2
          ·
          1 year ago

          I’ve got my pictrs backed by an S3, so that should scale well.

          I had some issues with the image server, though, and I had multiple of them running at the same time at some point, so that may have been the cause.

      • dudeami0@lemmy.dudeami.win
        link
        fedilink
        English
        arrow-up
        2
        ·
        1 year ago

        I’m not sure if there really is issues, I think it’s just new ground since most lemmy instances have been able to run on a single node due to the low populations. It seems most large public instances are just adding bigger servers to deal with the problem short term.

        From what I can tell (I am not an expert in this field), it seems most of the architecture would spread horizontally without much issue. I haven’t seen anywhere this is done yet, but I could be missing the obvious.

        The lemmy backend api just takes HTTP requests (and at the present websockets, but this is changing in 0.18 to only HTTP requests), and it uses postgres as the backend storage. Using a kubernetes postgres operator to scale the database and then running multiple lemmy backend api instances (and frontend as needed) seems like it would work, or would require minimal work to get running.

        • tj111@lemmy.fmhy.ml
          link
          fedilink
          English
          arrow-up
          1
          ·
          1 year ago

          Thanks for the input, yeah just rest apis over a postgres db is pretty standard k8s setup, so unless there’s something weird in the middle it would work fine. Curious why the OG design leaned into sockets over rest though, that’s an interesting choice.

    • anthr76@lemmy.kutara.ioOP
      link
      fedilink
      English
      arrow-up
      3
      ·
      1 year ago

      Very neat! I also considering writing a helm-chart with my close friend’s amazing helm library. In the end I decided against it since this is a pretty simple deployment as of today. Tomorrow I will clean up the Kustomize manifests and some CI with a non-federated config file and post it :)

  • Dax87@forum.stellarcastle.net
    link
    fedilink
    English
    arrow-up
    6
    ·
    1 year ago

    I run mine in a microk8s cluster.

    I live in an RV and I’m using T-Mobile 5g internet, so I have to deal with double-nat / cgnat networking.

    To overcome this, I’ve rented a vps with nginx on it, and have zero-tier on both machines for some vpn-like reverse proxying. Works very very well. I use this method for my other services too, like Plex.

    • gabe565@lemmy.cook.gg
      link
      fedilink
      English
      arrow-up
      2
      ·
      edit-2
      1 year ago

      Yep I’m still working on a helm chart. Currently, each service is deployed with the bjw-s app-template helm chart, but I’d like to combine it all into a single chart.

      The hardest part was getting ingress-nginx to pass ActivityPub requests to the backend, but we settled on a hack that seems to work well. We had to add the following configuration snippet to the frontend’s ingress annotations:

      nginx.ingress.kubernetes.io/configuration-snippet: |
        if ($http_accept = "application/activity+json") {
          set $proxy_upstream_name "lemmy-lemmy-8536";
        }
        if ($http_accept = "application/ld+json; profile=\"https://www.w3.org/ns/activitystreams\"") {
          set $proxy_upstream_name "lemmy-lemmy-8536";
        }
        if ($request_method = POST) {
          set $proxy_upstream_name "lemmy-lemmy-8536";
        }
      

      The value of the variable is $NAMESPACE-$SERVICE-$PORT.
      I tested this pretty thoroughly and haven’t been able to break it so far, but please let me know if anybody has a better solution!

      • anthr76@lemmy.kutara.ioOP
        link
        fedilink
        English
        arrow-up
        2
        ·
        edit-2
        1 year ago

        Firstly, awesome to hear you’re using bjw-s app-template helm chart. He’s my good friend and former coworker :)

        I’m also doing what @seang96@exploding-heads.com is doing.

        While I don’t consider this completed yet I have posted how I’m doing things so far here

        • gabe565@lemmy.cook.gg
          link
          fedilink
          English
          arrow-up
          3
          ·
          edit-2
          1 year ago

          That’s awesome! I love his Helm chart. It’s the most impressive Helm library I’ve ever seen. I maintain a bunch of charts and I exclusively use his library chart :)

          I just mentioned in a response to @seang96@exploding-heads.com, but I feel like deploying a separate nginx is probably cleaner, I just didn’t want another SPOF that I could break at some point in the future.

      • seang96@exploding-heads.com
        link
        fedilink
        English
        arrow-up
        1
        ·
        edit-2
        1 year ago

        Doing this made / path with Lemmy ui break, but posts and comments were actually updating according to logs. i set it up with a nginx container behind my ingress now and it appears everything works besides my comments and posts not being federated even though I can curl the links for troubleshooting federation without issue. Got any ideas?

        • gabe565@lemmy.cook.gg
          link
          fedilink
          English
          arrow-up
          2
          ·
          1 year ago

          Hmm I’m not sure! That code snippet should only affect routing conditionally. When you added the configuration snippet, did your ingress logs show the requests to / going to the frontend or backend?

          An nginx container behind ingress seems cleaner, I just didn’t want to add another point that I could possibly break lol

          • seang96@exploding-heads.com
            link
            fedilink
            English
            arrow-up
            2
            ·
            1 year ago

            I just found out my posts are finally going through without any changes using the nginx proxy container to nginx ingress method! If you do have a way to do it all directly through nginx I’d love to see how it’s all done, maybe I was missing something outside of the snippet you posted.

            Since it’s currently working I’ll look into spinning a test instance up when I get a chance and play with that ingress annotation.

            • gabe565@lemmy.cook.gg
              link
              fedilink
              English
              arrow-up
              2
              ·
              1 year ago

              Awesome! A separate nginx container is fine, so if it’s working I’d probably leave it. I’ll look through and see if there’s anything I missed in my comment though for brevity.

  • magus@l.tta.wtf
    link
    fedilink
    English
    arrow-up
    2
    ·
    1 year ago

    👋 I’m not using Kustomize, just throwing Deployment manifests and such at the cluster manually. Works pretty nicely, though I had some trouble setting up the custom nginx stuff to proxy stuff in - I ended up running a new nginx instance and pointing the Ingress at that rather than the Lemmy pods directly. Maybe there’s a more elegant solution I’m missing?