Hello all!
First of all, thank you for the help I have already received! I see the light in the end of the tunnel… i just hope it’s not a train!
So, my Lemmy server IS up and running, but the lemmy-ui seems to be incorrectly configured in nginx.
The setup:
lemmy.mindoki.com is redirected with an A redirect to my static IP, where my ISP-Router box forwards port 80 and 443 over TCP to my Lemmy PC.
In the Lemmy PC is a fresh Linux mint with a Lemmy Docker install.
.
Some things works:
http://0.0.0.0:1236/pictrs/image/730840b6-d6ec-4a40-8668-36b89c6c1d33.png => Gets the image
https://lemmy.mindoki.com/pictrs/image/730840b6-d6ec-4a40-8668-36b89c6c1d33.png => Gets the image
curl “https://lemmy.mindoki.com/api/v3/community/list?sort=Hot&Limit=1” => Works
.
Some don’t:
https://0.0.0.0/ => Server error
https://lemmy.mindoki.com/ => Server error
.
RoundSparrow (I don’t know how to link users, sorry!) helped me out (here: https://lemmy.ml/comment/1715961) a bunch and suggested trying a smarthphone client, and lo and behold, it Works!
So it seems, as RoundSparrow figured out, my nginx configuration is not correct comming to routing ssh to my lemmy-ui docker.
I have spent the evening tinkering like a cave-man but to no avail, so I’m asking you knowledgeable people out there for help, and any help greatly appreciated!
Thanks again! .
Here are my config files: (I changed pwd & keys with 'redacted_…)
.
This is my main nginx.conf
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 768;
# multi_accept on;
}
http {
limit_req_zone $binary_remote_addr zone=127.0.0.1_ratelimit:10m rate=1r/s;
server {
listen 81;
server_name lemmy.mindoki.com;
return 200 "Bonjour, mon ami!\n";
}
# Redirect http requests to the https version
server {
listen 80;
listen [::]:80;
server_name lemmy.mindoki.com;
# Hide nginx version
server_tokens off;
location /.well-known/acme-challenge/ {
root /var/www/certbot;
}
location / {
return 301 https://$host$request_uri;
}
}
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name lemmy.mindoki.com;
ssl_certificate /etc/letsencrypt/live/lemmy.mindoki.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/lemmy.mindoki.com/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers on;
ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
ssl_session_timeout 10m;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets on;
ssl_stapling on;
ssl_stapling_verify on;
# Hide nginx version
server_tokens off;
# Enable compression for JS/CSS/HTML bundle, for improved client load times.
# It might be nice to compress JSON, but leaving that out to protect against potential
# compression+encryption information leak attacks like BREACH.
gzip on;
gzip_types text/css application/javascript image/svg+xml;
gzip_vary on;
# Various content security headers
add_header Referrer-Policy "same-origin";
add_header X-Content-Type-Options "nosniff";
add_header X-Frame-Options "DENY";
add_header X-XSS-Protection "1; mode=block";
location / {
proxy_pass http://0.0.0.0:1236;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
access_log /var/log/nginx/access.log combined;
}
.
.
nginx_internal.conf
worker_processes auto;
events {
worker_connections 1024;
}
http {
# We construct a string consistent of the "request method" and "http accept header"
# and then apply soem ~simply regexp matches to that combination to decide on the
# HTTP upstream we should proxy the request to.
#
# Example strings:
#
# "GET:application/activity+json"
# "GET:text/html"
# "POST:application/activity+json"
#
# You can see some basic match tests in this regex101 matching this configuration
# https://regex101.com/r/vwMJNc/1
#
# Learn more about nginx maps here http://nginx.org/en/docs/http/ngx_http_map_module.html
map "$request_method:$http_accept" $proxpass {
# If no explicit matches exists below, send traffic to lemmy-ui
default "http://lemmy-ui";
# GET/HEAD requests that accepts ActivityPub or Linked Data JSON should go to lemmy.
#
# These requests are used by Mastodon and other fediverse instances to look up profile information,
# discover site information and so on.
"~^(?:GET|HEAD):.*?application\/(?:activity|ld)\+json" "http://lemmy";
# All non-GET/HEAD requests should go to lemmy
#
# Rather than calling out POST, PUT, DELETE, PATCH, CONNECT and all the verbs manually
# we simply negate the GET|HEAD pattern from above and accept all possibly $http_accept values
"~^(?!(GET|HEAD)).*:" "http://lemmy";
}
upstream lemmy {
# this needs to map to the lemmy (server) docker service hostname
server "lemmy:8536";
}
upstream lemmy-ui {
# this needs to map to the lemmy-ui docker service hostname
server "lemmy-ui:1234";
}
server {
# this is the port inside docker, not the public one yet
listen 1236;
listen 8536;
# change if needed, this is facing the public web
server_name 127.0.0.1;
server_tokens off;
gzip on;
gzip_types text/css application/javascript image/svg+xml;
gzip_vary on;
# Upload limit, relevant for pictrs
client_max_body_size 20M;
add_header X-Frame-Options SAMEORIGIN;
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";
# Send actual client IP upstream
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# frontend general requests
location / {
proxy_pass $proxpass;
rewrite ^(.+)/+$ $1 permanent;
}
# security.txt
location = /.well-known/security.txt {
proxy_pass "http://lemmy-ui";
}
# backend
location ~ ^/(api|pictrs|feeds|nodeinfo|.well-known) {
proxy_pass "http://lemmy";
# proxy common stuff
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}
.
.
The lemmy.hjson
{
# for more info about the config, check out the documentation
# https://join-lemmy.org/docs/en/administration/configuration.html
database: {
#host: 127.0.0.1
#password: "redacted_password"
#uri: "postgres://lemmy:5432?pass=redacted_password"
# NOTE: uri passwords etc is now configured in: /media/fediverse/Storage/lemmy/.bashrc
# password to connect to postgres
password: "redacted_password"
# host where postgres is running
host: "postgres"
}
hostname: 127.0.0.1
#hostname: lemmy.mindoki.com
#bind: "127.0.0.1"
federation: {
enabled: true
}
pictrs: {
url: "http://pictrs:8080/"
api_key: "redacted_key"
}
email: {
smtp_server: "postfix:25"
smtp_from_address: "[email protected]"
tls_type: "none"
}
}
.
.
and finally the docker_compose.yml
version: "3.7"
x-logging: &default-logging
driver: "json-file"
options:
max-size: "50m"
max-file: "4"
services:
proxy:
image: nginx:1-alpine
ports:
# actual and only port facing any connection from outside
# Note, change the left number if port 1236 is already in use on your system
# You could use port 80 if you won't use a reverse proxy
- "1236:8536"
volumes:
- ./nginx_internal.conf:/etc/nginx/nginx.conf:ro,Z
restart: always
logging: *default-logging
depends_on:
- pictrs
- lemmy-ui
lemmy:
image: dessalines/lemmy:0.18.2
hostname: lemmy
restart: always
logging: *default-logging
environment:
- RUST_LOG="warn"
volumes:
- ./lemmy.hjson:/config/config.hjson:Z
depends_on:
- postgres
- pictrs
lemmy-ui:
image: dessalines/lemmy-ui:0.18.2
environment:
- LEMMY_UI_LEMMY_INTERNAL_HOST=lemmy:8536
- LEMMY_UI_LEMMY_EXTERNAL_HOST=127.0.0.1
#lemmy.mindoki.com
- LEMMY_UI_HTTPS=true
volumes:
- ./volumes/lemmy-ui/extra_themes:/app/extra_themes
depends_on:
- lemmy
restart: always
logging: *default-logging
pictrs:
image: asonix/pictrs:0.4.0-rc.7
# this needs to match the pictrs url in lemmy.hjson
hostname: pictrs
# we can set options to pictrs like this, here we set max. image size and forced format for conversion
# entrypoint: /sbin/tini -- /usr/local/bin/pict-rs -p /mnt -m 4 --image-format webp
environment:
- PICTRS_OPENTELEMETRY_URL=http://otel:4137
- PICTRS__API_KEY=redacted_key
- RUST_LOG=debug
- RUST_BACKTRACE=full
- PICTRS__MEDIA__VIDEO_CODEC=vp9
- PICTRS__MEDIA__GIF__MAX_WIDTH=256
- PICTRS__MEDIA__GIF__MAX_HEIGHT=256
- PICTRS__MEDIA__GIF__MAX_AREA=65536
- PICTRS__MEDIA__GIF__MAX_FRAME_COUNT=400
user: 991:991
volumes:
- ./volumes/pictrs:/mnt:Z
restart: always
logging: *default-logging
deploy:
resources:
limits:
memory: 690m
postgres:
image: postgres:15-alpine
hostname: postgres
environment:
- POSTGRES_USER=lemmy
- POSTGRES_PASSWORD=redacted_password
- POSTGRES_DB=lemmy
volumes:
- ./volumes/postgres:/var/lib/postgresql/data:Z
- ./customPostgresql.conf:/etc/postgresql.conf
restart: always
logging: *default-logging
postfix:
image: mwader/postfix-relay
environment:
- POSTFIX_myhostname=127.0.0.1
restart: "always"
logging: *default-logging
http://0.0.0.0:1236/pictrs/image/730840b6-d6ec-4a40-8668-36b89c6c1d33.png
proxy_pass http://0.0.0.0:1236;
0.0.0.0 is not a valid IP and thus not a valid URL. 0.0.0.0 is a special value you can use as a listen IP to listen on all IPv4 local interface addresses (:: is the IPv6 equivalent which is the short form for the IPv6 address that is all zeroes; that one will listen on both IPv4 and IPv6 local interface addresses so you should prefer it 99% of the time in 2023) but you can not use it for an actual connection.
Well spotted, I misremembered it as the IPv6 localhost.
So the config file is okay (for proxy_pass http://0.0.0.0:1236;) but I shouldn’t hit it up on https://0.0.0.0:1236 right?
No, proxy_pass is the URL for the backend, that should be a valid IP, it is okay to use it in listen, where you have [::]:80 (the extra square brackets are just to distinguish the colon between IP and port from the internal ones in the IPv6 address).
Tried to change to:
proxy_pass http://127.0.0.1:1236
but it didn’t change anything.
Any other idea about what I’m doing wrong?
Should I have conf in both the nginx.conf file and the nginx_internal.conf file? Does it matter where it is declared (as long as the docker specific things stays in the nginx_internat.conf file)?
More info:
I logged in as the owner on a mobile app, and most things work okay, but upvoting another users post on my site gave an error:
proxy_1 | 172.18.0.1 - - [20/Jul/2023:15:56:07 +0000] "POST /api/v3/post/like HTTP/1.1" 200 569 "-" "Jerboa" proxy_1 | 172.18.0.1 - - [20/Jul/2023:15:56:07 +0000] "POST /api/v3/post/like HTTP/1.1" 200 569 "-" "Jerboa" proxy_1 | 172.18.0.1 - - [20/Jul/2023:15:56:09 +0000] "POST /api/v3/post/like HTTP/1.1" 200 568 "-" "Jerboa" proxy_1 | 172.18.0.1 - - [20/Jul/2023:15:56:16 +0000] "POST /api/v3/comment/like HTTP/1.1" 200 889 "-" "Jerboa" lemmy_1 | 2023-07-20T15:56:16.441551Z WARN lemmy_server::root_span_builder: Domains do not match lemmy_1 | 0: lemmy_server::root_span_builder::HTTP request lemmy_1 | with http.method=POST http.scheme="http" http.host=lemmy.mindoki.com http.target=/inbox otel.kind="server" request_id=0b06a8ae-3676-430a-a3c7-4008e748b1e9 lemmy_1 | at src/root_span_builder.rs:16 lemmy_1 | LemmyError { message: None, inner: Domains do not match, context: SpanTrace [{ target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"http\" http.host=lemmy.mindoki.com http.target=/inbox otel.kind=\"server\" request_id=0b06a8ae-3676-430a-a3c7-4008e748b1e9", file: "src/root_span_builder.rs", line: 16 }] } proxy_1 | 172.18.0.1 - - [20/Jul/2023:15:56:16 +0000] "POST /inbox HTTP/1.1" 400 20 "-" "Lemmy/0.18.2; +https://127.0.0.1"
Does this error spread some light about what is wrong? I’m thinking especially of the https://127.0.0.1 for which I don’t have a SSL certificate (I only have one for lemmy.mindoki.com).
I’ll try to hunt down that 127.0.07 and change it to lemmy.mindoki.com, but If you know the answer I’m all ears :-)
Okay so I “found” it, but I suppose there is an underlying problem, but well it works.
See that last commented line?
It shuts off the default lemmy-ui at all times… It’s even in the comment! I wonder how that could ever have worked?
So now I just have to find out if I need to forward the POST, PUT, DELETE, PATCH, CONNECT to lemmy …
Thank you all for your patience :-)
map "$request_method:$http_accept" $proxpass { # If no explicit matches exists below, send traffic to lemmy-ui default "http://lemmy-ui"; # GET/HEAD requests that accepts ActivityPub or Linked Data JSON should go to lemmy. # # These requests are used by Mastodon and other fediverse instances to look up profile information, # discover site information and so on. "~^(?:GET|HEAD):.*?application\/(?:activity|ld)\+json" "http://lemmy"; # All non-GET/HEAD requests should go to lemmy # # Rather than calling out POST, PUT, DELETE, PATCH, CONNECT and all the verbs manually # we simply negate the GET|HEAD pattern from above and accept all possibly $http_accept values #"~^(?!(GET|HEAD)).*:" "http://lemmy"; }
As I said last night, I don’t know docker very well, and 7 weeks ago when I first tried to install Lemmy, I couldn’t get docker install to work without problems, so I went with from-scratch install (which also gave me trouble, but a few of us figured it out).
I assume you are following https://join-lemmy.org/docs/administration/install_docker.html
Now, if I understand this, it is creating 3 different docker containers, one for nginx, one for lemmy-ui. Can someone who knows the docker setup chime in? Can’t we just nuke/delete the nginx one and lemmy-ui one - and install them from fresh?
P.S. No big deal, but it is “SSL”, and not “SSH”, you can edit post titles on Lemmy!
Hey thank you! Edited :-)
Yep, I’m following the official docker install.
Yes part of my confusion is how all these nginx conf files works together. I tried to replace the nginx.conf (but I couldn’t find some ‘default’ one, which seems logic as it potentially manages other things too, I tried to paste the ansible one in there, but maybe some pass through is the only thing needed) and the nginx_internal.conf is like the one recommended.
That said, it there are lots of places where you can put 127.0.0.1, lemmy.mindoki.com or localhost and so on. I potentially mixed something up there too…
Edit: Lemmy didn’t accept the first post so reposting.
Edit edit: yes I’m all with you, if someone has a working set of config files that would be Great!
I think you need to implement this in a stream. Try this: https://iyzico.engineering/proxy-ssh-using-tls-sni-and-nginx-8a18f93f709
Well thank you but it doesn’t really help me out with, for what I have figured out, connecting the server 443 in nginx to the lemmy-ui docker. Sweating