Nice! Good thing I backed up my DB. I’ll try changing the network settings and restoring my DB.

That did not fix my instance. I had to manually set the site’s image icon to null in the database to get my site working again. There are still many issues in both lemmy-ui and lemmy be logs.

Is your “external” NGINX proxy running on the same host server as Lemmy?

Originally posted by madeofstown in #1530

I’m using the docker containers. No additional nginx running on the host.

Originally posted by slash-zero in #1530

Note: pictrs logs show no errors.

When trying to access the image directly at https://SITE_URL_REDACTED/pictrs/image/a29da3fc-b6ce-4e59-82b0-1a9c94f8faed.webp, it works fine in a browser.

The issue appears to be that lemmy-ui is trying to load the image from 127.0.1.1:443.

ECONNREFUSED 127.0.1.1:443

Originally posted by slash-zero in #1530

Thanks! I just left that there for posterity, I added an edit to the top of my comment about how I “fixed” this.

It was caused due to a network error, so to fix it I just gave the container internet access, even though it otherwise wouldn’t need it. That fixed things immediately.

Still, network errors shouldn’t be fatal. Especially not for an icon.

Originally posted by ubergeek77 in #1530

My server is now completely inoperable and I can’t downgrade to 0.17.4 due to the database upgrades. Anything I can do to not lose my data?

@ubergeek77

short term fix is to remove your site’s image icon.

db query: update site set icon = null where id = 1;

then at least your site will work again, sort of. It loads at least. I am not sure federation is working 100% though and the pictrs errors still persist in the logs.

Originally posted by slash-zero in #1530

@slash-zero “It’s always DNS”

Originally posted by madeofstown in #1530

My server is now completely inoperable and I can’t downgrade to 0.17.4 due to the database upgrades. Anything I can do to not lose my data?

@ubergeek77

short term fix is to remove your site’s image icon.

db query: update site set icon = null where id = 1;

then at least your site will work again, sort of. It loads at least. I am not sure federation is working 100% though and the pictrs errors still persist in the logs.

For an instance set up with Ansible: Connect with ssh and cd /srv/lemmy/<domain>. Run docker-compose down and look for the name of your postgres container (should be something like lemmyml_postgres_1) as it shuts down. Run docker-compose up -d and wait for all the containers to start back up. Run docker exec -it <postgres-container> "bash" to connect to your postgres database container. Run psql -U lemmy to connect to the database with the lemmy user. Run update site set icon = null where id = 1; then \q then exit

Originally posted by madeofstown in #1530

UPDATE:

Issue “fixed” by putting the lemmy-ui container on a network with access to the internet. It looks like lemmy-ui didn’t previously need an internet connection on 0.17.x, but now it needs one. Probably because it used to use the internal docker network to get to pictrs, but now it’s trying to resolve using the real public hostname?

In OP’s case, it’s going to a local IP instead, but it’s using the wrong port. In both our cases, the URL it uses for pictrs is not calculated correctly for whatever reason.

The UI could fail to contact pictrs for any variety of reasons, but that shouldn’t be fatal. It should just break images.

Previous comment text:

---

I get a very similar error but it’s not quite the same:

 FetchError: request to https://<my-instance>/pictrs/image/<uuid>.png failed, reason: getaddrinfo EAI_AGAIN <my-instance>
     at ClientRequest.<anonymous> (/app/node_modules/node-fetch/lib/index.js:1505:11)
     at ClientRequest.emit (node:events:511:28)
     at TLSSocket.socketErrorListener (node:_http_client:495:9)
     at TLSSocket.emit (node:events:511:28)
     at emitErrorNT (node:internal/streams/destroy:151:8)
     at emitErrorCloseNT (node:internal/streams/destroy:116:3)
     at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
   type: 'system',
   errno: 'EAI_AGAIN',
   code: 'EAI_AGAIN'
 }

Just like OP, I can go to the URL that error is logging and it pulls the image just fine. No browser cache shenanigans, the request is being fully completed.

My server is now completely inoperable and I can’t downgrade to 0.17.4 due to the database upgrades. Anything I can do to not lose my data?

Originally posted by ubergeek77 in #1530

I think this is happening because there is not any error handling when the icon is fetched:

https://github.com/LemmyNet/lemmy-ui/blob/9e7fec772d873e7bc46a51403d9569f70619c9b8/src/server/utils/create-ssr-html.tsx#L28

https://github.com/LemmyNet/lemmy-ui/blob/main/src/server/utils/fetch-icon-png.ts

So if that call fails for any reason, it breaks the SSR render.

I think this should be changed to return a broken image icon, or a blank image, instead of aborting the entire SSR process. Otherwise, the pictrs service going down for any reason causes the entire UI to crash like this.

That goes for any other fetches for external data too. If it’s not mission critical (and images aren’t mission critical), the SSR would ideally continue the render.

Originally posted by ubergeek77 in #1530

I found a more permanent fix. The error seems to be caused by having the host name of your server set to the same thing as your lemmy instance domain-name. The easy and more permanent fix is to change the host name of your server!

Originally posted by madeofstown in #1530

I can confirm that this issue is still present in Lemmy-ui 0.18.1-rc.2. Every single time the docker images restart I have to run the commands to delete my site logo or the site will be unreachable. I think this may be partially because my external NGINX proxy is on a different different host. When Lemmy-ui resolves the IP for my domain it is still using the local IP and port 443.

If believe the fix would be to use the port specified by the pictures server in the internal NGINX configuration. I’m not aware of how to do this. I will test if this can be accomplished by configuring the pictures server to listen on 443 in the internal configuration.

Originally posted by madeofstown in #1530

Put pictrs and lemmy-ui on the external network if you used the ansible script. That should have been updated now though.

https://github.com/LemmyNet/lemmy-ansible/issues/96#event-9619766061

have a look here - try following the steps to remove the site icon via the database

Originally posted by Hellhound1 in #1530

Nice! Good thing I backed up my DB. I’ll try changing the network settings and restoring my DB.

That did not fix my instance. I had to manually set the site’s image icon to null in the database to get my site working again. There are still many issues in both lemmy-ui and lemmy be logs.

Originally posted by slash-zero in #1530

@randommouse interesting. I’ll give that a try.

Originally posted by slash-zero in #1530

@randommouse:

OK, I updated to:

• lemmy-ui: 0.18.1-rc.7
• lemmy be: 0.18.1-rc.4

Then, I changed my host’s server name to not match the domain name, and rebooted. Local images were still not working after a docker-compose down and docker-compose up -d --force-rebuild. I still couldn’t upload images either, and the site banner was not working (broken image).

The final step was to revert the lemmy.hjson back to http://pictrs:8080 (from the public domain name) and restart again, and now images are working, and having an icon for my instance no longer breaks the site.

So, to clarify, what I did was:

1. Change my host name in /etc/hostname so it no longer matches my site’s domain name
2. Update lemmy.hjson to set pictrs back to the original http://pictrs:8080
3. Reboot, because why not?
4. “Restart” all lemmy containers via down, then up
5. 👍

Originally posted by slash-zero in #1530