Alt account of poVoq@slrpnk.net here.
Our instance is currently down and I can’t get remote access to the servers. It appears that there might have been a hardware failure of the main firewall, which is the one thing I can’t work around remotely.
I am still trying a few things, but I am not very optimistic that I can get access.
The really unfortunate part is that just now I am on one of my rare work deployments abroad, so I also can’t access it physically during the next few weeks and my usual back up that could restart it is not available either.
As something like that never happened in 3 years operating the servers, I thought I can risk it, but murphy’s law seems inescapable 😓
I will try to keep you posted here on any updates, but probably there will not be much I can do for a while. Really bad timing 😥
Edit: we might use this “opportunity” to migrate the instance to Piefed, which has been an idea for quite some time now. I will keep you posted on that.
Found this post after a couple of days of hopeful reloading of the page! Took another couple of days to log in an old alt just to say I’m glad you communicated this, and let us know how we can help to make it more resilient (if at all necessary).
To the kind soul that was helping me with my greenhouse infested ants (in the unlikely event they are reading here - but iirc they were not from slrpnk.net but just passing by) I’m glad we spoke, it seems like they are harmlessly using my panels for hatching, and then moving on! Related, who has a nice gardening/greenhouse community to suggest? :)
I this it’s a law that servers run 100% perfect until the literal day one leaves town with zero way to return home. One of the many reasons I got all my services off of unraid.
Very cool to learn you’re running your own machines. Do you go into detail about this anywhere?
We have a small write up about the hardware on our wiki, but it is also down right now.
I think we will share a post-mortem write up of the actual improvements we will do to avoid this in the future.
One thing I will definitly do is to add a KVM remote management console to one of our server boards and move the main firewall into a VM with hardware passthrough of the NICs (this was anyways planned for a 10gbit network upgrade for the second half of 2025). This way I should be able to reboot and even reinstall the main ingress point remotely, so that only the fiber gateway remains as a failure point that requires physical access.
Thanks for letting us know
Thanks for the heads up! My Main account is in slrpnk, so I’ll set a reminder in an month to switch back. :)
Thanks for the heads up, try not to stress too much.
It looks like there is no enough solar power for solarpunk!
no need to show me where doors are, I will find them by myselfanyway, good luck with that!
Alt of ProdigalFrog@slrpnk.net chiming in, thanks for giving us the sitrep, poVoq!
This is an unusually unlucky set of circumstances, but for the Solarpunks who find their way here, let’s try to make the best of it. If you don’t already have an alternative account somewhere, you may want to create one to use until we’re back up and running. lemmy.dbzer0.com, sopuli.xyz, or lemmy.cafe are some solid choices to consider. After that, you may want to favorite poVoq’s post here so you can check back on any news.
Even without our home base, we can still share the solarpunky stuff we find and create around the threadiverse (though as Tofu mentioned, maybe this could be a time to take a breather to be refreshed when we come back 😄).
Besides, if the past is any indication, time will fly by and we’ll be back before you know it!
Lol love the new username, fitting! :)
Yarr, thank ye kindly, Lazy :p
Well, on the plus side lots of lessons learned and I think I might move at least the xmpp server to an external vps to have a backup communication channel.
Might it also be possible/feasible to send an “email blast” to everyone with a slrpnk.net account? It would probably be quite complicated to send that many emails at once without getting blocked by the major email providers, but hypothetically we’ve all registered with emails…
Just spitballing ideas for the future. Everything’s fine now. Gives me an excuse to spend a little more time on the Canadian end of the fediverse. :)
As email in profile is an optional feature on our instance about 50% of the userbase opted to not share one so we would only reach some people. But yes, getting spam filtered woukd be another problem of mass-mailing.
But I found a way to put up a simple html error page and will add this later to inform our members and link to this thread.
Not sure about your area, but a wireguard accessible OOB connection is a great piece of kit to keep handy. I use a cheap 768kbps SIM in an Ethernet connected switch into my personal systems. It’s saved my skin numerous times.
I’m sure this is obvious, especially in hindsight, but just mentioning because the existence of IoT LTE data plans for a minimal fee ( $100/year for me in Midwest US) was NOT obvious to me until 2 years ago.
Yeah I had plans to set up something like that, but always other priorities and in this specific case I could maybe access other internal servers but i would need KVM access to reboot the firewall or some other way to cut physical power. And exfiltrating hundreds of GBs of lemmy database wouldn’t work over such a small pipe either.
This is something that even larger corps struggle with. My old company would buy some other company, lay off a bunch of people, others would quit, and then it was shocked Pikachu faces all around when That One Thing stopped working in an office they turned into a ghost town and, well, no, nobody was going to be there until Monday morning to power cycle etc. True lights out/OOB reachability is WORK. And there’s always going to be a SPOF somewhere that requires hands on-site.
Do what you can, when you can. And thank you for all that y’all have done so far.
For that, I’d highly recommend a power conditioner with outlet controls (and preferably sequencing).
Most are pretty expensive, but you can get something like a surgex squid or a gude expert power control for a few hundred here and there, which will give you great data on power usage as well. You can also grab an older model with serial and/or relay controls, and use a pi or whatever is handy to trigger it. Actually have a setup like that in a few racks for clients (mostly orange pis and a couple of Asus tinker boards).
Hope your trip goes well!
Thanks for the update, good luck with your trip.
Thanks for letting us know! Try not to worry about it too much, it’s all good
Shit happens. Thanks for the update 🙂
Thanks for all that you do for us and this community! A little downtime won’t hurt anyone. I would also be in gavour of PieFed migration 💙
I totally agree with everything you said :)
I just created an account in PieFed and it looks very interesting because there is the option to create your own feeds! I had the wrong impression that you can only choose from the existing ones. This is so exciting!!!
That sounds pretty cool. I don’t know much about Piefed — would we still be able to access posts from Lemmy instances?
Yes, they’re 100% cross compatible. As an example, you can access !piefed_meta@piefed.social from your Lemmy instance seamlessly.
I’d probably make an account there if it moves to piefed.
Sending positive vibes, and passing offer of technical assistance once you get access again.
Hopefully it’s an easy fix once you get into things again.
Very very bad timing. I just moved to that server and now it is just gone. I have moved to another server now.
One issue in three years is poor management now? Wat
Anything less than perfection is unacceptable! 😤 /s
I signed up with them yesterday and then they go down for I don’t know how long. This is my only experience with them.
Ok, and?
That doesn’t change the fact this is a one off issue and theyve never had that issue in three years.This is a very big issue for a server no mater when it happens. A sever you can’t reset and you go on a trip ridiculous.
It usually can be reset, it’s entirely unexpected that a component that prevents that would break. You’re essentially implying that any admin who self hosts cannot ever leave their house for any length of time without being irresponsible because they cannot ensure with 100% certainly that a computer component won’t fail while their gone, or a lightning storm won’t fry something, or a flood won’t harm the servers.
Like, dude, c’mon. We’re volunteers, and poVoq is running this stuff for free. We’re not an enterprise business with multiple staff on hand to swap out failed components 24/7 so that our ‘customers’ don’t lose millions of dollars.
If anything, this only proves the strength of decentralized media, as we’re able to swap to a functioning node and continue on our merry way while the other is being repaired. Like what an incredible and resilient system considering none of this is making anyone any money.
You are acting like this is a life or death situation. I think you may need to take a break from the internet.
You’re going to want to try “my little Internet” by Fisher-Price before using this adult Internet we’re on.
Out of 3 years of reliable service, this is the one time the server has been down for any length of time, and only due to the admin not being able to be physically present for a wildly unlikely and unexpected problem.
I signed up with them yesterday and then they go down for I don’t know how long. This is my only experience with them.
I know, I’m the one who approved your application.
You happened to sign up right before we’ve had our first particularly bad issue in 3 years. Unlucky, yes, but it’s a bit out of order (IMO) to say we’re a poorly managed instance based on that, especially seeing as you lost nothing besides the time it took to write your application. :\
Does Lemmy let you retroactively deny applications?
As a matter of fact it does, actually 😅
My loss of time is small true. I will just move on. But keep in mind new users may not be as knowledgeable as myself.
Knowledgeable? Lmao
If you want a professionally managed instance, you are welcome to pay for a hosted one.
Which is exactly what I did 2 years ago. Turns out the uptime is about the same and the price is higher than 0. But on the upside, if there’s downtime I can (usually) immediately reboot the server.
The server got a little more accounts on there than I initially expected though. But I’m fine with that.
Yeah, I know. Complete Murphy’s law situation.
I appreciate the update was thoroughly confused
Thanks for letting us know what is going on.
Sorry that it took so long to get the word out. I guess we relied a bit too much on the xmpp server for internal communication.
Even the big operations break in unexpected ways. Facebook accidentally broke the authentication for their door locks when they took the site down a few years back and had to fly people to a datacenter and break down a door to get to servers and restart things.
We managed to resolve the issue and I have many ideas how to avoid this in the future 🤓
