tl:dr there was a catastrophic failure at host level for the server we live on, coupled with bad backups, so this account was truly gone for some hours.
If you DM'd us in the past 4 days since last station update with music submission or response to our call for visual art please resend.
We need help to reach the ~600 friends federation has not or might not realign to us.
https://blog.radiofreefedi.net/anatomy-of-a-disappearance
If you didn't notice, sweet! The music never stopped.
anatomy of a disappearance
a community perspective on losing a federated conduit( please do not follow this account, this is for long form reading, expanded masto posts, specialty pages/posts and archive reference. Please follow and interact with us at @radiofreefedi@musician.social where we announce and link the material here)
Some time overnight RFF HQ time, whilst we were having a rest, multiple instances including musician.social where the RFF main interactive account lives quite literally disappeared.
The basic synopsis is some very bad update flagging processes on a hosting provider did some wild overwriting completely blanking the instance.
Ok, that's not irreparable. No it shouldn't be but then something else went wrong with supposed tested backups through we think another provider making them apparently borked for some time having falsely reported on manual checks that they were regular and in good health. So yeah, wombo combo, literal wipe with no rollback state. yikes.
Upon being greeted with server downs this morning and then realising the instance was redeployed with the same name but on a new server/host/id and reading that yes all data was gone (didn't exist), the panic set in that the community conduit that has rallied to lift up each other was “razed to the ground” in a few CPU cycles.
Is the fedi account RFF? no. Were any other services that RFF are in control of affected? no. The music kept playing, we just lost the primary means to communicate with each other.
RFF is all of us and let us just say how in this crazy panic time how nice it was to reach out via alternate channels to some amazing and trusted sounding boards to figure out our options, and to see many of the RFF community help to get the word out as to what happened, how it was beyond our control and how so many were ready to help, organise and communicate as needed. Awesome!
So we scrambled to whiteboard:
-If our original instance had come back in a new install/location with the same name, what would be the ramifications getting back on the same name for existing relationship at all, asap, or after time if any relationships degrade based on other's federation rules?
-Where could we go to start fresh with reasonable and safe fediquette server federation that would still maximise our ability to see and support artists as far and wide as possible balanced with also not resource pummeling a small resource or hobby instance.
-How feasible to push for even more community support to standup a single user RFF instance with the irony that we could not reach the community. We were ready to burn some of the community infra fund if needed to act quick on this.
-And who was lined up to help with any of these options.
We are so thankful that we had friends ready to hear this frantic white boarding and help us with any and all of these options to make sure RFF didn't go away and/or came back stronger.
Was this demoralising, heck yes, was it also affirming of the kick ass folk that support this community, ALSO HECK YES. Did your hamster in a wheel need the stress while still working in alternative manners to get station stuff done after our local lappy failure, heck no.
Where we are at:Since the original instance name was brought back on a new host so quickly (credit to the admin), we decided on balance to risk any paradoxes, collisions or federated weirdness of putting our same account name on the new version of the instance and see if any followers restored.
There are still some server settling in, federation, images, etc issues to calm.
All old posts are gone! Whereas you can still see them to some level as posts stored out on federated servers, they are now dead links if you try to browse to original.
We have plenty of our own backups locally and out here on the blog to recreate important pinned posts and things going forward. Just be advised all older posts are effectively gone as federated copies degrade.
We will have lost any music submission DMs from the past 4 or so days that had not been finalised into a station update so please re-send if that was you.
We have lost a few follows from the past week or two. We have used the RFF follow file to get back many but that also may need some time and/or manual sleuthing to get back to where it was.
We sadly also lost any DMs from perspective visual artists from the recent post asking for contributors for backing imagery for the video channel so we will resend that when things settle.
We have been testing and workshopping how to be able to side channel communicate with the RFF contributor community and this incident has galvanised that need, not just for discussing new content and station initiatives but also mobilising for other things like today's event in a less public than fedi posts and more tidy than lost in hundreds of DMs.
A surprising ~80% of station friends have popped up again as followers as federation reconnected with the new server.
However, we are still missing at the time of posting about 600 friends from previous followers. No idea how many are lost in queues, need more time, or may never automatically line back up. Just us posting going forward and you all interacting should help get more of these accounts aligned.
**So we could use your help getting the word out that folk may need to un and/or re follow @radiofreefedi@musician.social **
Sorry for the disappearance especially after we'd recently been struggling to keep up with station initiatives and updates due to personal gear and life troubles.
Let's get back to the music and as always how to listen, contribute, help is at:https://radiofreefedi.net
With your continued support we hope to keep showing how kick ass it is to support fedi artists!
Keep lifting each other up and stay awesome out there friends.
Kia ora rawa atu
Kia kaha
