Notices by disroot (disroot@social.weho.st), page 17

@gerd Yeah thats true. Though two years ago 90% of our users were asleep when we did them so they all thought its magic :P (mostly eurpoean)

Now we have to start developing no-downtime voodoo :P

** Nextcloud and Matrix downtime due to db migration **

We are about to migrate databases of matrix and nextcloud, and deply new server. We hope things will go back online asap but it might take an hour or two.

Sorry.

https://state.disroot.org/incident/101

@gerd
Yeah having global userbase is not easy.
Though i think we should stop doing the crazy middle of the night operations cause its not healthy for us and there will always be people suffering from downtime.

@gerd
We have had some issues today and decided to shutdown the server at night to try to figure out the cause. Its coming back shortly

@tdemin Cool. Its anyway time to look into improvements on XMPP front. We have been neglecting it for a while.

@toche
Yo. We finished our morning coffee and restarted nextcloud news updater process. It will probably take a while before your feed is updated, but its on the way 😃

@toche
Ahhh... good one. We havent switch on news updater which leaves as a seperate process.

One moment...

@rodarima
Backups. Always make backups 😋

There are two kinds of people. Ones that make backups and the ones that will.

cloud.disroot.org is back up.

We are very sorry it took so long. We shouldn't have try updating nextclloud to the newest version without prior throught testing.

Disroot admins are very very sorry.

@tdemin will have to look into it. Is is possible to have it as an opt-in for individual users?
I think you cant set it up on the client side.

@shevek no pain no gain.

Hokay. Searx instance is finally back (dont remember when was the last time i had to type in url to another searx instance).

Now for the nextcloud headacke.

Due to some issues while uprading nextcloud, we've quickly deployed webmail for all those who use it.
https://mail.disroot.org

Seems like we need to revert the upgrade of nextcloud, and re-test it (yes. it was a bold move form our side, but we thought we could get both the server upgrade and nextcloud update in one go).

disocurse is back online.

Server upgrade update:

Most of the things are up except for searx (working on it), discourse (working on it), nextcloud (upgrading to v13)
#disroot

Putting services up. 😃

@gerd
Hahaha just realized the same now 😃

Guys in 10 min we should be up and running.

In the datacenter, waiting for last files to sync.
#disroot #amiga #retro

https://social.weho.st/media/BnselULk729Ig8f0bnQ

Last cold data-sync in progress. We're on the way to the datacenter (again)

https://social.weho.st/media/3sxFBhkVJtO31A6hUQE

Hi guys.
We're doing it. Server upgrade is on the way. For last days we tried to commence server upgrade with minimal downtime possible. However this seems to not be possible (damn you rsync speeds) and the longer we wait the more complicated it becomes.

We therefor decided to do it now instead of spending another night in the datacenter.
Currently cloud is down, and in an hour we are going to put the entire plaform down for about 3 hours (could be longer)

Sorry for the inconvenience.