Page 1 of 1

Fuuuuuu ... Thank God for Fox self-hosting.

Posted: 02 Feb 2017, 22:34
by tsimmons

Re: Fuuuuuu ... Thank God for Fox self-hosting.

Posted: 02 Feb 2017, 22:55
by sleeper_service
fox, god is your sysadmin now?

Re: Fuuuuuu ... Thank God for Fox self-hosting.

Posted: 02 Feb 2017, 23:16
by tsimmons
God is everyone's admin. ;-)

Re: Fuuuuuu ... Thank God for Fox self-hosting.

Posted: 02 Feb 2017, 23:26
by fox
yeah what a ride for that poor guy lol

Re: Fuuuuuu ... Thank God for Fox self-hosting.

Posted: 02 Feb 2017, 23:33
by tsimmons
At 2017/01/31 11pm-ish UTC, team-member-1 thinks that perhaps pg_basebackup is refusing to work due to the PostgreSQL data directory being present (despite being empty), decides to remove the directory. After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com.


Been there, done that, only not at that scale.

Re: Fuuuuuu ... Thank God for Fox self-hosting.

Posted: 02 Feb 2017, 23:34
by JustAMacUser
On a positive note, it's nice that he manually took a snapshot so only six hours of data was lost.

Re: Fuuuuuu ... Thank God for Fox self-hosting.

Posted: 02 Feb 2017, 23:38
by sleeper_service
tsimmons wrote:God is everyone's admin. ;-)

which god?

Re: Fuuuuuu ... Thank God for Fox self-hosting.

Posted: 02 Feb 2017, 23:41
by fox
tbh i would assume a company like that would at least use ansible or puppet or whatever instead of running shit on random servers via login shell