Fuuuuuu ... Thank God for Fox self-hosting.

tsimmons
Bear Rating Trainee
Bear Rating Trainee
Posts: 17
Joined: 06 Nov 2013, 20:46

Fuuuuuu ... Thank God for Fox self-hosting.

Postby tsimmons » 02 Feb 2017, 22:34

[url]https://about.gitlab.com/2017/02/01/gitlab-dot-com-database-incident/[/url]

Ugh.

User avatar
sleeper_service
Bear Rating Overlord
Bear Rating Overlord
Posts: 884
Joined: 30 Mar 2013, 23:50
Location: Dallas, Texas

Re: Fuuuuuu ... Thank God for Fox self-hosting.

Postby sleeper_service » 02 Feb 2017, 22:55

fox, god is your sysadmin now?

tsimmons
Bear Rating Trainee
Bear Rating Trainee
Posts: 17
Joined: 06 Nov 2013, 20:46

Re: Fuuuuuu ... Thank God for Fox self-hosting.

Postby tsimmons » 02 Feb 2017, 23:16

God is everyone's admin. ;-)

User avatar
fox
^ me reading your posts ^
Posts: 6318
Joined: 27 Aug 2005, 22:53
Location: Saint-Petersburg, Russia
Contact:

Re: Fuuuuuu ... Thank God for Fox self-hosting.

Postby fox » 02 Feb 2017, 23:26

yeah what a ride for that poor guy lol

tsimmons
Bear Rating Trainee
Bear Rating Trainee
Posts: 17
Joined: 06 Nov 2013, 20:46

Re: Fuuuuuu ... Thank God for Fox self-hosting.

Postby tsimmons » 02 Feb 2017, 23:33

[quote]At 2017/01/31 11pm-ish UTC, team-member-1 thinks that perhaps pg_basebackup is refusing to work due to the PostgreSQL data directory being present (despite being empty), decides to remove the directory. After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com.


Been there, done that, only not at that scale.