Fuuuuuu ... Thank God for Fox self-hosting.

[tsimmons]

https://about.gitlab.com/2017/02/01/gitlab-dot-com-database-incident/

Ugh.

[sleeper_service]

fox, god is your sysadmin now?

[tsimmons]

God is everyone's admin.

[fox]

yeah what a ride for that poor guy lol

[tsimmons]

At 2017/01/31 11pm-ish UTC, team-member-1 thinks that perhaps pg_basebackup is refusing to work due to the PostgreSQL data directory being present (despite being empty), decides to remove the directory. After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com.

Been there, done that, only not at that scale.

[JustAMacUser]

On a positive note, it's nice that he manually took a snapshot so only six hours of data was lost.

[sleeper_service]

God is everyone's admin.

which god?

[fox]

tbh i would assume a company like that would at least use ansible or puppet or whatever instead of running shit on random servers via login shell