Hacking:Big Repo Break

On August 17th, 2011 the repo broke really bad. This are the steps we took to recover it.

1 First problem

User:Fauno discovers there were many repo-update processes hanging around and kills them. A manual update shows ~3000 packages needed sync. When that threshold is reached, User:Fauno discovers almost all packages were removed from repos and moved to old/packages, apparently by overzealous ftpdir-cleanup script.

2 Recovery attempts

Fauno: modified dbscripts.git/repo-sanity-check into repo-restore-to-normal. The scripts takes packages from abs and looks for them on old/packages and places them back into staging/$repo. Then db-update is ran, but it fails because many "duplicate" packages where staged at the same time... [to be continued]

[some time after]

Jyaworski: On the server, there are files in ~ called reposcript.sh and broken-symlinks.txt. The reposcript goes through the repo and checks for broken symlinks. Currently, these symlinks are only echoed, not removed. The output of the command is in broken-symlinks.txt.

[some time after on IRC]

<fauno> i don't get it
<fauno> i asked jyaworski to find repos' broken symlinks
<fauno> there're 23000 broken symlinks
<fauno> that means everything on repos is broken ¬¬
<fauno> i have ~15000 packages on pool
<fauno> i'm going to do this:
<fauno> * rename pool/packages to pool/restore
<fauno> * run repo-restore-to-normal from pool/restore to staging/
<fauno> only for [core]
<fauno> and see how it goes
<fauno> i also merged pool/community to pool/restore
<fauno> restoring [core]...
<fauno> it was working but then i realized some packages weren't supposed to be there

[some time later] Fauno realized repo-restore-to-normal had several bugs and after fixing them and emptying pool/packages, [core] could be restored. [to be continued]