Ticket #509 (closed defect: fixed)

Opened 6 years ago

Last modified 6 years ago

spanish subversion repository corruption

Reported by: kmaclean Owned by: kmaclean
Priority: critical Milestone:
Component: Audio Version: SpeechSubmission
Keywords: Cc:

Description (last modified by kmaclean) (diff)


when trying

# svnadmin verify /data/svnroot/svn/es

this command would verify everything was OK from version 1 to 1116 and then die with this error:

can't open file ... Input/Output error

looking at /data/svnroot/svn/es/db/revprops and revisions with nautilus, only revisions 1 to 1115 would display

looking at same folders from command line (using ls) showed all the revision files, but problems with folders 1116 to 1163.

turned out to be a bad sector on the ssd used in prod - however SMART data shows no errors - not a very comforting indicator... will monitor.

These were the steps taken to correct things:

1. on the web server/subversion server
turn off webgui & httpd
turnoff webgui monitor in crontab

2. set runlevel to 1
# init 1

3a. unmount all regular and bind mounts from the ssd drive
# umount /data
# umount /mnt/birch
# blkid   # to confirm drive id

3b. check disk
# fsck /dev/sdc
(same problem folders in svn show up here... fsck cleans these up)
# fsck   # to confirm verything was cleaned up
# reboot

4. backup spanish repository
# tar -zcvf  es_backup  /data/svnroot/es

5. get last good backup: July 1,2014 - why subversion allowed commits for all this time is beyond me... 

6. get a dump file of versions from July 1:
# svnadmin dump --incremental -r 8212:8787 /data/svnroot/svn/es > svndump

7. rename old /data/svnroot/svn/es folder to /data/svnroot/svn/es_bad_dec9_2014

8. extract last good backup to /data/svnroot/svn/es

9. load versions from last good backup to present into es (from backup)
# svnadmin load /data/svnroot/svn/es < svndump

10. verify that repository has been fixed
# svnadmin verify /data/svnroot/svn/es

(verified revisions 1 to 8787 - everything OK)

11. restart httpd & webgui

12. restore password file in /data/svnroot/svn/es/conf 
(does not get backed up)

Change History

comment:1 Changed 6 years ago by kmaclean

  • Status changed from new to closed
  • Resolution set to fixed

comment:2 Changed 6 years ago by kmaclean

manually reviewed Nov 28 submissions to make sure that subversion, master prompts files, webgui listen page, and repository website (voxforge1.org) all updated correctly.

comment:3 Changed 6 years ago by kmaclean

  • Description modified (diff)
Note: See TracTickets for help on using tickets.