Initial svn fetch_changesets takes 12 hours...
Added by Piers Wombwell over 12 years ago
I thought I'd try redmine with our repository.
So far it has taken 12 hours to do the import, though I do suspect most of this is because of swapping. I'm using a 2GB VM and the process is 2.4GB, but it is only taking 1.7% CPU.
top - 09:13:12 up 1 day, 21:04, 3 users, load average: 13.98, 13.53, 12.65
Tasks: 103 total, 2 running, 100 sleeping, 0 stopped, 1 zombie
Cpu(s): 0.0%us, 1.5%sy, 0.0%ni, 24.9%id, 73.6%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2056800k total, 2003604k used, 53196k free, 2220k buffers
Swap: 1048572k total, 679020k used, 369552k free, 9700k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
18989 root 20 0 2491m 1.7g 544 R 1.7 88.0 12:10.39 ruby
(I ran fetch_changesets as root -- I don't run the server as root)
This is using Ubuntu Oneiric's packaged up redmine, so version 1.1.3, but I can't see anything obvious in the changelogs for newer versions that might make any difference.
Any ideas? The repo is around 2GB with many files. Can I break it down into batches of revisions somehow?
Ruby version 1.8.7 (x86_64-linux)
RubyGems version 1.7.2
Rack version 1.1
Rails version 2.3.14
Active Record version 2.3.14
Active Resource version 2.3.14
Action Mailer version 2.3.14
Active Support version 2.3.14
Application root /usr/share/redmine
Environment production
Database schema version 20101114115359
Replies (2)
RE: Initial svn fetch_changesets takes 12 hours...
-
Added by Piers Wombwell over 12 years ago
I had to give up as the same machine hosts our svn repo and svn became painfully slow.
I guess there's no solution? It was all time spent in ruby rather than an 'svn' process, so I'm not sure the suggestion in a previous forum post of using libsvn would help -- though I haven't tried it.
RE: Initial svn fetch_changesets takes 12 hours...
-
Added by Piers Wombwell over 12 years ago
Not that anyone seem to have cared ;-) but I've figured out how to fix it.
Edit /usr/share/redmine/app/models/repository/subversion.rb
In fetch_changesets, modify:
# loads changesets by batches of 200
identifier_to = [identifier_from + 199, scm_revision].min
to
# loads changesets by batches of 1
identifier_to = [identifier_from + 1, scm_revision].min
This reduced the import time from >12 hours (with me aborting) to around five minutes.