Feature #412

multiple cloning of gits - an idea for load reduction for servers + slow connections

Added by Simon Schulz over 6 years ago. Updated over 5 years ago.

Status:ResolvedStart date:2015-06-10
Priority:HighDue date:
Assignee:Jan Moringen% Done:

50%

Category:build-generatorSpent time:-
Target version:Toolkit 0.2

Description

The build generator and jenkins does multiple checkouts from git rpositories.
This is annoying and produces high load on the repository servers.
It also makes it impossible to be used on slow DSL connections at home (waiting hours for a build...).

Another thing is that we have problems with building the distribution at the lsp-csra due to the big size and the amounts of checkouts (high load on repo servers).

What about implenting the following caching behaviour ?

1) the buildgenerator:

if (does not exist $CACHEDIR/$projname){
  clone repo to $CACHEDIR/$projname
}else{
  cd $CACHEDIR/$projname && git pull
}

2) the buildgen enters $CACHEDIR/$projname as git url in the jenkins project configuration (that works, this is how i use jenkins at home)

It might be necessary to add a git pull to the project files, but i think jenkins does that automatically.

By implementin this we would reduce 3x checkout to 1x or even no full checkout at all.
This would vastly reduce the load on our repository/network infrastructure and would
also reduce the total build times!

I would really prefer using more diskspace due to the caching instead of network bandwidth usage.

That should not be to hard to implement. I do this manually when i build large distributions at home.


Related issues

Related to Cognitive Interaction Toolkit - Feature #338: using the toolkit installer on "slow" (<=10 MBit/s) conne... Closed 2014-11-22

History

#1 Updated by Robert Haschke over 6 years ago

Nice idea. I do it the same way on my laptop from home. However, this doesn't work anymore if the build-generator and jenkins are on different machines / cannot access the cached repository dirs the same. However, build-generator could do the caching anyway.

Additionally, projects repos should be designed modular, i.e. not required to checkout a monolitic repo if their parts can be (and are) build separately...

#2 Updated by Patrick Holthaus over 6 years ago

That could actually save a lot of traffic (also for the redmine itself!)

#3 Updated by Florian Lier over 6 years ago

  • Target version set to Toolkit 0.2

bump

#4 Updated by Jan Moringen about 6 years ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 50

build-generator side of this strategy has been prototypically implemented and is currently undergoing testing.

#5 Updated by Jan Moringen about 6 years ago

  • Related to Feature #338: using the toolkit installer on "slow" (<=10 MBit/s) connections - tar snapshots instead of git? added

#6 Updated by Jan Moringen over 5 years ago

  • Status changed from In Progress to Resolved

Generator-side caching with incremental updates is implemented.

Not wiping out workspaces can be used to make the Jenkins-side incremental.

Also available in: Atom PDF