Having the same problem. Jenkins 1.479, Jenkins Subversion Plugin 1.42

Seeing this on Jenkins 1.479 with plugin 1.42 as well.

Worth noting that at Last.fm we merge down branches which add files quite often, but this behaviour only started happening recently when we upgraded. Probably within the last few point versions of the plugin.

So this only happens with the mentioned svn strategy?

It happens also when using 'use svn update as much as possible'.

When the build fails we wipeout the workspace and re-build.

Same issue here, Jenkins 1.485, Jenkins Subversion Plugin 1.43

I have same issue,Jenkins 1.459 ,subversion plugin 1.43

As this bug is really annoying i've created a 50$ reward to fix this, expires Nov/09/2012: http://www.freedomsponsors.org/core/issue/61/an-update-which-contains-added-files-merged-from-a-branch-results-in-those-files-having-doubled-content

Same issue: Jenkins 1.487, Jenkins Subversion Plug-in 1.43

It occurs always after a merge from a branch. I'm using the "Use 'svn update' as much as possible" check-out strategy.

Same issue: Jenkins 1.489, Jenkins Subversion Plug-in 1.43 (it was already bugged in previous versions)

Same for me. Jenkins 1.424.2 and Subversion plugin 1.43.

Same for me. (It's really annoying!). Jenkins 1.492 and Subversion 1.43

The problems happens if, in a commit which already committed to the subversion server, a branch has been merged into
trunk that where some files are added.

Who or what is merging the branch into trunk?

• A developer, on their own machine, which is then committed to svn.

And what if you checkout the trunk with command line svn - is the content duplicated, too?

• no, if you use the command line tools, the checkout is as expected, with no duplicated content.

Okay.
So what would really help to debug the issue would be if you could create a minimal subversion repository which has this characteristics.
Ideally also a snapshot of the repository BEFORE and AFTER the branch has been merged.

I have the same issue, file content is duplicated. update strategy is 'emulate clean...' and I only see this behavior when new files are added after merge.

How to reproduce:

• Do this on the Jenkins host:

cd /tmp
mkdir svn ws
svnadmin create svn
svn mkdir file:///tmp/svn/trunk -m trunk
svn mkdir file:///tmp/svn/branch -m branch

cd ws
svn co file:///tmp/svn/branch
cd branch
echo test > singleline
svn add singleline
svn commit -m added

• Now create a Jenkins "freestyle" project with SVN URL file:///tmp/svn/trunk and build it once, then continue in the shell:

cd /tmp/ws
svn co file:///tmp/svn/trunk
cd trunk
svn merge -r 2:3 ../branch/
svn commit -m merged

• Now build the project again. Check the workspace, the file "singleline" will contain 2 lines!

Reproducible with: Jenkins 1.502, Subversion Plugin 1.45, Host SVN 1.6.12 (Debian/CollabNet).

Same issue: Jenkins 1.501, Jenkins Subversion Plug-in 1.45
Check-out Strategy: Emulate clean checkout by first deleting unversioned/ignored files, then 'svn update'

Seems not to be file-type related since it occurred on both xml and java files.

This is Critical! Issues like these can make chain-reaction on ant build script based projects that do not utilize distributed build model or do use any kind of reverting mechanism on failed build.

Same problem here.
Jenkins ver. 1.505
Jenkins Subversion Plug-in ver. 1.45

We are seeing file contents double when using SVN mv commands as well. This means files that are svn mv'd to a location, then added back to the repo have their contents duplicated.

Here are the reproduction steps in action:

$ vim double.txt 
$ cat double.txt 
1
2
3

$ svn add double.txt 
A         double.txt
$ svn ci . -m "Initial commit "
Adding         double.txt
Transmitting file data .

Committed revision <revision>.

$ svn mv double.txt double_new.txt
A         double_new.txt
D         double.txt

$ svn ci . -m "Renaming double"
Deleting       double.txt
Adding         double_new.txt

Committed revision <revision>.

Here is the output for Jenkins when it runs a job. The job updates a checkout which contains file “double_new.txt”, then does a ‘cat’ command on the file. Here’s the output:

Started by user Kenneth Ayers
Building remotely on builder in workspace /test
Updating <repository> at revision <revision>
A         double_new.txt
At revision <revision>

[test] $ /bin/sh -xe /tmp/hudson5312350812342753447.sh
+ cat double_new.txt
1
2
3

1
2
3

Finished: SUCCESS

As you can see the file contents are now doubled up.

Versions:
Jenkins: 1.508
Jenkins Subversion Plug-in: 1.45

We were able to reproduce this very simply:

• Create a file in SVN
• Rename this file in SVN (should stay in same folder, content must not change!) (A manual delete/add does not work)
• Build the project with Jenkins and let him update its project with the "Revert before Update" strategy. I didn't check other strategies.

Now it is very sure the file has double its content.

Jenkins: 1.494
Jenkins Subversion Plugin: 1.45

PLEASE FIX THIS! Many ways to reproduce the Bug have been shown now. And since this is only a very subtile fail in a critical component which leads to hard to track and reproduce errors (dev environemnt works, production fails), this bug should be fixed immediatly. Because of the bug we had valid, but wrong, datafiles sent to our customers! And this bug ruins our convidence with the whole Jenkins toolchain in my company.

This bug may be in SVNKit, which is the Java library that subversion-plugin uses. I determined this by modifying hudson.scm.subversion.UpdateUpdater by adding a loop to print the contents of each file after the SVN checkout is completed. Immediately after the doUpdate() function which is responsible for updating a checkout is called, I inspect the file contents to find they are corrupted.

Here's the git patch to hudson.scm.subversion.UpdateUpdater with the debug output. Please note I'm not a Java developer by trade, and this code has cruft from a lot of debugging iterations. I've also modified the parameters being used by doUpdate() as part of the debugging process:

diff --git a/src/main/java/hudson/scm/subversion/UpdateUpdater.java b/src/main/java/hudson/scm/subversion/UpdateUpdater.java
index f00ebdb..021e18c 100755
--- a/src/main/java/hudson/scm/subversion/UpdateUpdater.java
+++ b/src/main/java/hudson/scm/subversion/UpdateUpdater.java
@@ -40,11 +40,21 @@ import org.tmatesoft.svn.core.wc.SVNInfo;
 import org.tmatesoft.svn.core.wc.SVNRevision;
 import org.tmatesoft.svn.core.wc.SVNUpdateClient;
 import org.tmatesoft.svn.core.wc.SVNWCClient;
+import org.tmatesoft.svn.core.wc.SVNClientManager;
+import org.tmatesoft.svn.core.SVNDepth;
+import org.tmatesoft.svn.core.wc.SVNRevision;
 
 import java.io.File;
 import java.io.IOException;
 import java.util.ArrayList;
+import java.util.Arrays;
 import java.util.List;
+import java.io.InputStream;
+import java.io.BufferedInputStream;
+import java.io.FileInputStream;
+import java.io.DataInputStream;
+import java.io.InputStreamReader;
+import java.io.BufferedReader;
 
 /**
  * {@link WorkspaceUpdater} that uses "svn update" as much as possible.
@@ -154,7 +164,38 @@ public class UpdateUpdater extends WorkspaceUpdater {
                 switch (svnCommand) {
                     case UPDATE:
                         listener.getLogger().println("Updating " + location.remote + " at revision " + revisionName);
-                        svnuc.doUpdate(local.getCanonicalFile(), r, svnDepth, true, true);
+                        //svnuc.doUpdate(local.getCanonicalFile(), r, svnDepth, true, true);
+                        SVNClientManager cM = SVNClientManager.newInstance();
+                        SVNUpdateClient updateClient = cM.getUpdateClient();
+                        updateClient.doUpdate(local.getCanonicalFile(), SVNRevision.HEAD, SVNDepth.INFINITY, true, true);
+
+                        ArrayList<File> files = new ArrayList<File>(Arrays.asList(local.listFiles()));
+                        for (int i = 0; i < files.size(); i++)
+                        {
+                            if (files.get(i).isFile())
+                            {
+                                String fname = files.get(i).getName();
+                                listener.getLogger().println("File: " + fname);
+                                try{
+                                  // Open the file that is the first 
+                                  // command line parameter
+                                  FileInputStream fstream = new FileInputStream(files.get(i));
+                                  // Get the object of DataInputStream
+                                  DataInputStream in = new DataInputStream(fstream);
+                                  BufferedReader br = new BufferedReader(new InputStreamReader(in));
+                                  String strLine;
+                                  //Read File Line By Line
+                                  while ((strLine = br.readLine()) != null)   {
+                                  // Print the content on the console
+                                  listener.getLogger().println(strLine);
+                                  }
+                                  //Close the input stream
+                                  in.close();
+                                    }catch (Exception e){//Catch exception if any
+                                  listener.getLogger().println("Error: " + e.getMessage());
+                                  }
+                            }
+                        }
                         break;
                     case SWITCH:
                         listener.getLogger().println("Switching to " + location.remote + " at revision " + revisionName);

I've posted a bug report to the SVNKit issue tracker, hopefully this will get some attention: http://issues.tmatesoft.com/issue/SVNKIT-368

All,

I may have found the issue. I'd like the Jenkins devs to take a look, and I've also appended this information to the SVNKit ticket, here: http://issues.tmatesoft.com/issue/SVNKIT-368

In org.tmatesoft.svn.core.internal.wc.SVNUpdateEditor15.java, in function addFileWithHistory (line 867), there's a code block that calls myFileFetcher.fetchFile() twice. Each time this is called, baseTextOS is written to. Upon the second write, the file contents are duplicated. Here's the code:

baseTextOS = SVNFileUtil.openFileForWriting(info.copiedBaseText);
myFileFetcher.fetchFile(copyFromPath, copyFromRevision, baseTextOS, baseProperties);
SVNChecksumOutputStream checksumBaseTextOS = new SVNChecksumOutputStream(baseTextOS, 
SVNChecksumOutputStream.MD5_ALGORITHM, true);
baseTextOS = checksumBaseTextOS;
myFileFetcher.fetchFile(copyFromPath, copyFromRevision, baseTextOS, baseProperties);
info.copiedBaseChecksum = checksumBaseTextOS.getDigest();

I was able to find this by stepping through the code using NetBeans IDE 7.3 attached to a remote debugging session on Jenkins. I've compiled and tested this change inside the context of the subversion-plugin and the file contents are no longer duplicated.

I've forked the svnkit repo used in Jenkins here, and committed this change if anyone would like to download the fix and do some testing:

https://github.com/theotherwhitemeat/svnkit-1/commit/27decb28216ee4fd15b8fcbdb769bf41d81978eb

Here's my patch:

Index: SVNUpdateEditor15.java
===================================================================
--- SVNUpdateEditor15.java	(revision 9722)
+++ SVNUpdateEditor15.java	(working copy)
@@ -864,7 +864,6 @@
             OutputStream baseTextOS = null;
             try {
                 baseTextOS = SVNFileUtil.openFileForWriting(info.copiedBaseText);
-                myFileFetcher.fetchFile(copyFromPath, copyFromRevision, baseTextOS, baseProperties);
                 SVNChecksumOutputStream checksumBaseTextOS = new SVNChecksumOutputStream(baseTextOS, 
                         SVNChecksumOutputStream.MD5_ALGORITHM, true);
                 baseTextOS = checksumBaseTextOS;

Thank you,

Kenny Ayers

I went ahead and compiled this change into the subversion plugin, at least I think I did. This is using the latest rev from subversion-plugin along with my svnkit library modification. If you guys would like to do some additional testing to verify my change, the plugin is located here:

https://github.com/theotherwhitemeat/misc_binaries

I do not suggest using this for anything but testing / verification.

-Kenny

I just hit this too, version 1.44 on Jenkins 1.513. Since it's fixed in 1.7.10, updating svnkit should fix this. However, the subversion plugin uses a patched version of svnkit so it's not necessarily as easy as updating the dependency. I would love to see the pull request: https://github.com/jenkinsci/svnkit/pull/1 merged and the subversion plugin updated to prevent this from corrupting builds.

I ended up patching the jenkins fork of svnkit locally, building the subversion-plugin 1.45 with the patched version of svnkin. It does indeed appear to fix the behavior.

Kenny, Bret,

Did you do any modification to the pom files of the fork of jenkinsci-svnkit or to the svnPlugin?

In fact I'm using the svnkit code patched by Kenny:

https://github.com/theotherwhitemeat/svnkit-1

...and the SVNPlugin 1.45:

https://github.com/jenkinsci/subversion-plugin/tree/subversion-1.45

The first one was successfully built, but then I get tons of test errors while building the second one. Any idea?

I know I could simply use the build of Kenny (btw, thx! ) but I wanted to try to build it on my own.

Thanks for any help!

p.s. Kenny, I had the very same problem mentioned by you here:

http://jenkins.361315.n4.nabble.com/Compiling-SVNKit-for-subversion-plugin-td4663822.html

...and I ended up changing the exec task in the pom file of svnKit from this:

<exec command="./gradlew clean assemble" />

...to this:

<exec command="./gradlew clean build -x test -x signMaven"/>

Did you do the same? I then copied the jar file located in the gradle/build folder into my local maven repo, I hope this was not an incredible stupid idea...

I had similar issues, building on windows.

Hey LuFrija,

If my memory is correct, I had to do the same thing - modify the build command for SVNKit, then copy the resultant jar file into my local maven repo, then build subversion-plugin against that.

This seems like a hacky workflow, but it's the best I could stitch together given there are no instructions on how to build this properly, and the mailing list didn't respond to my question on how to build the correct way.

Are you good to go now?

-Kenny

Hi guys,

Thanks for your fast reply!

I guess I found the problem...it's all about my company's proxy! In fact having a closer look at the failing tests I could see something like:

ERROR: Failed to check out https://svn.jenkins-ci.org/trunk/hudson/test-projects/trivial-ant

I'm now copying with it somehow, in the meantime I'm testing the hpi file uploaded by Kenny.

Code changed in jenkins
User: Kenny Ayers
Path:
svnkit/src/main/java/org/tmatesoft/svn/core/internal/wc/SVNUpdateEditor15.java
http://jenkins-ci.org/commit/svnkit/27decb28216ee4fd15b8fcbdb769bf41d81978eb
Log:
Removing duplicate myFileFetcher.fetchFile() call. JENKINS-14551

Code changed in jenkins
User: Nicolas De loof
Path:
svnkit/src/main/java/org/tmatesoft/svn/core/internal/wc/SVNUpdateEditor15.java
http://jenkins-ci.org/commit/svnkit/aec5b5033ebcf927978e1764379bcbcb3989b900
Log:
Merge pull request #1 from theotherwhitemeat/master

A fix for JENKINS-14551

Compare: https://github.com/jenkinsci/svnkit/compare/9bea57e044b4...aec5b5033ebc

Code changed in jenkins
User: Christoph Kutzinski
Path:
pom.xml
http://jenkins-ci.org/commit/subversion-plugin/3599af503fc0a10bf29e3dfab1619ed53a2e71fc
Log:
[FIXED JENKINS-14551] added files merged from a branch results in those files having doubled content

Compare: https://github.com/jenkinsci/subversion-plugin/compare/5fb0b85fcf2c^...3599af503fc0