Loading...

Type: Bug
Resolution: Unresolved
Priority: Major
Component/s: workflow-scm-step-plugin
Labels:
None

Similar Issues:

Show

Hi,

I need to poll hundreds of (fat) Git repos for changes and I can't use webhooks for that.

The problem: Using 'dir() + checkout scmGit' inside a loop gives an unexpected result because for a given iteration, the previous iteration may not have finished, so the .git inside the current iteration is polluted with information from the previous iteration. Each checkout seems to run asynchronously.

Consequence: using 'trigger pollscm', at the next build changes are detected as the last builds revisions in each dir/git repo are fulfilled with wrong data.

This problem doesn't occur using parallel as each checkout runs in a separate thread/stage, but it's not an option right now for me.

Code sample:

stage('Checkout repos') {
    steps {
        script {
            int size = reposMap.size()

            reposMap.eachWithIndex {name, branches, index ->
                println("\n######\n>> Checkout [${index}/${size}]: ${name}\nBranches:${branches}\n######")

                for (branch in branches) {
                    println(">>> Starting Checkout of ${name} on ${branch}")
                    dir("${TOP_DIR}/${name}/${branch}") {
                        checkout scmGit(
                            [
                                branches: [[name: branch]],
                                extensions: [
                                    cloneOption(depth: 1, noTags: true, shallow: true),
                                    pruneStaleBranch(),
                                    pruneTags(true)
                                ],
                                userRemoteConfigs: [[url: "${BASE_URL}/${name}", credentialsId: CRED_ID]]
                            ]
                        )
                    }
                    println(">>> Ending Checkout of ${name} on ${branch}")
                }
            }
        }
    }
}

And the output with comments

######
>> Checkout [7/118]: my_project/my_repo_B
Branches:[my_branch_B]
######
[Pipeline] echo
>>> Starting Checkout of my_project/my_repo_B on my_branch_B
[Pipeline] dir
Running in /opt/jenkins/workspace/test_poll_error/my_project/my_repo_B/my_branch_B
[Pipeline] {
[Pipeline] checkout
The recommended git tool is: NONE
using credential integration
Cloning the remote Git repository
Using shallow clone with depth 1
Avoid fetching tags
Avoid second fetch
Checking out Revision c34c9426ff31a1832829b0ceda6df3d575ce3f0a (origin/my_branch_B)
Commit message: "<BLANK>"
First time build. Skipping changelog.
[Pipeline] }
 > git config remote.origin.url https://my_server.com/scm/my_project/my_repo_A # timeout=10  // Here the previous iteration A writes in the current dir of B  
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git rev-parse origin/my_branch_A^{commit} # timeout=10
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7f4423aabd9865ad2bdfd9708d31ef115ce8b239 # timeout=10  // Wrong revision for B
[Pipeline] // dir
[Pipeline] echo
>>> Ending Checkout of my_project/my_repo_B on my_branch_B
[Pipeline] echo

######
>> Checkout [8/118]: my_project/my_repo_C
Branches:[my_branch_C]
######
[Pipeline] echo
>>> Starting Checkout of my_project/my_repo_C on my_branch_C
[Pipeline] dir
Running in /opt/jenkins/workspace/test_poll_error/my_project/my_repo_C/my_branch_C
[Pipeline] {
[Pipeline] checkout
The recommended git tool is: NONE
using credential integration
Cloning the remote Git repository
Using shallow clone with depth 1
Avoid fetching tags
Cloning repository https://my_server.com/scm/my_project/my_repo_B     // Here the previous iteration B writes in the current iteration C 
 > git init /opt/jenkins/workspace/test_poll_error/my_project/my_repo_B/my_branch_B # timeout=10
Fetching upstream changes from https://my_server.com/scm/my_project/my_repo_B
 > git --version # timeout=10
 > git --version # 'git version 2.25.1'
using GIT_ASKPASS to set credentials
 > git fetch --no-tags --force --progress --depth=1 -- https://my_server.com/scm/my_project/my_repo_B +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://my_server.com/scm/my_project/my_repo_B # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git rev-parse origin/my_branch_B^{commit} # timeout=10
 > git config core.sparsecheckout # timeout=10
 > git checkout -f c34c9426ff31a1832829b0ceda6df3d575ce3f0a # timeout=10
Cloning repository https://my_server.com/scm/my_project/my_repo_C
 > git init /opt/jenkins/workspace/test_poll_error/my_project/my_repo_C/my_branch_C # timeout=10
Fetching upstream changes from https://my_server.com/scm/my_project/my_repo_C
 > git --version # timeout=10
 > git --version # 'git version 2.25.1'
using GIT_ASKPASS to set credentials
 > git fetch --no-tags --force --progress --depth=1 -- https://my_server.com/scm/my_project/my_repo_C +refs/heads/*:refs/remotes/origin/* # timeout=10
Avoid second fetch
Checking out Revision 6b299692657347e89b3249303cc3ad644a306400 (origin/my_branch_C)
 > git config remote.origin.url https://my_server.com/scm/my_project/my_repo_C # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git rev-parse origin/my_branch_C^{commit} # timeout=10
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 6b299692657347e89b3249303cc3ad644a306400 # timeout=10
Commit message: "<BLANK>"
First time build. Skipping changelog.
[Pipeline] }
[Pipeline] // dir
[Pipeline] echo
>>> Ending Checkout of my_project/my_repo_C on my_branch_C
[Pipeline] echo

Details

Description

Attachments

Activity

People

Dates