-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
Jenkins 2.252
Gerrit 3.2.3
plugin 0.4.4
AWS EC2 slaves
I haven't been able to pinpoint what triggers the issue, but seemingly randomly, when my multibranch pipeline job is triggered by a new patchset to a change, or using the "Re-run" link in the Gerrit Checks panel, the build sometimes skips the SCM checkout step, with nothing logged to the console, and nothing in the jenkins.log. It's as if it just ignores the checkout completely. I tried adding a checkout scm step to the pipeline, but that does not improve anything.
When scanning for changes (which runs on the master), it is able to communicate properly with the Gerrit server, finds the change refs, and triggers the build as expected. But after selecting the slave agent to run on, the SCM checkout simply does nothing. The console shows:
12:59:57 Branch indexing
12:59:57 > git rev-parse --is-inside-work-tree # timeout=10
12:59:57 Setting origin to https://gerrit.local/a/project
12:59:57 > git config remote.origin.url https://gerrit.local/a/project # timeout=10
12:59:57 Fetching origin...
12:59:57 Fetching upstream changes from origin
12:59:57 > git --version # timeout=10
12:59:57 > git --version # 'git version 2.14.5'
12:59:57 > git config --get remote.origin.url # timeout=10
12:59:57 using GIT_ASKPASS to set credentials Gerrit REST API
12:59:57 > git fetch --tags --progress -- origin +refs/heads/dev:refs/remotes/origin/dev +refs/heads/master:refs/remotes/origin/master +refs/heads/release*:refs/remotes/origin/release* # timeout=10
12:59:58 Seen branch in repository origin/04/106504/3
12:59:58 Seen branch in repository origin/26/97726/2
12:59:58 Seen branch in repository origin/27/97727/2
...
12:59:58 Seen 177 remote branches
13:00:04 Obtained gerrit-checks.jenkins from 3addffcd266a2117b2394e458f6b6dd5f70ec89e
13:00:04 Running in Durability level: MAX_SURVIVABILITY
13:00:05 [Pipeline] Start of Pipeline
13:00:06 [Pipeline] node
13:00:06 Running on jenkins-macmini-06 in /var/lib/jenkins/workspace/rrit-pipeline_53_106353_35
13:00:06 [Pipeline] {
13:00:07 [Pipeline] stage
13:00:07 [Pipeline] { (Declarative: Checkout SCM)
13:00:07 [Pipeline] checkout
13:00:07 [Pipeline] }
13:00:07 [Pipeline] // stage
...
13:00:10 [Pipeline] sh
13:00:11 + ls -a
13:00:11 .
13:00:11 ..
13:00:11 [Pipeline] sh
13:00:11 + git show --stat
13:00:11 fatal: not a git repository (or any of the parent directories): .git
Note that the Declarative: Checkout SCM stage, which is where it should be cloning the repository, is empty with no git commands executed, no errors or status reported. Those last 2 sh commands are added in my pipeline, to verify that the repo was cloned correctly, and they both show that it was not. At the same time, nothing is reported to jenkins.log on the master.
However, sometimes it does clone the repository as expected, but I don't know yet what causes it to work or not work. Any way to increase logging so that I can identify what is causing this?
When the clone does work correctly, the same "Declarative: Checkout SCM" stage shows the expected git commands being executed:
18:41:34 Running on EC2 (TMG MGMT) - jenkins-mobile-spot (sir-cm69gqwj) in /var/lib/jenkins/workspace/rrit-pipeline_53_106353_28
18:41:34 [Pipeline] {
18:41:34 [Pipeline] stage
18:41:34 [Pipeline] { (Declarative: Checkout SCM)
18:41:34 [Pipeline] checkout
18:41:34 using credential gerrit-http-rest-as-jenkins
18:41:35 Cloning the remote Git repository
18:41:35 Cloning repository https://gerrit.local/a/project
18:41:35 > git init /var/lib/jenkins/workspace/rrit-pipeline_53_106353_28 # timeout=10
18:41:35 Using reference repository: /var/lib/jenkins/git-reference/project.git/
18:41:35 Fetching upstream changes from https://gerrit.local/a/project
18:41:35 > git --version # timeout=10
18:41:35 using GIT_ASKPASS to set credentials Gerrit REST API
18:41:35 > git fetch --tags --progress -- https://gerrit.local/a/project +refs/heads/*:refs/remotes/origin/* # timeout=10
18:41:42 > git config remote.origin.url https://gerrit.local/a/project # timeout=10
18:41:42 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
18:41:42 > git config remote.origin.url https://gerrit.local/a/project # timeout=10
18:41:42 Fetching upstream changes from https://gerrit.local/a/project
18:41:42 using GIT_ASKPASS to set credentials Gerrit REST API
18:41:42 > git fetch --tags --progress -- https://gerrit.local/a/project refs/changes/53/106353/28:refs/remotes/origin/53/106353/28 # timeout=10
18:41:44 Checking out Revision fabfae86429b68d13fa8a814683a144e36502160 (53/106353/28)
18:41:45 Commit message: "Add gerrit checks jenkinsfile"
18:41:45 First time build. Skipping changelog.
18:41:45 [Pipeline] }
18:41:45 [Pipeline] // stage
I don't think the AWS slave is the problem, because I tried switching the agent directive to select a macOS slave, and I see the same problem. I also ruled out the specific jenkins slave that executes the job, as when there's only a single slave to choose from, and it picks that slave every time, I still see the same random failures.
Early on the success rate was seemingly 50%, but as I've been trying to diagnose the issue and figure out why it's happening, it actually seems more like 10% success.
[JENKINS-63398] Pipeline randomly skips SCM Checkout step
Looking more into this, I'm beginning to think the missing checkout step may be due to the "filter by pending checks" behavior. But I'm having a hard time confirming if that is the case.
It meets the criteria and triggers the build during the GerritSCMSource discovery, but intermittently skips checkout at the execution phase. Looking at the pipeline-model-definitions-plugin, it looks like it is possible for the "Declarative: Checkout SCM" stage (name comes from SyntheticStageNames.checkout()) to be empty:
private static Map performCheckout(CpsScript script, DeclarativeAgent agent) { def checkoutMap = [:] if (!agent.inStage) { script.stage(SyntheticStageNames.checkout()) { checkoutMap.putAll(script.checkout(script.scm) ?: [:]) } } else { // No stage when we're in a nested stage already checkoutMap.putAll(script.checkout(script.scm) ?: [:]) } return checkoutMap }
The only way I can think that this would come up empty is if the filter included the change during discovery, but excludes it during execution – is that possible?
More info about my set up:
- Gerrit has 4 checkers defined: "my-project:style", "my-project:compile", "my-project:tests", and "my-project:assemble"
- All 4 checkers are intended to be evaluated by a single job, each in separate stages
- At the start of each checker stage in my pipeline file, I use gerritCheck checks: ["my-project:compile": "RUNNING"]
- In the post.successful and post.unsuccessful blocks, I set the relevant check results respectively
- In Jenkins, the job is configured with:
- a query change filter that targets only my own changes (owner:jhansche -age:1w), because I'm still in the testing phase, and don't want it to execute for everyone else's changes until I can verify it working properly
- "filter by pending checks", with "multiple checkers having the same scheme", and scheme = "my-project"
When a new patchset is created, all checks in Gerrit are set to status "Unevaluated". When the "Scan" task completes and evaluates the change, then all 4 checks become "Scheduled", until the respective stage executes as described above.
If I remove the "filter by pending checks" behavior, I see the Checkout SCM step working correctly every time (so far).
Am I misunderstanding how the filter should be applied? Is there any logging I can enable to see exactly why the checkout stage would become skipped?
This does seem to be related.
It looks like I'm misunderstanding the purpose of "Filter by pending checks", perhaps? Since my pipeline is going to run for all 4 of those checks and executes all checks in one pipeline, I don't think it makes sense to do that filtering. The downside, I think, is that by removing that filter, I think running the "Scan Multibranch Pipeline Now" action will automatically re-trigger the job for every change that otherwise meets the criteria – even if that change has already been verified before.
Is there another way that we can prevent it from re-triggering a job if a change has already been verified, but not skip the Checkout SCM step? It sounds like this configuration should work for most people, so it seems likely that something is misconfigured on my end. But it's not clear what I'm doing wrong – and that might indicate that a warning or log message would be worth adding, to explain what is happening and why.
If I update my Jenkinsfile to print:
steps {
echo "SCM=${scm}"
}
When it's working, I get an instance of GitSCM, and when it's not working, I get NullSCM.
So far, I'm still seeing it work every time, but only when I have the "Filter by pending checks" behavior removed.
I think this is related to https://issues.jenkins-ci.org/browse/JENKINS-61469.
felipecassiors can you expand on what part of JENKINS-61469 relates to this? I couldn't find anything in common.
It seems to me that the issue is when the filter by pending checks option is set, it will skip any changes that have a resolved status for the given filter. When it's filtered, the GitSCM is not applied (NullSCM takes its place), and therefore the "Checkout SCM" step does nothing – but also does not fail. It just skips over it as if it passed, but the rest of the job will fail because the workspace is empty.
I mentioned above that I might be misusing the filter, so if that's the case I would just need to understand the right way to use these options. So far the job has been working fine with the filter option removed, and that also allows the checks to be re-run from the Gerrit UI.
Yes, you're right. Sorry. Perhaps you should update the Issue type to "Bug"?
Andrew
added a comment - I'm currently experiencing this bug as well. I've validated this by modifying my `Jenkinsfile
` to have the `echo "SCM=${scm}"` snippet.
This is quite easy to reproduce with approx a 50% prevalence and was introduced once we turned on the checks plugin integration. I would be very happy helping in the debugging process in any way someone would like.
jhansche, do you have a work-around for this that you've been able to find?
Thank you so much!
Andrew
added a comment - I'm currently experiencing this bug as well. I've validated this by modifying my `Jenkinsfile
` to have the `echo "SCM=${scm}"` snippet.
This is quite easy to reproduce with approx a 50% prevalence and was introduced once we turned on the checks plugin integration. I would be very happy helping in the debugging process in any way someone would like.
jhansche , do you have a work-around for this that you've been able to find?
Thank you so much! From comments above, the workaround that has worked for me is to remove the "Filter by pending checks" behavior.
The problem seems to be that if the gerrit check already has been completed once, it does not count as "pending", and therefore it will not trigger, and the GitSCM does not get applied (replaced with NullSCM, which does nothing). The other way to work around that is to make the check pending again, which you can usually do by clicking the "Re-run" link in Gerrit.
If that doesn't solve your issue, then it might be a different filter behavior that isn't being applied the way you expect.
I haven't followed up on this because our switch to the multi-branch pipeline + checks has been put on hold for now. I suspect that I may have misunderstood the "Filter by pending checks" behavior, but I haven't gotten clarification on that yet.
Andrew
added a comment - Joe, thank you for the response. I removed the filter step where it matches against only changes that are unscheduled.
> The other way to work around that is to make the check pending again, which you can usually do by clicking the "Re-run" link in Gerrit.
Interestingly, when we were "Re-run"ing from inside of gerrit we had about the same error rate. Now that I've turned that off and switched to having people click "Replay" in the Jenkins UI if they ned a rerun it seems happy.
Andrew
added a comment - Joe, thank you for the response. I removed the filter step where it matches against only changes that are unscheduled.
> The other way to work around that is to make the check pending again, which you can usually do by clicking the "Re-run" link in Gerrit.
Interestingly, when we were "Re-run"ing from inside of gerrit we had about the same error rate. Now that I've turned that off and switched to having people click "Replay" in the Jenkins UI if they ned a rerun it seems happy.
I'm not sure if I can really get an exact one-for-one replacement using just the Git SCM Source, but this is what I tried:
On save, it automatically scans, and correctly finds my change and schedules a build:
> git fetch --tags --progress --prune -- origin +refs/changes/53/106353/35:refs/remotes/origin/changes/53/106353/35 # timeout=10 Checking other refs... Checking ref other-ref (refs/changes/53/106353/35) ‘gerrit-checks.jenkins’ found Met criteriaThis triggers "other-ref" (I could give it a name, and I confirmed that works if I do).
But that has all sorts of other issues, probably because I can't use the "Gerrit" chooser strategy, like I would have with the Gerrit Trigger Plugin? Instead, I see things like:
19:40:39 > git fetch --tags --progress -- origin +refs/changes/53/106353/35:refs/remotes/origin/changes/53/106353/35 # timeout=10 19:40:39 > git rev-parse refs/changes/53/106353/35^{commit} # timeout=10 19:40:39 ERROR: Could not determine exact tip revision of other-ref; falling back to nondeterministic checkoutand
19:40:40 Fetching upstream changes from https://gerrit.server/a/project 19:40:40 > git --version # timeout=10 19:40:40 > git --version # 'git version 2.14.5' 19:40:40 using GIT_ASKPASS to set credentials Gerrit REST API 19:40:40 > git fetch --no-tags --progress -- https://gerrit.server/a/project +refs/changes/53/106353/35:refs/remotes/origin/changes/53/106353/35 refs/changes/53/106353/35 # timeout=10 19:41:08 > git config remote.origin.url https://gerrit.server/a/project # timeout=10 19:41:08 > git config --add remote.origin.fetch +refs/changes/53/106353/35:refs/remotes/origin/changes/53/106353/35 # timeout=10 19:41:08 > git config --add remote.origin.fetch refs/changes/53/106353/35 # timeout=10 19:41:08 > git config remote.origin.url https://gerrit.server/a/project # timeout=10 19:41:08 Fetching without tags 19:41:08 Fetching upstream changes from https://gerrit.server/a/project 19:41:08 using GIT_ASKPASS to set credentials Gerrit REST API 19:41:08 > git fetch --no-tags --progress -- https://gerrit.server/a/project +refs/changes/53/106353/35:refs/remotes/origin/changes/53/106353/35 refs/changes/53/106353/35 # timeout=10 19:41:08 > git rev-parse origin/other-ref^{commit} # timeout=10 19:41:08 > git rev-parse other-ref^{commit} # timeout=10 19:41:08 ERROR: Couldn't find any revision to build. Verify the repository and branch configuration for this job. 19:41:08 ERROR: Maximum checkout retry attempts reached, aborting 19:41:08 Finished: FAILUREIf I set the discover-other-ref mapping to "C-106353" (just for arguments sake, to get it to build), and add "specify refspec" to fetch +refs/changes/53/106353/35:refs/heads/C-106353, then I can at least get it to discover the right git commit to build, but I'm seeing a partially similar issue with that:
19:39:05 Fetching upstream changes from origin 19:39:05 > git --version # timeout=10 19:39:05 > git --version # 'git version 2.14.5' 19:39:05 > git config --get remote.origin.url # timeout=10 19:39:05 using GIT_ASKPASS to set credentials Gerrit REST API 19:39:05 > git fetch --tags --progress -- origin +refs/changes/53/106353/35:refs/remotes/origin/changes/53/106353/35 +refs/changes/53/106353/35:refs/heads/C-106353 # timeout=10 19:39:06 > git rev-parse refs/changes/53/106353/35^{commit} # timeout=10 19:39:06 ERROR: Could not determine exact tip revision of C-106353; falling back to nondeterministic checkoutthat error is same as above, but it does finally figure out how to check out the right commit:
19:39:06 > git fetch --no-tags --progress -- https://gerrit.server/a/project +refs/changes/53/106353/35:refs/remotes/origin/changes/53/106353/35 +refs/changes/53/106353/35:refs/heads/C-106353 refs/changes/53/106353/35 # timeout=10 19:39:06 > git rev-parse origin/C-106353^{commit} # timeout=10 19:39:06 > git rev-parse C-106353^{commit} # timeout=10 19:39:06 Checking out Revision 3addffcd266a2117b2394e458f6b6dd5f70ec89e (C-106353) 19:39:06 > git config core.sparsecheckout # timeout=10 19:39:06 > git checkout -f 3addffcd266a2117b2394e458f6b6dd5f70ec89e # timeout=10 19:39:07 Commit message: "Add gerrit checks jenkinsfile" 19:39:07 First time build. Skipping changelog. 19:39:07 Running in Durability level: MAX_SURVIVABILITY 19:39:07 [Pipeline] Start of Pipeline 19:39:08 [Pipeline] node 19:39:08 Running on jenkins-macmini-06 in /var/lib/jenkins/workspace/test-pipeline_C-106353 19:39:08 [Pipeline] { 19:39:08 > git rev-parse --is-inside-work-tree # timeout=10 19:39:08 Setting origin to https://gerrit.server/a/project 19:39:08 > git config remote.origin.url https://gerrit.server/a/project # timeout=10 19:39:08 Fetching origin... 19:39:08 Fetching upstream changes from origin 19:39:08 > git --version # timeout=10 19:39:08 > git --version # 'git version 2.14.5' 19:39:08 > git config --get remote.origin.url # timeout=10 19:39:08 using GIT_ASKPASS to set credentials Gerrit REST API 19:39:08 > git fetch --tags --progress -- origin +refs/changes/53/106353/35:refs/remotes/origin/changes/53/106353/35 +refs/changes/53/106353/35:refs/heads/C-106353 # timeout=10 19:39:09 > git rev-parse refs/changes/53/106353/35^{commit} # timeout=10 19:39:09 [Pipeline] throttle 19:39:09 [Pipeline] { 19:39:09 [Pipeline] stage 19:39:09 [Pipeline] { (Declarative: Tool Install) 19:39:09 [Pipeline] tool 19:39:09 [Pipeline] envVarsForTool 19:39:09 [Pipeline] } 19:39:09 [Pipeline] // stage 19:39:09 [Pipeline] withEnv 19:39:09 [Pipeline] { 19:39:09 [Pipeline] stage 19:39:09 [Pipeline] { (Prepare) 19:39:09 [Pipeline] tool 19:39:09 [Pipeline] envVarsForTool 19:39:09 [Pipeline] withEnv 19:39:09 [Pipeline] { 19:39:09 [Pipeline] sh 19:39:09 + pwd 19:39:09 /private/var/lib/jenkins/workspace/test-pipeline_C-106353 19:39:09 [Pipeline] sh 19:39:10 + ls -a 19:39:10 . 19:39:10 .. 19:39:10 [Pipeline] sh 19:39:10 + git show --stat 19:39:10 fatal: not a git repository (or any of the parent directories): .git 19:39:10 [Pipeline] }So, a few things:
However, it isn't able to figure out how to build that 3addffc commit, and I end up with the same empty workspace, no .git directory, and a git command reports an error (as expected without a cloned workspace).
Could it be that it requires an updated version of the git cli, and it's just not reporting any errors to that effect? I can't imagine that's the problem, and I see the issue on both AWS linux slaves, as well as our own data center Mac Pro and Mac Mini slaves.
I'm at a loss, how I can continue debugging this to see where exactly it's going wrong.