-
Bug
-
Resolution: Unresolved
-
Minor
Jobs are not executing the Build section without any reason.
After the SCM section build section is not being executed and job flow is directly going to post build actions with out any reason/log.
This is happening intermittently for all jobs in our environment.
Here is the log.
Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on Docker-eff8c69cef9f (docker-slave) in workspace /workspace/mybuild Wiping out workspace first. Cloning the remote Git repository Cloning repository ssh://git@something.git > /usr/bin/git init /workspace/mybuild # timeout=10 Fetching upstream changes from ssh://git@something.git > /usr/bin/git --version # timeout=10 using GIT_SSH to set credentials > /usr/bin/git -c core.askpass=true fetch --tags --progress ssh://git@something +refs/heads/*:refs/remotes/origin/* > /usr/bin/git config remote.origin.url ssh://git@something # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url ssh://git@something.git# timeout=10 Fetching upstream changes from ssh://git@something.git using GIT_SSH to set credentials > /usr/bin/git -c core.askpass=true fetch --tags --progress ssh://git@somethinggit +refs/heads/*:refs/remotes/origin/* > /usr/bin/git rev-parse origin/develop^{commit} # timeout=10 Checking out Revision 7d80e72e571df52e45f9639583654 (origin/develop) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f 7d80e72e57c06c2e45f9639583654 > /usr/bin/git rev-list 4e04bb9876b85e78afd830dc5411225 # timeout=10 No emails were triggered. [ANALYSIS-COLLECTOR] Skipping publisher since build result is FAILURE Skipping Cobertura coverage report as build was not UNSTABLE or better ... TestNG Reports Processing: START Looking for TestNG results report in workspace using pattern: **/testng-results.xml, **/surefire-reports/*.xml, **/TEST-*.xml Did not find any matching files. [PostBuildScript] - Execution post build scripts. [PostBuildScript] Build is not success : do not execute script Publish artifacts to S3 Bucket Using S3 profile: s3-upload-snapshot Publish artifacts to S3 Bucket Skipping publishing on S3 because build failed Email was triggered for: Failure - Any Sending email for trigger: Failure - Any Sending email to: us ERROR: Build step failed with exception Finished: FAILURE
Ideally before the part "[ANALYSIS-COLLECTOR]" it should run our build part which is maven invoke.
This issue is happening very frequently.
Here is the detail about our env
Jenkins - 1.625.3
java - 1.8
OS - ubuntu 14.04
Tomcat container based jenkins installation
Not about idea the plugin as this is the core feature of the jenkins.
Detail jenkins system info is attached with the ticket.
[JENKINS-32121] Jenkins is skipping Build section intermittently
vishal sahasrabuddhe
added a comment - I do not think that this is causing because of slow checkout.
I have tested with one of the project without any git checkout and here is the log
Started by user Vishal [EnvInject] - Loading node environment variables. Building remotely on Hulk (Prod Traditional) in workspace /home/jenkins/workspace/Feedback_Deploy_OpsWorks No emails were triggered. Email was triggered for: Failure - Any Email was triggered for: Always Sending email for trigger: Failure - Any Request made to compress build log An attempt to send an e-mail to empty list of recipients, ignored. Sending email for trigger: Always Request made to compress build log Sending email to: us ERROR: Build step failed with exception Warning: ‘Access Control for Builds’ in global security configuration is empty, so falling back to legacy behavior of permitting any downstream builds to be triggered Finished: FAILURE
We have shell script executable block in build section which has not been executed by jenkins, it should be run before email trigger section.
vishal sahasrabuddhe
added a comment - I do not think that this is causing because of slow checkout.
I have tested with one of the project without any git checkout and here is the log
Started by user Vishal
[EnvInject] - Loading node environment variables.
Building remotely on Hulk (Prod Traditional) in workspace /home/jenkins/workspace/Feedback_Deploy_OpsWorks
No emails were triggered.
Email was triggered for : Failure - Any
Email was triggered for : Always
Sending email for trigger: Failure - Any
Request made to compress build log
An attempt to send an e-mail to empty list of recipients, ignored.
Sending email for trigger: Always
Request made to compress build log
Sending email to: us
ERROR: Build step failed with exception
Warning: ‘Access Control for Builds’ in global security configuration is empty, so falling back to legacy behavior of permitting any downstream builds to be triggered
Finished: FAILURE
We have shell script executable block in build section which has not been executed by jenkins, it should be run before email trigger section. It certainly could be timeout related.
For example, I defined a 1 minute clone timeout for a job which attempts to clone a 12GB+ git repo then run "mvn clean" and report unit test results and coverage results. It ran for one minute, then reported:
... Receiving objects: 5% (61763/1121901), 1.71 GiB | 52.34 MiB/s at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1640) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1388) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:62) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:313) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:505) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1003) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1043) at hudson.scm.SCM.checkout(SCM.java:485) at hudson.model.AbstractProject.checkout(AbstractProject.java:1275) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:610) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:532) at hudson.model.Run.execute(Run.java:1741) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:408) ERROR: null Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error? Publishing Javadoc Finished: FAILURE
I refined the job definition to use a reference repository, a shallow clone, a 20 minute timeout on clone, and a 1 minute timeout on checkout. That reported:
Fetching upstream changes from git://git.dev.fco/var/git/UIM/monitoring.git
> git -c core.askpass=true fetch --tags --progress git://git.dev.fco/var/git/UIM/monitoring.git +refs/heads/*:refs/remotes/origin/* # timeout=20
> git rev-parse refs/remotes/origin/master^{commit} # timeout=10
> git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 099e82be841f733e7210e1fe9b4a3111711d4626 (refs/remotes/origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 099e82be841f733e7210e1fe9b4a3111711d4626 # timeout=1
ERROR: Timeout after 1 minutes
FATAL: Could not checkout 099e82be841f733e7210e1fe9b4a3111711d4626
hudson.plugins.git.GitException: Could not checkout 099e82be841f733e7210e1fe9b4a3111711d4626
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$8.execute(CliGitAPIImpl.java:1907)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1060)
at hudson.scm.SCM.checkout(SCM.java:485)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1275)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:610)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:532)
at hudson.model.Run.execute(Run.java:1741)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:408)
Caused by: hudson.plugins.git.GitException: Command "git checkout -f 099e82be841f733e7210e1fe9b4a3111711d4626" returned status code 143:
stdout:
stderr:
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1640)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:62)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$8.execute(CliGitAPIImpl.java:1899)
... 10 more
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Publishing Javadoc
Finished: FAILURE
Note that in the second case (timeout on checkout, not timeout on clone), there is a timeout failure message in the log file, while in the clone (fetch) timeout case there is no message in the log file. There is a stack trace on timeout in both cases, but I am not entirely confident that timeout will always generate a stack trace. Thus, further support for your theory that the problem may be a timeout. I think it is probably timeout on clone (fetch) rather than timeout on checkout.
vishal sahasrabuddhe
added a comment - Hi Mark,
As per my 2nd comment/example, My SCM section is none, that means i am not cloning/checkout anything. It is just plan job running bash (execute build step ) printing hello world.
So in my 2nd example, my job does not have any clone or checkout and its still failing and we are facing same issue.
-Vishal
vishal sahasrabuddhe
added a comment - Hi Mark,
As per my 2nd comment/example, My SCM section is none, that means i am not cloning/checkout anything. It is just plan job running bash (execute build step ) printing hello world.
So in my 2nd example, my job does not have any clone or checkout and its still failing and we are facing same issue.
-Vishal That's a great answer, since that says that the problem is not related to the git plugin. I'm assigning this to core rather than the git plugin.
Please provide a minimal job config.xml for an affected job, as well as the contents of the jenkins.log file.
Attaching 2 config files,
one with scm plugin and checkout added.
2nd without SCM plugin and checkout task.
Couldn't find the log with the error, as its intermittent.
vishal sahasrabuddhe
added a comment - - edited Attaching 2 config files,
one with scm plugin and checkout added.
2nd without SCM plugin and checkout task.
Couldn't find the log with the error, as its intermittent. vishal sahasrabuddhe
added a comment - OK, i was able to capture the error log with one of our job.
I have seen a same log pattern whenever our job failed because of same issue.
Attaching log (jenkins log with build error).
vishal sahasrabuddhe
added a comment - OK, i was able to capture the error log with one of our job.
I have seen a same log pattern whenever our job failed because of same issue.
Attaching log (jenkins log with build error). 
Adam L
added a comment - Has any work been done to address this issue, or any similar?
I, too, am getting builds that are breaking because of the plugin. The only information I get in my log is:
Build step 'Audit job info to Database' marked build as failure
The side effects are maven not being installed on a slave, any of my build steps executing, and the job fails immediately after checking out source code. If I remove the audit step, everything works fine. if I add it back in, reverts to failure.
I've checked my connection from the main config page, and it works fine. I have jobs that are successfully logging to the audit db, and others that are randomly failing.
What's odd is the order in which the failures are listed:
> /usr/bin/git rev-list 12345678xxxx # timeout=10
No emails were triggered.
Skipping sonar analysis due to bad build status FAILURE
[BFA] Scanning build for known causes...
[BFA] No failure causes found
[BFA] Done. 0s
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Sending email to: xxxxx
Build step 'Audit job info to Database' marked build as failure
Finished: FAILURE
any thoughts?
Adam L
added a comment - Has any work been done to address this issue, or any similar?
I, too, am getting builds that are breaking because of the plugin. The only information I get in my log is:
Build step 'Audit job info to Database' marked build as failure
The side effects are maven not being installed on a slave, any of my build steps executing, and the job fails immediately after checking out source code. If I remove the audit step, everything works fine. if I add it back in, reverts to failure.
I've checked my connection from the main config page, and it works fine. I have jobs that are successfully logging to the audit db, and others that are randomly failing.
What's odd is the order in which the failures are listed:
> /usr/bin/git rev-list 12345678xxxx # timeout=10
No emails were triggered.
Skipping sonar analysis due to bad build status FAILURE
[BFA] Scanning build for known causes...
[BFA] No failure causes found
[BFA] Done. 0s
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Sending email to: xxxxx
Build step 'Audit job info to Database' marked build as failure
Finished: FAILURE
any thoughts?
Probably an error during checkout. My suspicion is a terrible connection to the repo so the checkout runs into the timeout. Configure timeouts in advanced checkout behaviors.