Code changed in jenkins
User: Daniel Beck
Path:
src/main/resources/artifact-ignores.properties
http://jenkins-ci.org/commit/backend-update-center2/3c532bdbf62c389fc5125c97c52f94dd5a702917
Log:
Merge pull request #183 from oleg-nenashev/patch-2

JENKINS-48357 - Fix the JIRA 2.5.1 blacklist string

Compare: https://github.com/jenkins-infra/backend-update-center2/compare/343ff422ca2c...3c532bdbf62c

perhaps a cheap way to fix this is just copy the code from jira-rest-api to the jira-plugin and set the dependencies we want?

We can do that. In the mean time, we should make dependency on jira-plugin optional and make sure blueocean-jira-plugin degrades without blowing up. Either that or make dependency on blueocean-jira-plugin optional?

vivek
Will this really fix https://issues.jenkins-ci.org/browse/JENKINS-43106 ?
I have a terrible problem with git rev-list which hangs on latest Jenkins and Blue Ocean and MB Pipeline. How the Apache client may cause this native Git (git rev-list) problem if I configured Git Plugin in Jenkins Management to use native Git?

tibor17 it likely would be related yes. Disable the JIRA plugin and try if you like. 

..but which jira plugin to disable?
I hope you do not mean MB Pipeline plugin because our company has the biggest benefit from it.

michaelneale
When this issue will be fixed and why it takes time?

tibor17 firstly no one is quite sure how it happens, and secondly the JIRA plugin has to basically be rewritten to not use atlassians libraries

No - just the "jira" plugin should do - https://plugins.jenkins.io/jira

I thought Jenkins was using OSGi and so that plugins run in isolated ClassLoaders.

tibor17 definitely not using OSGi, which probably could prevent things like this!

michaelneale
The influence of JIRA plugin to Git executable would be some environment variable.
If you run with/out JIRA plugin, the difference in the environment variables is found.
My problem that Git rev-list hangs happens only if I force push to a branch and I push back to master.

tibor17 it may be unrelated in that case. 

I tried to execute exactly the same command by hands and it did not hang.
So the case would be that the command displayed in the logs is not exactly identical to the executed one, or the environment variables was added by JIRA plugin.

I will try to list --global, --local, --system properties in Jenkinsfile and I will compare with my workspace. Next week because now I am on holiday.
It's strange to me that Java and Git executable have anything related. Therefore I mentioned env vars because I see only those in common and transaferable.

Blueocean JIRA integration is causing lots of grief among users. For whats worth, we are disabling blueocean JIRA integration temporarily - blueocean-jira plugin won't be shipped in 1.5.0 release.

We re-opened https://issues.jenkins-ci.org/browse/JENKINS-45920 to do a proper fix in jira-plugin and then re-enable JIRA integration.

Users who have been experience this issue should do the following to fix the binary compatibility issue with jira plugin:

• Upgrade to 1.5.0 release (not released yet, should go out sometime this week)
• From update center uninstall blueocean-jira 1.4.x plugin
• From update center uninstall jira plugin

1.4.x users who are using JIRA integration without compatibility problem can continue doing so, however we recommend uninstalling blueocean-jira and jira plugin after upgrading to 1.5.0 to avoid encountering jira binary compatibility problem.

vivek michaelneale exclusion from BlueOcean does not fix this issue (binary compat between JIRA Plugin and Apache HttpComponents 4.x client). Yes it fixes BlueOcean on instances without JIRA plugin, but all users who have JIRA plugin are still affected. Independently from BlueOcean...

oleg_nenashev very true - I think vivek was going to open a follow on ticket to fix up the JIRA plugin itself as a next step (inline the Atlassian stuff to not have that problem) - but this could be the ticket for that, given it doesn't explicitly mention blue ocean. 

Taking the number of voters/watchers, it's better to keep this ticket IMHO.
BlueOcean patch was never a fix for this problem as we discussed before

My problem was with the JIRA plugin.

I recently upgraded to jenkins 2.107.2 and the communication problem has disappeared.

ldevos What version of jira plugin was giving you the problem described in this ticket? so the only change on your side was upgrade to 2.107.2?

I'm using version 2.5 of the plugin. The problem was resolved after upgrading to 2.107.2 and updating all of my other plugins that had a new version.

I am using Version 2.107.2, restarted of course and using JIRA plugins, all plugins updated.
But the git still hangs on this line in logs:
16:25:09 > git rev-list --no-walk 3bb1f0fc92a0cdb29c6df7cd1eb109dbd372929d # timeout=10
Who prints this? The Git or Java?
Can you add more debug info? We have to find the root cause.
Interesting is that I pushed changes to Jenkinsfile with two commits in the morning. And now in the afternoon I committed Java code. Could it be that Jenkinsfile represents configuration of the job and maybe this behavior (Jenkinsfile -> next commit with some other file(s)) breaks something.
Now I have only one possibility to interrupt the build and trigger the build manually. After I did the Jenkins job did not hang.
Could it be related to what Jenkinsfile made in job configuration. I don't think it's my fault because I only renamed function isTrunk to isMaster which is easy refactoring.

tibor17 the message

16:25:09 > git rev-list --no-walk 3bb1f0fc92a0cdb29c6df7cd1eb109dbd372929d # timeout=10

is logged by the git client plugin.

I suspect that the git plugin is not the cause of the hang. I can't see the hang myself, so have no way to diagnose it. I've submitted a git client plugin pull request that logs the conclusion of a command line git call so that we can see if the `git rev-list` command hangs or if execution proceeds beyond that point.

You'll need to download and install the git-client-plugin.hpi from the pull request artifacts.

markewaite
Hi Mark,
I found some hint [1] however I use Ubuntu and not Windows, but this forced me to ask you about git credentials and some other question. Maybe the pull request can be extended with some more logs.
How credentials are passed to these commands and each of them?
Can we see them and is this real command executed in sub-process?
Is Git running in a non-interactive mode?
Can we detect that Git is interactively asking for some input on STD/IN? I guess Jenkins plugin runs Git in Java sub-process.
Can we see std/err as well as std/out in the console including the error code returned by the sub-process?
Why timeout 10 minutes is not applied however it is seen in the console log after the command (# timeout=10)?
[1]: https://support.cloudbees.com/hc/en-us/articles/221046888-Build-Hang-or-Fail-with-Git-for-Windows

markewaite
I have a new findings (yet without your pull request - I will use it soon).
The Git plugin hangs on git rev-list --no-walk if the time trigger started the job. It has never been hanged when I triggered manually. Not related to changes in Jenkinsfile as I wrote before.
Interesting is that I launched the native commit in WORKSPACE/build which did not hang. Notice that my .git directory is in build directory and not directly in workspace.
Not sure why Jenkins and Git plugin calls this command due to it always returns the commit hash itself I used in the command:
git rev-list --no-walk 21980ad7f91ab4eb0aeb09b166d66baf7433b7a9
dioes not matter if I checked out the HEAD or some previous commit.

tibor17 thanks for investigating further. Your efforts are appreciated very much!

Your questions with my answers following each question:

How credentials are passed to these commands and each of them?

The git rev-list command does not require credentials. None are provided to it.

Commands which need credentials like fetch and ls-remote receive them through temporary files identified by environment variables.

Credentials for http and https repository URL's are written to a temporary file in the form of a response script. The name of the temporary file is assigned to the environment variables GIT_ASKPASS and SSH_ASKPASS. Command line git calls that temporary file and the temporary file responds to the command line git prompts for username and password.

Credentials for ssh repository URL's are written to a temporary file. The name of the temporary file is assigned to the environment variable SSH_ASKPASS. The path to the ssh command is assigned to the environment variable GIT_SSH. The value ssh is assigned to the environment variable GIT_SSH_VARIANT. Command line git uses those environment variables to find the temporary files for authentication.

Can we see them and is this real command executed in sub-process?

No, they are not shown to users. Showing them to users would expose credentials to users who should not see them.

Yes, command line git executes in a subprocess of the Jenkins Java process. The temporary file is executed in a subprocess of command line git so that it can provide the credential responses (http/https) or the ssh connection (ssh).

Is Git running in a non-interactive mode?

The git plugin attempts to run git in non-interactive mode if the command line git version is new enough to support non-interactive mode.

For command line git versions 2.3 and newer, the git plugin sets the environment variable GIT_TERMINAL_PROMPT=false. If running on Windows, it also sets the environment variable GCM_INTERACTIVE=false.

Can we detect that Git is interactively asking for some input on STD/IN?

The plugin does not detect that git is interactively asking for input on stdin. I don't know any technique to do that. 

The timeout option was added to commands which require authentication specifically because there was not a way to detect command line git was waiting for input.

Can we see std/err as well as std/out in the console including the error code returned by the sub-process?

Yes, we can see stdout and stderr outputs and the subprocess error code. When the git plugin prints an exception from command line git, it includes stderr and stdout of the command in the exception message.

Why timeout 10 minutes is not applied however it is seen in the console log after the command (# timeout=10)?

I suspect that the command completed and control has returned from the git plugin to some other part of Jenkins. The timeout was not applied because command line git had returned and job execution continued in other parts of Jenkins.  The diagnostics I inserted into the patched plugin will confirm or refute that suspicion.

git rev-list --no-walk 21980ad7f91ab4eb0aeb09b166d66baf7433b7a9

I found when I had JIRA plugin and JiraTestResultReporter plugin enabled and used a JIRA ticket number in the commit message it would hang.

JENKINS-48357 This is a commit that would hang.

This would not.

I am not sure it is related, but it seems so.

michaelpporter, if you're able to explore further in your failure case, could you download and install the git-client-plugin.hpi from the pull request artifacts.

That version of the git client plugin adds an additional logging statement which shows that command line git has completed.  That will allow us to confirm that git rev-list is not hanging.  It doesn't help us identify what is hanging, but it at least allows us to remove the git rev-list call from the list of suspects.

4 solutions:

1. upgrade to atlassian jira client 5.0.4 which is using Apache httpclient 4.5.3 (but might cause issue in the future as well)
2. import sources from atlassian source code which is possible as it's Apache license.
3. build our own reusable (as simple as possible) jira client
4. use an other jira client such https://github.com/rcarz/jira-client (but it's LGPL)

2 might be real nightmare... as the jira client from Atlassian contains a lot of abstraction layers on the top of Apache httpclient..... Furthermore it use an other abstraction layer for Promise on the top of Guava....
So definitely 2 might tend to an over complicated maintenance... as we need to copy jira client sources, atlassian http client sources etc...

So 1 is a possible temporary solution. ( vivek ?) 

3 is an other possible option.

olamy Option 1 doesn't work as user won't be able to upgrade to later version of jira plugin.

Option 3 can work if we include support for Jira site configuration as well. However everyone out there know about jira plugin and if they upgrade to later version its going to be problem.

How about we do #2 and do in a way as you plan option 3? That is keep external interfaces exposed by jira-plugin intact and throw away rest stuff that interacts with atlassian libraries and replace it by something simple?

go for it #2 with only the minimum JIRA  services we need I will try to do this not too messy  

too bad access denied to https://bitbucket.org/atlassian/atlassian-httpclient whereas the pom is here https://packages.atlassian.com/maven-external/com/atlassian/httpclient/atlassian-httpclient-parent/0.23.0/

olamy are you looking for https://bitbucket.org/atlassian/jira-rest-java-client? Its multi-module with sources for api api and core.

olamy there is also https://bitbucket.org/atlassian/atlassian-http.

got the sources from maven repo. I don't mind the rest client I just to override the Atlassian httpclient ( https://packages.atlassian.com/maven-external/com/atlassian/httpclient/atlassian-httpclient-plugin/0.23.0/ ) 

The goal is to have a local copy of those classes so we can simply have our version of Apache httpclient.

I have it done locally, I have to do more testing (separate Jenkins install with bo and real JIRA etc...)

pr https://github.com/jenkinsci/jira-plugin/pull/145

snapshot available for test https://repo.jenkins-ci.org/snapshots/org/jenkins-ci/plugins/jira/2.5.2-SNAPSHOT/jira-2.5.2-20180501.075458-1.hpi

warden Olivier has PR up and snapshot available at https://repo.jenkins-ci.org/snapshots/org/jenkins-ci/plugins/jira/2.5.2-SNAPSHOT/jira-2.5.2-20180501.231733-2.hpi, mind giving it a try?

markewaite

The link you provided [ git-client-plugin.hpi |https://ci.jenkins.io/job/Plugins/job/git-client-plugin/view/change-requests/job/PR-311/lastSuccessfulBuild/artifact/target/git-client.hpi] is no longer reachable. I want to check it out. Can you create it again pls? I have updated the Jira Plugin from Olivier and I would like to do the same with the Git Plugin.

tibor17 thanks for your willingness to test.  Download git client plugin from the artifact repository.

markewaite
The first commit did not hang and this is the log:

16:11:24 [WS-CLEANUP] Deleting project workspace...[WS-CLEANUP] done
[Pipeline] }
[Pipeline] // stage
[Pipeline] echo
16:11:25 shortDescription: Branch indexing
[Pipeline] echo
16:11:25 Job started by timer or branching.
[Pipeline] sh
16:11:25 [ibranch_Pipeline_playground-PRVK7X6T2XH7C3SUJ4JNYHLNG2EVZ6W7C37SOIOU6LWELPYIKHXQ] Running shell script
16:11:25 + mkdir -p build
[Pipeline] dir
16:11:25 Running in /var/lib/jenkins/workspace/ibranch_Pipeline_playground-PRVK7X6T2XH7C3SUJ4JNYHLNG2EVZ6W7C37SOIOU6LWELPYIKHXQ/build
[Pipeline] {
[Pipeline] stage
[Pipeline] { (checkout)
[Pipeline] checkout
16:11:25 Cloning the remote Git repository
16:11:25 Cloning with configured refspecs honoured and without tags
16:11:25 Cloning repository http://git.ads.local/phfa-entervonextgen/audit
16:11:25  > git init /var/lib/jenkins/workspace/ibranch_Pipeline_playground-PRVK7X6T2XH7C3SUJ4JNYHLNG2EVZ6W7C37SOIOU6LWELPYIKHXQ/build # timeout=10
16:11:26  > FINISHED git init /var/lib/jenkins/workspace/ibranch_Pipeline_playground-PRVK7X6T2XH7C3SUJ4JNYHLNG2EVZ6W7C37SOIOU6LWELPYIKHXQ/build # timeout=10
16:11:26 Fetching upstream changes from http://git.ads.local/phfa-entervonextgen/audit
16:11:26  > git --version # timeout=10
16:11:26  > FINISHED git --version # timeout=10
16:11:26 using GIT_ASKPASS to set credentials git private new rsa
16:11:26  > git fetch --no-tags --progress http://git.ads.local/phfa-entervonextgen/audit +refs/heads/*:refs/remotes/origin/*
16:11:30  > FINISHED git fetch --no-tags --progress http://git.ads.local/phfa-entervonextgen/audit +refs/heads/*:refs/remotes/origin/*
16:11:30  > git config remote.origin.url http://git.ads.local/phfa-entervonextgen/audit # timeout=10
16:11:30  > FINISHED git config remote.origin.url http://git.ads.local/phfa-entervonextgen/audit # timeout=10
16:11:30  > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
16:11:30  > FINISHED git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
16:11:30  > git config remote.origin.url http://git.ads.local/phfa-entervonextgen/audit # timeout=10
16:11:30  > FINISHED git config remote.origin.url http://git.ads.local/phfa-entervonextgen/audit # timeout=10
16:11:30 Fetching without tags
16:11:30 Fetching upstream changes from http://git.ads.local/phfa-entervonextgen/audit
16:11:30 using GIT_ASKPASS to set credentials git private new rsa
16:11:30  > git fetch --no-tags --progress http://git.ads.local/phfa-entervonextgen/audit +refs/heads/*:refs/remotes/origin/*
16:11:30  > FINISHED git fetch --no-tags --progress http://git.ads.local/phfa-entervonextgen/audit +refs/heads/*:refs/remotes/origin/*
16:11:30 Checking out Revision 512f79c739a04d5001d0c89dc889add0efbf06b9 (playground)
16:11:30  > git config core.sparsecheckout # timeout=10
16:11:30  > FINISHED git config core.sparsecheckout # timeout=10
16:11:30  > git checkout -f 512f79c739a04d5001d0c89dc889add0efbf06b9
16:11:31  > FINISHED git checkout -f 512f79c739a04d5001d0c89dc889add0efbf06b9
16:11:31 Commit message: "Test new Git and JIRA SNAPSHOT plugin with maven-site-plugin:3.7.1"
16:11:31  > git rev-list --no-walk 1535e85a8421d931ea2f54cd26dff47f647ed24b # timeout=10
16:11:31  > FINISHED git rev-list --no-walk 1535e85a8421d931ea2f54cd26dff47f647ed24b # timeout=10
[Pipeline] sh
16:11:31 

Thanks tibor17. The most interesting is the output the next time you see it hang.

That unreleased plugin build is based on the git client plugin 3.0.0-beta. If that unreleased plugin version resolves all the instances of hanging in your environment then I'll probably need to build a different unreleased plugin version which contains exactly git client plugin 2.7.1 plus that additional logging statement.

Rather than risk further delay, if you need git client plugin 2.7.1 plus the extra 'FINISHED' logging statements, download git client plugin 2.7.2-SNAPSHOT unreleased

olamy
Maybe OSGi would be quite a big deal for the developers. Would it be a generic solution of such problems in all Jenkins plugins?
Each Jenkins plugin may have isolated version of JIRA Client and Apache HTTP client.

markewaite
Ok Mark, just ping me when you need my test again. I will try to make few more commits next days on my system.

tibor17 I provided the extra logging in the git client plugin so that we could confirm my belief that what you're reporting as "hang in git rev-list" is not actually in the "git rev-list" command or even in the git client plugin that executes "git rev-list".

I can't duplicate this problem in my environment. The extra logging in the git client plugin is intended to help you confirm (or refute) my hypothesis that the problem is happening somewhere other than the git client plugin.

If prior to installing the unreleased git client plugin you saw 100% of the executions of that job hang, and the unreleased git client plugin never hangs, then you should switch to the unreleased git client plugin 2.7.2 to reduce the changes between the system that hangs and the system that never hangs.

olamy
markewaite
I updated both plugins Git Client and JIRA. So it means the change in JIRA was the one which made the build to continue without hanging. The second commit runs as well.

tibor17 Thanks for confirming jira plugin fixes the issue. olamy another data point to go ahead with merge and release jira plugin

olamy
vivek
After I have downgraded Git Client back to 2.7.1 release and still used Olivier's SNAPSHOT of JIRA plugin the build did not hang, hopefully it's fine.

Code changed in jenkins
User: Olivier Lamy
Path:
README.md
pom.xml
src/main/java/com/atlassian/httpclient/apache/httpcomponents/ApacheAsyncHttpClient.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/CommonBuilder.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/DefaultHttpClientFactory.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/DefaultMessage.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/DefaultRequest.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/DefaultResponse.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/EntityByteArrayInputStream.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/Headers.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/MavenUtils.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/PromiseHttpAsyncClient.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/RedirectStrategy.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/RequestEntityEffect.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/SettableFuturePromiseHttpPromiseAsyncClient.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/proxy/ProvidedProxyConfig.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/proxy/ProxyConfig.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/proxy/ProxyConfigFactory.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/proxy/ProxyCredentialsProvider.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/proxy/SystemPropertiesProxyConfig.java
src/main/java/com/atlassian/httpclient/base/event/AbstractHttpRequestEvent.java
src/main/java/com/atlassian/httpclient/base/event/HttpRequestCompletedEvent.java
src/main/java/com/atlassian/httpclient/base/event/HttpRequestFailedEvent.java
src/main/resources/atlassian-httpclient-plugin-0.23.0.pom
http://jenkins-ci.org/commit/jira-plugin/abd092848b1e617ceb31b16de07b1445d34c4d03
Log:
JENKINS-48357 get rid of atlassian http client (#145)

JENKINS-48357

• import sources from Atlassian Async HttpClient so we can use whatever we want as apache httpclient version

Signed-off-by: olivier lamy <olamy@apache.org>

• remove non used imported classes

Signed-off-by: olivier lamy <olamy@apache.org>

• get rid of this class

Signed-off-by: olivier lamy <olamy@apache.org>

• modern

Signed-off-by: olivier lamy <olamy@apache.org>

• add note on Atlassian sources import

Signed-off-by: olivier lamy <olamy@apache.org>

• add Atlassian pom for history purpose

Signed-off-by: olivier lamy <olamy@apache.org>

• JENKINS-48357 get rid of non used class

Signed-off-by: olivier lamy <olamy@apache.org>

• get rid of this httpasyncclient-cache

Signed-off-by: olivier lamy <olamy@apache.org>

• use last apache-httpcomponents-client-4-api 4.5.5-2.0 which include asynchttpclient now

Signed-off-by: olivier lamy <olamy@apache.org>

• add back jenkins-ci repo

Signed-off-by: olivier lamy <olamy@apache.org>

• exclude atlassian classes from coverage

Signed-off-by: olivier lamy <olamy@apache.org>

*NOTE:* This service been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/

Functionality will be removed from GitHub.com on January 31st, 2019.

olamy
Now my build does not hang but I have a new issue with Jira plugin because when I commit with messages which matches a pattern, the Jira plugin times out immediately. Maybe in less than one second. In the Management are two sections: JIRA and JIRA Sites. The first has successful login but the second fails. Why this happens?