Apparently warden has released the master branch. Now we have the wider binary compatibility issue in the wild. E.g. JENKINS-49208.

I will see whether we can fix it without using a pluginFirstClassLoader (which is not reliable)

Hi oleg_nenashev, I've finally traced the root cause.

jira-rest-java-client-core need by Jira-plugin depends on httpcomponents-asyncclient which transitively depends on httpclient-cache. The class org.apache.http.impl.client.cache.CachingHttpAsyncClient tries to instantiate package visible class org.apache.http.impl.client.cache.BasicHttpCache. It is OK in normal case, but fails if CachingHttpAsyncClient and BasicHttpCache are loaded from different classloaders, becoming two different run-time packages, causing throwing java.lang.IllegalAccessError.

We unfortunately encounter this abnormal case, that Jira-plugin depends on httpcomponents-asyncclient loaded by plugin classloader and httpclient-cache loaded by another(which loads apache-httpcomponents-client-4-api).

So, it comes the solution. The simplest one, remove dependency of apache-httpcomponents-client-4-api, as warden mentioned above. Or we can add httpcomponents-asyncclient dependency into apache-httpcomponents-client-4-api.

 refer:

1. https://jenkins.io/doc/developer/plugin-development/dependencies-and-class-loading/
2. https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-5.html#jvms-5.3

gmshake yes, it was my plan as well. I have already started prototyping the Async Http Client plugin for that yesterday. There is already https://github.com/jenkinsci/async-http-client-plugin, but it uses another library.

Ideally I would be happy to get rid of JRJC at all since it's not well maintained anymore, but it requires a serious plugin update

danielbeck warden My proposal would be to blacklist 2.5.1 in the update center for now (see the reported issue).
michaelneale jamesdumay do you plan to finalize the fix?

+1, that's the simpler scenario - any write-up how to do that? I don't see anything on confluence...
Alternatively, I could do a 2.5.2 hotfix release from a separate branch with cherry-picked commits w/o httpclient changes.

warden the problem is that the plugin collides with other plugins which use new Apache HttpComponents Client 4.x API Plugin or which bundle the httpclient-cache library directly. E.g. JENKINS-49208 which happens with JIRA Plugin 2.3. Just reverting the fix won't help though it would make the plugin self-consistent at least

oleg_nenashev may make sense to blacklist if it is that bad. Rolling forward with a fix (from here I guess?) will make sense but I don't think jamesdumay will be around this week to look at it. 

Can I just confirm that this is the same bug I'm experiencing in the below stacktrace since upgrading to 2.5.1 :

WARNING: Step ‘JIRA: Update relevant issues’ aborted due to exception: 
java.lang.IllegalAccessError: tried to access class org.apache.http.impl.client.cache.BasicHttpCache from class org.apache.http.impl.client.cache.CachingHttpAsyncClient
	at org.apache.http.impl.client.cache.CachingHttpAsyncClient.<init>(CachingHttpAsyncClient.java:173)
	at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.<init>(ApacheAsyncHttpClient.java:239)
	at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.<init>(ApacheAsyncHttpClient.java:123)
	at com.atlassian.httpclient.apache.httpcomponents.DefaultHttpClientFactory.doCreate(DefaultHttpClientFactory.java:68)
	at com.atlassian.httpclient.apache.httpcomponents.DefaultHttpClientFactory.create(DefaultHttpClientFactory.java:35)
	at com.atlassian.jira.rest.client.internal.async.AsynchronousHttpClientFactory.createClient(AsynchronousHttpClientFactory.java:63)
	at com.atlassian.jira.rest.client.internal.async.AsynchronousJiraRestClientFactory.create(AsynchronousJiraRestClientFactory.java:35)
	at com.atlassian.jira.rest.client.internal.async.AsynchronousJiraRestClientFactory.createWithBasicHttpAuthentication(AsynchronousJiraRestClientFactory.java:42)
	at hudson.plugins.jira.JiraSite.createSession(JiraSite.java:303)
	at hudson.plugins.jira.JiraSite.getSession(JiraSite.java:278)
	at hudson.plugins.jira.Updater.perform(Updater.java:95)
	at hudson.plugins.jira.JiraIssueUpdater.perform(JiraIssueUpdater.java:64)
	at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
	at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:45)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
	at hudson.model.Build$BuildExecution.post2(Build.java:186)
	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
	at hudson.model.Run.execute(Run.java:1749)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:421)

... & also what is reported in JENKINS-49214

Is it expected behaviour that an error in a post-build step, causes a build to fail that otherwise passed 100%?

Yes, it is the same issue.

My proposals:

• Create a new "Apache HttpComponents Async Client API Plugin", which depends on Apache HttpComponents Client 4.x API Plugin
• Update JIRA plugin to use the new plugin and exclude dependencies

I am not fully convinced it's going to solve all the issues, but at least it will solve Async Client Compat issues reported above

I confirmed that the issue happens on the 2.0 release of JIRA...

Since we're getting more bugs, oleg_nenashev are you able to blacklist 2.5.1 or point me to some how-to for doing it?

warden Example: https://github.com/jenkins-infra/backend-update-center2/pull/167/files

oleg_nenashev so the blacklist won't necessarily help? (but your client api plugin should help)

jamesdumay this seems to be causing a lot of pain - is the root of it the Atlassian dependencies (eg JIRA rest api) which does unpleasant things? is there any solution to this on the JIRA plugin side? 

It is almost like a re-write of the api side or shading of things needs to be done, given the pain caused (I think) by these external dependencies. THis plugin is not in a good state. 

Code changed in jenkins
User: Radek Antoniuk
Path:
src/main/resources/artifact-ignores.properties
http://jenkins-ci.org/commit/backend-update-center2/2f8d6be099d1ef9a64d0d32b67f2713b5ad54dea
Log:
JENKINS-48357 - blacklist jira-plugin 2.5.1 because of Apache HTTP dependencies conflict

Code changed in jenkins
User: Daniel Beck
Path:
src/main/resources/artifact-ignores.properties
http://jenkins-ci.org/commit/backend-update-center2/343ff422ca2c3c1d1c39506c55ac81033c54b442
Log:
Merge pull request #182 from warden/JENKINS-48357-blacklist-2.5.1

JENKINS-48357 - blacklist jira-plugin 2.5.1

Compare: https://github.com/jenkins-infra/backend-update-center2/compare/b3e36b8b66c6...343ff422ca2c

Hey Guys,

any ETA when it will get fixed?

ariele well in the short term - 2.5.1 was blacklisted from update centre which eases things for some people. 

Otherwise there is some action in https://github.com/jenkinsci/jira-plugin/pull/139 where gmshake is suggesting a newer version of the Atlassian libraries  - but not sure what the real solution is. The problem is with the Atlassian libraries and their dependencies and use of maven, not the JIRA plugin (which depends on it - although does make me think maybe it should not use them as they seem to cause so many problems). 

michaelneale

as it seems, the issue of the JIRA Plugin is with the plugin "Apache HttpComponents Client 4.x API Plugin".
currently i have the latest version : 4.5.3-2.1

if i use other JIRA plugins that are using user/password, then it works just fine.

for example: JIRA Pipeline Steps plugin is able to connect.

ariele I'm getting confused by 

if i use other JIRA plugins that are using user/password, then it works just fine.

.

Do you mean use other versions of JIRA plugins or something else?

gmshake

for example: JIRA Pipeline Steps plugin is able to connect.

ariele Got it! Thanks for that  

gmshake
happy to help, but can you please share the ETA for this fix once you have it?

ariele Sorry but no ETA from my perspective since I'm not the maintainer. Could you please rollback JIRA plugin from 2.5.1 to 2.5 or try the latest snapshot version with fix https://ci.jenkins.io/job/Plugins/job/jira-plugin/job/PR-139/2/artifact/target/jira.hpi ?

Some issues will be still happening on 2.5. E.g. the original report from jamesdumay in this ticket and JENKINS-49208.

gmshake
i didnt upgrade from 2.5.0 to 2.5.1.

i just installed from fresh - version 2.5.1

how can i downgrade it?

ariele

Good question !
Maybe uninstall 2.5.1 first and then reinstall 2.5 ?
I've not tried that yet

It seems like JIRA plugin 2.5.1 is still in update center at this moment. warden Can you please confirm that?

gmshake warden
guys, we really need a fix for this, its CRITICAL.
if version 2.5.0 is good, please revert it from the update center.

Thanks,

oleg_nenashev Sorry I missed the original report.

ariele there is an architecture issue in the plugin. All versions of JIRA plugin are generally affected. Using caps in messages does not help. If you are ready to contribute your time and help to get it fixed (like gmshake does), it would really help.

oleg_nenashev
i'd be happy too, the caps are just to mention that its impact on our business.
from my point of view, we should revert the store to version 2.5.
if you can share with me on how to make it happen, i will be happy to do it.

if its simply to go to the github and revert to 2.5 - i'll do it.  

ariele It should have been already removed from UC, but there was a mistake in the blacklisting file. Created https://github.com/jenkins-infra/backend-update-center2/pull/183

oleg_nenashev
i see, so what does it mean?

that should be switched to 2.5?

When aforementioned PR is merged, then in UC you'll see only 2.5.
Regardless, you should be able to go to GH and pull 2.5 and install manually.

warden
i always install via jenkins, never did it manually.
i can pull the repo but how can i install it?

1. http://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/jira/2.5/
2. download jira-2.5.hpi
3. install via `pluginManager/advanced` / uploadPlugin

ariele In plugin manager there is an Advanced section, which allows uploading a plugin.
HPIs for old versions can be downloaded from here: https://updates.jenkins.io/download/plugins/jira/

oleg_nenashev
sent above the page i get...where can i find the download action?

"Upload plugin".
Download links are not in Jenkins, see the responses above

oleg_nenashev
sorry, missed the comment above.
checking it now
thanks.

oleg_nenashev
upon installing jira-2.5.hpi
connection is working.

although i suspect methods in that version are still not working, i will advise if there is any problem.
thanks for now.

oleg_nenashev
so i'm back with my original issue i had (didnt report it).
post build job: "JIRA: Move issues matching JQL to a specified version" is not doing anything.
not giving any logs.

is there any known issue with that?

As you noticed this issue is handling different problem and there are many watchers of the ticket, so let's try not to spam them
Please search through issues in https://issues.jenkins-ci.org/issues/?jql=project+%3D+JENKINS+AND+component+%3D+jira-plugin; if it's not there, please report a new bug.
Also feel free to dig into the problem and contribute by creating a Pull Request

warden
sure thing...thanks for the help.

Code changed in jenkins
User: Oleg Nenashev
Path:
src/main/resources/artifact-ignores.properties
http://jenkins-ci.org/commit/backend-update-center2/47580ef481d5c103bfcd5c9943d604029a151ebe
Log:
JENKINS-48357 - Fix the JIRA 2.5.1 blacklist string

Code changed in jenkins
User: Daniel Beck
Path:
src/main/resources/artifact-ignores.properties
http://jenkins-ci.org/commit/backend-update-center2/3c532bdbf62c389fc5125c97c52f94dd5a702917
Log:
Merge pull request #183 from oleg-nenashev/patch-2

JENKINS-48357 - Fix the JIRA 2.5.1 blacklist string

Compare: https://github.com/jenkins-infra/backend-update-center2/compare/343ff422ca2c...3c532bdbf62c

perhaps a cheap way to fix this is just copy the code from jira-rest-api to the jira-plugin and set the dependencies we want?

We can do that. In the mean time, we should make dependency on jira-plugin optional and make sure blueocean-jira-plugin degrades without blowing up. Either that or make dependency on blueocean-jira-plugin optional?

vivek
Will this really fix https://issues.jenkins-ci.org/browse/JENKINS-43106 ?
I have a terrible problem with git rev-list which hangs on latest Jenkins and Blue Ocean and MB Pipeline. How the Apache client may cause this native Git (git rev-list) problem if I configured Git Plugin in Jenkins Management to use native Git?

..but which jira plugin to disable?
I hope you do not mean MB Pipeline plugin because our company has the biggest benefit from it.

michaelneale
When this issue will be fixed and why it takes time?

tibor17 firstly no one is quite sure how it happens, and secondly the JIRA plugin has to basically be rewritten to not use atlassians libraries

No - just the "jira" plugin should do - https://plugins.jenkins.io/jira

I thought Jenkins was using OSGi and so that plugins run in isolated ClassLoaders.

tibor17 definitely not using OSGi, which probably could prevent things like this!

michaelneale
The influence of JIRA plugin to Git executable would be some environment variable.
If you run with/out JIRA plugin, the difference in the environment variables is found.
My problem that Git rev-list hangs happens only if I force push to a branch and I push back to master.

tibor17 it may be unrelated in that case. 

I tried to execute exactly the same command by hands and it did not hang.
So the case would be that the command displayed in the logs is not exactly identical to the executed one, or the environment variables was added by JIRA plugin.

I will try to list --global, --local, --system properties in Jenkinsfile and I will compare with my workspace. Next week because now I am on holiday.
It's strange to me that Java and Git executable have anything related. Therefore I mentioned env vars because I see only those in common and transaferable.

Blueocean JIRA integration is causing lots of grief among users. For whats worth, we are disabling blueocean JIRA integration temporarily - blueocean-jira plugin won't be shipped in 1.5.0 release.

We re-opened https://issues.jenkins-ci.org/browse/JENKINS-45920 to do a proper fix in jira-plugin and then re-enable JIRA integration.

Users who have been experience this issue should do the following to fix the binary compatibility issue with jira plugin:

• Upgrade to 1.5.0 release (not released yet, should go out sometime this week)
• From update center uninstall blueocean-jira 1.4.x plugin
• From update center uninstall jira plugin

1.4.x users who are using JIRA integration without compatibility problem can continue doing so, however we recommend uninstalling blueocean-jira and jira plugin after upgrading to 1.5.0 to avoid encountering jira binary compatibility problem.

vivek michaelneale exclusion from BlueOcean does not fix this issue (binary compat between JIRA Plugin and Apache HttpComponents 4.x client). Yes it fixes BlueOcean on instances without JIRA plugin, but all users who have JIRA plugin are still affected. Independently from BlueOcean...

oleg_nenashev very true - I think vivek was going to open a follow on ticket to fix up the JIRA plugin itself as a next step (inline the Atlassian stuff to not have that problem) - but this could be the ticket for that, given it doesn't explicitly mention blue ocean. 

Taking the number of voters/watchers, it's better to keep this ticket IMHO.
BlueOcean patch was never a fix for this problem as we discussed before

My problem was with the JIRA plugin.

I recently upgraded to jenkins 2.107.2 and the communication problem has disappeared.

ldevos What version of jira plugin was giving you the problem described in this ticket? so the only change on your side was upgrade to 2.107.2?

I'm using version 2.5 of the plugin. The problem was resolved after upgrading to 2.107.2 and updating all of my other plugins that had a new version.

I am using Version 2.107.2, restarted of course and using JIRA plugins, all plugins updated.
But the git still hangs on this line in logs:
16:25:09 > git rev-list --no-walk 3bb1f0fc92a0cdb29c6df7cd1eb109dbd372929d # timeout=10
Who prints this? The Git or Java?
Can you add more debug info? We have to find the root cause.
Interesting is that I pushed changes to Jenkinsfile with two commits in the morning. And now in the afternoon I committed Java code. Could it be that Jenkinsfile represents configuration of the job and maybe this behavior (Jenkinsfile -> next commit with some other file(s)) breaks something.
Now I have only one possibility to interrupt the build and trigger the build manually. After I did the Jenkins job did not hang.
Could it be related to what Jenkinsfile made in job configuration. I don't think it's my fault because I only renamed function isTrunk to isMaster which is easy refactoring.

tibor17 the message

16:25:09 > git rev-list --no-walk 3bb1f0fc92a0cdb29c6df7cd1eb109dbd372929d # timeout=10

is logged by the git client plugin.

I suspect that the git plugin is not the cause of the hang. I can't see the hang myself, so have no way to diagnose it. I've submitted a git client plugin pull request that logs the conclusion of a command line git call so that we can see if the `git rev-list` command hangs or if execution proceeds beyond that point.

You'll need to download and install the git-client-plugin.hpi from the pull request artifacts.

markewaite
Hi Mark,
I found some hint [1] however I use Ubuntu and not Windows, but this forced me to ask you about git credentials and some other question. Maybe the pull request can be extended with some more logs.
How credentials are passed to these commands and each of them?
Can we see them and is this real command executed in sub-process?
Is Git running in a non-interactive mode?
Can we detect that Git is interactively asking for some input on STD/IN? I guess Jenkins plugin runs Git in Java sub-process.
Can we see std/err as well as std/out in the console including the error code returned by the sub-process?
Why timeout 10 minutes is not applied however it is seen in the console log after the command (# timeout=10)?
[1]: https://support.cloudbees.com/hc/en-us/articles/221046888-Build-Hang-or-Fail-with-Git-for-Windows

markewaite
I have a new findings (yet without your pull request - I will use it soon).
The Git plugin hangs on git rev-list --no-walk if the time trigger started the job. It has never been hanged when I triggered manually. Not related to changes in Jenkinsfile as I wrote before.
Interesting is that I launched the native commit in WORKSPACE/build which did not hang. Notice that my .git directory is in build directory and not directly in workspace.
Not sure why Jenkins and Git plugin calls this command due to it always returns the commit hash itself I used in the command:
git rev-list --no-walk 21980ad7f91ab4eb0aeb09b166d66baf7433b7a9
dioes not matter if I checked out the HEAD or some previous commit.

tibor17 thanks for investigating further. Your efforts are appreciated very much!

Your questions with my answers following each question:

How credentials are passed to these commands and each of them?

The git rev-list command does not require credentials. None are provided to it.

Commands which need credentials like fetch and ls-remote receive them through temporary files identified by environment variables.

Credentials for http and https repository URL's are written to a temporary file in the form of a response script. The name of the temporary file is assigned to the environment variables GIT_ASKPASS and SSH_ASKPASS. Command line git calls that temporary file and the temporary file responds to the command line git prompts for username and password.

Credentials for ssh repository URL's are written to a temporary file. The name of the temporary file is assigned to the environment variable SSH_ASKPASS. The path to the ssh command is assigned to the environment variable GIT_SSH. The value ssh is assigned to the environment variable GIT_SSH_VARIANT. Command line git uses those environment variables to find the temporary files for authentication.

Can we see them and is this real command executed in sub-process?

No, they are not shown to users. Showing them to users would expose credentials to users who should not see them.

Yes, command line git executes in a subprocess of the Jenkins Java process. The temporary file is executed in a subprocess of command line git so that it can provide the credential responses (http/https) or the ssh connection (ssh).

Is Git running in a non-interactive mode?

The git plugin attempts to run git in non-interactive mode if the command line git version is new enough to support non-interactive mode.

For command line git versions 2.3 and newer, the git plugin sets the environment variable GIT_TERMINAL_PROMPT=false. If running on Windows, it also sets the environment variable GCM_INTERACTIVE=false.

Can we detect that Git is interactively asking for some input on STD/IN?

The plugin does not detect that git is interactively asking for input on stdin. I don't know any technique to do that. 

The timeout option was added to commands which require authentication specifically because there was not a way to detect command line git was waiting for input.

Can we see std/err as well as std/out in the console including the error code returned by the sub-process?

Yes, we can see stdout and stderr outputs and the subprocess error code. When the git plugin prints an exception from command line git, it includes stderr and stdout of the command in the exception message.

Why timeout 10 minutes is not applied however it is seen in the console log after the command (# timeout=10)?

I suspect that the command completed and control has returned from the git plugin to some other part of Jenkins. The timeout was not applied because command line git had returned and job execution continued in other parts of Jenkins.  The diagnostics I inserted into the patched plugin will confirm or refute that suspicion.

git rev-list --no-walk 21980ad7f91ab4eb0aeb09b166d66baf7433b7a9

I found when I had JIRA plugin and JiraTestResultReporter plugin enabled and used a JIRA ticket number in the commit message it would hang.

JENKINS-48357 This is a commit that would hang.

This would not.

I am not sure it is related, but it seems so.

michaelpporter, if you're able to explore further in your failure case, could you download and install the git-client-plugin.hpi from the pull request artifacts.

That version of the git client plugin adds an additional logging statement which shows that command line git has completed.  That will allow us to confirm that git rev-list is not hanging.  It doesn't help us identify what is hanging, but it at least allows us to remove the git rev-list call from the list of suspects.

4 solutions:

1. upgrade to atlassian jira client 5.0.4 which is using Apache httpclient 4.5.3 (but might cause issue in the future as well)
2. import sources from atlassian source code which is possible as it's Apache license.
3. build our own reusable (as simple as possible) jira client
4. use an other jira client such https://github.com/rcarz/jira-client (but it's LGPL)

2 might be real nightmare... as the jira client from Atlassian contains a lot of abstraction layers on the top of Apache httpclient..... Furthermore it use an other abstraction layer for Promise on the top of Guava....
So definitely 2 might tend to an over complicated maintenance... as we need to copy jira client sources, atlassian http client sources etc...

So 1 is a possible temporary solution. ( vivek ?) 

3 is an other possible option.

olamy Option 1 doesn't work as user won't be able to upgrade to later version of jira plugin.

Option 3 can work if we include support for Jira site configuration as well. However everyone out there know about jira plugin and if they upgrade to later version its going to be problem.

How about we do #2 and do in a way as you plan option 3? That is keep external interfaces exposed by jira-plugin intact and throw away rest stuff that interacts with atlassian libraries and replace it by something simple?

go for it #2 with only the minimum JIRA  services we need I will try to do this not too messy  

too bad access denied to https://bitbucket.org/atlassian/atlassian-httpclient whereas the pom is here https://packages.atlassian.com/maven-external/com/atlassian/httpclient/atlassian-httpclient-parent/0.23.0/

olamy are you looking for https://bitbucket.org/atlassian/jira-rest-java-client? Its multi-module with sources for api api and core.

olamy there is also https://bitbucket.org/atlassian/atlassian-http.

got the sources from maven repo. I don't mind the rest client I just to override the Atlassian httpclient ( https://packages.atlassian.com/maven-external/com/atlassian/httpclient/atlassian-httpclient-plugin/0.23.0/ ) 

The goal is to have a local copy of those classes so we can simply have our version of Apache httpclient.

I have it done locally, I have to do more testing (separate Jenkins install with bo and real JIRA etc...)

pr https://github.com/jenkinsci/jira-plugin/pull/145

snapshot available for test https://repo.jenkins-ci.org/snapshots/org/jenkins-ci/plugins/jira/2.5.2-SNAPSHOT/jira-2.5.2-20180501.075458-1.hpi

warden Olivier has PR up and snapshot available at https://repo.jenkins-ci.org/snapshots/org/jenkins-ci/plugins/jira/2.5.2-SNAPSHOT/jira-2.5.2-20180501.231733-2.hpi, mind giving it a try?

markewaite

The link you provided [ git-client-plugin.hpi |https://ci.jenkins.io/job/Plugins/job/git-client-plugin/view/change-requests/job/PR-311/lastSuccessfulBuild/artifact/target/git-client.hpi] is no longer reachable. I want to check it out. Can you create it again pls? I have updated the Jira Plugin from Olivier and I would like to do the same with the Git Plugin.

tibor17 thanks for your willingness to test.  Download git client plugin from the artifact repository.

markewaite
The first commit did not hang and this is the log:

16:11:24 [WS-CLEANUP] Deleting project workspace...[WS-CLEANUP] done
[Pipeline] }
[Pipeline] // stage
[Pipeline] echo
16:11:25 shortDescription: Branch indexing
[Pipeline] echo
16:11:25 Job started by timer or branching.
[Pipeline] sh
16:11:25 [ibranch_Pipeline_playground-PRVK7X6T2XH7C3SUJ4JNYHLNG2EVZ6W7C37SOIOU6LWELPYIKHXQ] Running shell script
16:11:25 + mkdir -p build
[Pipeline] dir
16:11:25 Running in /var/lib/jenkins/workspace/ibranch_Pipeline_playground-PRVK7X6T2XH7C3SUJ4JNYHLNG2EVZ6W7C37SOIOU6LWELPYIKHXQ/build
[Pipeline] {
[Pipeline] stage
[Pipeline] { (checkout)
[Pipeline] checkout
16:11:25 Cloning the remote Git repository
16:11:25 Cloning with configured refspecs honoured and without tags
16:11:25 Cloning repository http://git.ads.local/phfa-entervonextgen/audit
16:11:25  > git init /var/lib/jenkins/workspace/ibranch_Pipeline_playground-PRVK7X6T2XH7C3SUJ4JNYHLNG2EVZ6W7C37SOIOU6LWELPYIKHXQ/build # timeout=10
16:11:26  > FINISHED git init /var/lib/jenkins/workspace/ibranch_Pipeline_playground-PRVK7X6T2XH7C3SUJ4JNYHLNG2EVZ6W7C37SOIOU6LWELPYIKHXQ/build # timeout=10
16:11:26 Fetching upstream changes from http://git.ads.local/phfa-entervonextgen/audit
16:11:26  > git --version # timeout=10
16:11:26  > FINISHED git --version # timeout=10
16:11:26 using GIT_ASKPASS to set credentials git private new rsa
16:11:26  > git fetch --no-tags --progress http://git.ads.local/phfa-entervonextgen/audit +refs/heads/*:refs/remotes/origin/*
16:11:30  > FINISHED git fetch --no-tags --progress http://git.ads.local/phfa-entervonextgen/audit +refs/heads/*:refs/remotes/origin/*
16:11:30  > git config remote.origin.url http://git.ads.local/phfa-entervonextgen/audit # timeout=10
16:11:30  > FINISHED git config remote.origin.url http://git.ads.local/phfa-entervonextgen/audit # timeout=10
16:11:30  > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
16:11:30  > FINISHED git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
16:11:30  > git config remote.origin.url http://git.ads.local/phfa-entervonextgen/audit # timeout=10
16:11:30  > FINISHED git config remote.origin.url http://git.ads.local/phfa-entervonextgen/audit # timeout=10
16:11:30 Fetching without tags
16:11:30 Fetching upstream changes from http://git.ads.local/phfa-entervonextgen/audit
16:11:30 using GIT_ASKPASS to set credentials git private new rsa
16:11:30  > git fetch --no-tags --progress http://git.ads.local/phfa-entervonextgen/audit +refs/heads/*:refs/remotes/origin/*
16:11:30  > FINISHED git fetch --no-tags --progress http://git.ads.local/phfa-entervonextgen/audit +refs/heads/*:refs/remotes/origin/*
16:11:30 Checking out Revision 512f79c739a04d5001d0c89dc889add0efbf06b9 (playground)
16:11:30  > git config core.sparsecheckout # timeout=10
16:11:30  > FINISHED git config core.sparsecheckout # timeout=10
16:11:30  > git checkout -f 512f79c739a04d5001d0c89dc889add0efbf06b9
16:11:31  > FINISHED git checkout -f 512f79c739a04d5001d0c89dc889add0efbf06b9
16:11:31 Commit message: "Test new Git and JIRA SNAPSHOT plugin with maven-site-plugin:3.7.1"
16:11:31  > git rev-list --no-walk 1535e85a8421d931ea2f54cd26dff47f647ed24b # timeout=10
16:11:31  > FINISHED git rev-list --no-walk 1535e85a8421d931ea2f54cd26dff47f647ed24b # timeout=10
[Pipeline] sh
16:11:31 

Thanks tibor17. The most interesting is the output the next time you see it hang.

That unreleased plugin build is based on the git client plugin 3.0.0-beta. If that unreleased plugin version resolves all the instances of hanging in your environment then I'll probably need to build a different unreleased plugin version which contains exactly git client plugin 2.7.1 plus that additional logging statement.

Rather than risk further delay, if you need git client plugin 2.7.1 plus the extra 'FINISHED' logging statements, download git client plugin 2.7.2-SNAPSHOT unreleased

olamy
Maybe OSGi would be quite a big deal for the developers. Would it be a generic solution of such problems in all Jenkins plugins?
Each Jenkins plugin may have isolated version of JIRA Client and Apache HTTP client.

markewaite
Ok Mark, just ping me when you need my test again. I will try to make few more commits next days on my system.

tibor17 I provided the extra logging in the git client plugin so that we could confirm my belief that what you're reporting as "hang in git rev-list" is not actually in the "git rev-list" command or even in the git client plugin that executes "git rev-list".

I can't duplicate this problem in my environment. The extra logging in the git client plugin is intended to help you confirm (or refute) my hypothesis that the problem is happening somewhere other than the git client plugin.

If prior to installing the unreleased git client plugin you saw 100% of the executions of that job hang, and the unreleased git client plugin never hangs, then you should switch to the unreleased git client plugin 2.7.2 to reduce the changes between the system that hangs and the system that never hangs.

olamy
markewaite
I updated both plugins Git Client and JIRA. So it means the change in JIRA was the one which made the build to continue without hanging. The second commit runs as well.

tibor17 Thanks for confirming jira plugin fixes the issue. olamy another data point to go ahead with merge and release jira plugin

olamy
vivek
After I have downgraded Git Client back to 2.7.1 release and still used Olivier's SNAPSHOT of JIRA plugin the build did not hang, hopefully it's fine.

Code changed in jenkins
User: Olivier Lamy
Path:
README.md
pom.xml
src/main/java/com/atlassian/httpclient/apache/httpcomponents/ApacheAsyncHttpClient.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/CommonBuilder.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/DefaultHttpClientFactory.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/DefaultMessage.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/DefaultRequest.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/DefaultResponse.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/EntityByteArrayInputStream.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/Headers.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/MavenUtils.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/PromiseHttpAsyncClient.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/RedirectStrategy.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/RequestEntityEffect.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/SettableFuturePromiseHttpPromiseAsyncClient.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/proxy/ProvidedProxyConfig.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/proxy/ProxyConfig.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/proxy/ProxyConfigFactory.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/proxy/ProxyCredentialsProvider.java
src/main/java/com/atlassian/httpclient/apache/httpcomponents/proxy/SystemPropertiesProxyConfig.java
src/main/java/com/atlassian/httpclient/base/event/AbstractHttpRequestEvent.java
src/main/java/com/atlassian/httpclient/base/event/HttpRequestCompletedEvent.java
src/main/java/com/atlassian/httpclient/base/event/HttpRequestFailedEvent.java
src/main/resources/atlassian-httpclient-plugin-0.23.0.pom
http://jenkins-ci.org/commit/jira-plugin/abd092848b1e617ceb31b16de07b1445d34c4d03
Log:
JENKINS-48357 get rid of atlassian http client (#145)

JENKINS-48357

• import sources from Atlassian Async HttpClient so we can use whatever we want as apache httpclient version

Signed-off-by: olivier lamy <olamy@apache.org>

• remove non used imported classes

Signed-off-by: olivier lamy <olamy@apache.org>

• get rid of this class

Signed-off-by: olivier lamy <olamy@apache.org>

• modern

Signed-off-by: olivier lamy <olamy@apache.org>

• add note on Atlassian sources import

Signed-off-by: olivier lamy <olamy@apache.org>

• add Atlassian pom for history purpose

Signed-off-by: olivier lamy <olamy@apache.org>

• JENKINS-48357 get rid of non used class

Signed-off-by: olivier lamy <olamy@apache.org>

• get rid of this httpasyncclient-cache

Signed-off-by: olivier lamy <olamy@apache.org>

• use last apache-httpcomponents-client-4-api 4.5.5-2.0 which include asynchttpclient now

Signed-off-by: olivier lamy <olamy@apache.org>

• add back jenkins-ci repo

Signed-off-by: olivier lamy <olamy@apache.org>

• exclude atlassian classes from coverage

Signed-off-by: olivier lamy <olamy@apache.org>

*NOTE:* This service been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/

Functionality will be removed from GitHub.com on January 31st, 2019.

olamy
Now my build does not hang but I have a new issue with Jira plugin because when I commit with messages which matches a pattern, the Jira plugin times out immediately. Maybe in less than one second. In the Management are two sections: JIRA and JIRA Sites. The first has successful login but the second fails. Why this happens?

Tibor, I think you'll get better results with a new bug report rather than asking questions on this closed report