Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-62583

Git background operations should alert when they fail

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • git-plugin
    • None
    • Linux

      We have a job based on polling git-lab. Something went wrong in one day and it has left a .git/config.lock file in the workspace.

      From that moment on, all git polling were happening but errored with the config file issue:

      ===

      FATAL: hudson.plugins.git.GitException: Failed to fetch from git@xxx.git

      FATAL: hudson.plugins.git.GitException: Failed to fetch from git@xxx.git

      Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to XXX at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1788) at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356) at hudson.remoting.Channel.call(Channel.java:998) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:285) at com.sun.proxy.$Proxy77.setRemoteUrl(Unknown Source) at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:299) at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:897) at hudson.plugins.git.GitSCM.compareRemoteRevisionWithImpl(GitSCM.java:787) at hudson.plugins.git.GitSCM.compareRemoteRevisionWith(GitSCM.java:670) at hudson.scm.SCM.compareRemoteRevisionWith(SCM.java:401) at hudson.scm.SCM.poll(SCM.java:418) at hudson.model.AbstractProject.pollWithWorkspace(AbstractProject.java:1410) at hudson.model.AbstractProject._poll(AbstractProject.java:1380) at hudson.model.AbstractProject.poll(AbstractProject.java:1291) at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:605) at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:651) at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:119) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)hudson.plugins.git.GitException: Command "git config remote.origin.url git@xxx.git" returned status code 255:stdout:stderr: error: could not lock config file .git/config: File exists
      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2430) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2360) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2356) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1916) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1928) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1542) at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160) at jdk.internal.reflect.GeneratedMethodAccessor25.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:931) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:905) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:857) at hudson.remoting.UserRequest.perform(UserRequest.java:211) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)Caused: hudson.plugins.git.GitException: Failed to fetch from git@xxx.git at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:909) at hudson.plugins.git.GitSCM.compareRemoteRevisionWithImpl(GitSCM.java:787) at hudson.plugins.git.GitSCM.compareRemoteRevisionWith(GitSCM.java:670)Caused: java.io.IOException at hudson.plugins.git.GitSCM.compareRemoteRevisionWith(GitSCM.java:672) at hudson.scm.SCM.compareRemoteRevisionWith(SCM.java:401) at hudson.scm.SCM.poll(SCM.java:418) at hudson.model.AbstractProject.pollWithWorkspace(AbstractProject.java:1410) at hudson.model.AbstractProject._poll(AbstractProject.java:1380) at hudson.model.AbstractProject.poll(AbstractProject.java:1291) at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:605) at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:651) at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:119) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)

      Done. Took 70 ms

      No changes

      ===

       

      Please note that the final conclusion: "No changes"

       

      This is not the desired outcome as the polling error should have been notified, maybe by the methods defined in the job configuration.

      Possible Solution (based on GitHub Webhooks Warning)

      When alerts on failure of background tasks is implemented in the git plugin, it could be done through a similar user interface technique as is used by GitHub webhooks. A number appears in the top right of the Jenkins interface. When that number is clicked, a dialog appears that includes a "View" button that would open a page that lists all the problems of that type. The "Dismiss" button would clear the contents of the page that lists all the problems of that type.

          [JENKINS-62583] Git background operations should alert when they fail

          Mark Waite added a comment -

          Agreed that it would be good to have a system level alert where failures in background tasks could be displayed. Webhooks display warnings in that location when their credentials are invalid or not working as expected. Some of the examples of places that should be reported include:

          • Polling failures
          • Repository scan failures

          I'm sure there are other git background operations that would benefit from a global alert to tell the administrator that an issue needs investigation.

          Mark Waite added a comment - Agreed that it would be good to have a system level alert where failures in background tasks could be displayed. Webhooks display warnings in that location when their credentials are invalid or not working as expected. Some of the examples of places that should be reported include: Polling failures Repository scan failures I'm sure there are other git background operations that would benefit from a global alert to tell the administrator that an issue needs investigation.

          Lestin Liu added a comment - - edited

          Hi Mark,

           

          Thanks for looking at this. However in our case/configuration, it doesn't show up at the monitor area.
          //We do have notification saying new version of Jenkins avail etc so the monitor was working.

          Maybe it's because we're using git rather than git-hub?

          Sorry I'm not a github user, but felt like the notification you posted came from a webhook.
          Does that mean it's a git-hub initiated job rather than a Jenkins SCM initiated job for polling?

          Thanks.

          Lestin Liu added a comment - - edited Hi Mark,   Thanks for looking at this. However in our case/configuration, it doesn't show up at the monitor area. //We do have notification saying new version of Jenkins avail etc so the monitor was working. Maybe it's because we're using git rather than git-hub? Sorry I'm not a github user, but felt like the notification you posted came from a webhook. Does that mean it's a git-hub initiated job rather than a Jenkins SCM initiated job for polling? Thanks.

          Mark Waite added a comment -

          Sorry that I wasn't clear lestinliu. You're absolutely correct that the git plugin does not provide any notice when a background task fails.

          I placed that screenshot as a reminder to whoever implements this that there is already a location in the user interface that is able to display these types of notices. The git plugin does not use that available notification technique. It should be extended to use that area so that the case you've described would have a location where it could show a failure.

          Mark Waite added a comment - Sorry that I wasn't clear lestinliu . You're absolutely correct that the git plugin does not provide any notice when a background task fails. I placed that screenshot as a reminder to whoever implements this that there is already a location in the user interface that is able to display these types of notices. The git plugin does not use that available notification technique. It should be extended to use that area so that the case you've described would have a location where it could show a failure.

            Unassigned Unassigned
            lestinliu Lestin Liu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: