Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-15065

Gerrit server suddenly shutdown and as a consequence the Jenkins Gerrit-Trigger fails to restart.

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Minor
    • Resolution: Unresolved
    • gerrit-trigger-plugin
    • None

    Description

      Manage Jenkins page displays this Error: “The connection to Gerrit is down! Check your settings and the Gerrit server”

      Issue: When the Gerrit server shuts down suddenly, the Gerrit trigger loses connectivity and stops working. The only way to get it restarted is to restart the whole Jenkins instance. Attempting to restart the gerrit trigger in Jenkins by using the Control box does not work. Firebug POSTs to this: http://jenkins:8210/descriptorByName/com.sonyericsson.hudson.plugins.gerrit.trigger.GerritManagement/startConnection

      and never receives a response from the server.

      Jenkins log files doesn't provide more information.

      Attachments

        Activity

          pszwed Piotr Szwed added a comment -

          I am facing exactly the same problem with the latest versions of Gerrit and Jenkins - it is a bit confusing as I need to cancel or pause big number of builds with every Gerrit restart.

          Jenkins ver. 1.518
          Gerrit Code Review (2.7-rc1)
          Gerrit Trigger plugin: 2.10.0

          pszwed Piotr Szwed added a comment - I am facing exactly the same problem with the latest versions of Gerrit and Jenkins - it is a bit confusing as I need to cancel or pause big number of builds with every Gerrit restart. Jenkins ver. 1.518 Gerrit Code Review (2.7-rc1) Gerrit Trigger plugin: 2.10.0

          I have the same issue on Jenkins ver. 1.509.3 Gerrit Code Review (2.6.1) Gerrit Trigger Plugin 2.10.1.
          It's confusing to stop all the jobs of my company to restart the server.
          I can see NPE when we click on Stop or Restart (unfortunatelly I can't manage to find out the log).

          ejuste Emmanuel Juste added a comment - I have the same issue on Jenkins ver. 1.509.3 Gerrit Code Review (2.6.1) Gerrit Trigger Plugin 2.10.1. It's confusing to stop all the jobs of my company to restart the server. I can see NPE when we click on Stop or Restart (unfortunatelly I can't manage to find out the log).
          kevin_brotcke Kevin Brotcke added a comment -

          I also saw this issue. Gerrit went down in the middle of the night and Gerrit Trigger was not able to reconnect, although I was able to restart Gerrit Trigger through the console without restarting Jenkins. I saw the errors below many times in the logs. It looks like the server was down for several hours and the Gerrit Trigger connection timed out eventually.

          SEVERE: IOException:
          com.sonyericsson.hudson.plugins.gerrit.gerritevents.ssh.SshException: com.jcraft.jsch.JSchException: java.net.ConnectException: Connection timed out
          at com.sonyericsson.hudson.plugins.gerrit.gerritevents.ssh.SshConnectionImpl.<init>(SshConnectionImpl.java:126)
          at com.sonyericsson.hudson.plugins.gerrit.gerritevents.ssh.SshConnectionFactory.getConnection(SshConnectionFactory.java:76)
          at com.sonyericsson.hudson.plugins.gerrit.gerritevents.GerritConnection.connect(GerritConnection.java:365)
          at com.sonyericsson.hudson.plugins.gerrit.gerritevents.GerritConnection.run(GerritConnection.java:293)
          Caused by: com.jcraft.jsch.JSchException: java.net.ConnectException: Connection timed out
          at com.jcraft.jsch.Util.createSocket(Util.java:341)
          at com.jcraft.jsch.Session.connect(Session.java:192)
          at com.jcraft.jsch.Session.connect(Session.java:160)
          at com.sonyericsson.hudson.plugins.gerrit.gerritevents.ssh.SshConnectionImpl.<init>(SshConnectionImpl.java:122)
          ... 3 more
          Caused by: java.net.ConnectException: Connection timed out
          at java.net.PlainSocketImpl.socketConnect(Native Method)
          at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
          at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
          at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
          at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:385)
          at java.net.Socket.connect(Socket.java:546)
          at java.net.Socket.connect(Socket.java:495)
          at java.net.Socket.<init>(Socket.java:392)
          at java.net.Socket.<init>(Socket.java:206)
          at com.jcraft.jsch.Util.createSocket(Util.java:335)
          ... 6 more
          Feb 6, 2014 2:17:32 AM com.sonyericsson.hudson.plugins.gerrit.gerritevents.GerritConnection connect
          SEVERE: Could not connect to Gerrit server! Host: myserver.com Port: #####

          kevin_brotcke Kevin Brotcke added a comment - I also saw this issue. Gerrit went down in the middle of the night and Gerrit Trigger was not able to reconnect, although I was able to restart Gerrit Trigger through the console without restarting Jenkins. I saw the errors below many times in the logs. It looks like the server was down for several hours and the Gerrit Trigger connection timed out eventually. SEVERE: IOException: com.sonyericsson.hudson.plugins.gerrit.gerritevents.ssh.SshException: com.jcraft.jsch.JSchException: java.net.ConnectException: Connection timed out at com.sonyericsson.hudson.plugins.gerrit.gerritevents.ssh.SshConnectionImpl.<init>(SshConnectionImpl.java:126) at com.sonyericsson.hudson.plugins.gerrit.gerritevents.ssh.SshConnectionFactory.getConnection(SshConnectionFactory.java:76) at com.sonyericsson.hudson.plugins.gerrit.gerritevents.GerritConnection.connect(GerritConnection.java:365) at com.sonyericsson.hudson.plugins.gerrit.gerritevents.GerritConnection.run(GerritConnection.java:293) Caused by: com.jcraft.jsch.JSchException: java.net.ConnectException: Connection timed out at com.jcraft.jsch.Util.createSocket(Util.java:341) at com.jcraft.jsch.Session.connect(Session.java:192) at com.jcraft.jsch.Session.connect(Session.java:160) at com.sonyericsson.hudson.plugins.gerrit.gerritevents.ssh.SshConnectionImpl.<init>(SshConnectionImpl.java:122) ... 3 more Caused by: java.net.ConnectException: Connection timed out at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:385) at java.net.Socket.connect(Socket.java:546) at java.net.Socket.connect(Socket.java:495) at java.net.Socket.<init>(Socket.java:392) at java.net.Socket.<init>(Socket.java:206) at com.jcraft.jsch.Util.createSocket(Util.java:335) ... 6 more Feb 6, 2014 2:17:32 AM com.sonyericsson.hudson.plugins.gerrit.gerritevents.GerritConnection connect SEVERE: Could not connect to Gerrit server! Host: myserver.com Port: #####
          jkugler Joshua Kugler added a comment -

          Just experienced this on Jenkins 1.583, Gerrit Trigger 2.3.1 (yes, old version, but from above comments, seems to exist in recent versions as well).

          jkugler Joshua Kugler added a comment - Just experienced this on Jenkins 1.583, Gerrit Trigger 2.3.1 (yes, old version, but from above comments, seems to exist in recent versions as well).
          alexanderlink alexanderlink added a comment -

          We could not reproduce this issue with Jenkins 1.577, Gerrit Trigger 2.11.1, Git Client 1.10.1 and Git Plugin 2.2.5.
          The Gerrit Trigger plugin automatically reconnected to the Gerrit Server. Any further pushes to Gerrit resulted in properly triggered Gerrit voter jobs.

          We also tested Gerrit shutdown while a Gerrit voter job was in the build step. Of course the Gerrit Trigger plugin could not vote when the job finished. This voting is lost.
          And the Gerrit Trigger plugin did not trigger the voter job for this patch set again, but it has been triggered for all new patch sets. And for the lost patch set a manual retrigger worked as well.

          alexanderlink alexanderlink added a comment - We could not reproduce this issue with Jenkins 1.577, Gerrit Trigger 2.11.1, Git Client 1.10.1 and Git Plugin 2.2.5. The Gerrit Trigger plugin automatically reconnected to the Gerrit Server. Any further pushes to Gerrit resulted in properly triggered Gerrit voter jobs. We also tested Gerrit shutdown while a Gerrit voter job was in the build step. Of course the Gerrit Trigger plugin could not vote when the job finished. This voting is lost. And the Gerrit Trigger plugin did not trigger the voter job for this patch set again, but it has been triggered for all new patch sets. And for the lost patch set a manual retrigger worked as well.
          jkugler Joshua Kugler added a comment -

          @alexanderlink: how long was your Gerrit server down? In our case, it was down for several hours, and finally the Gerrit trigger just gave up, or became hung/wedged.

          jkugler Joshua Kugler added a comment - @alexanderlink: how long was your Gerrit server down? In our case, it was down for several hours, and finally the Gerrit trigger just gave up, or became hung/wedged.

          Confirming that for us as well it was down a few hours.

          I'd also like to add that this isn't 100% reproducible even for us. Sometimes the server is down and comes back up with no issues, other times we encounter the issue in this ticket - it appears to be intermittent.

          krw1243 Kelsey Prantis added a comment - Confirming that for us as well it was down a few hours. I'd also like to add that this isn't 100% reproducible even for us. Sometimes the server is down and comes back up with no issues, other times we encounter the issue in this ticket - it appears to be intermittent.

          People

            rsandell rsandell
            audrey Audrey Azra
            Votes:
            6 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated: