Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45928

Jenkins job crashes and dissapears when long running a checkmark security scan

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Blocker Blocker
    • checkmarx-plugin
    • None

      Hi,

       

      Can you please please help with this issue?

       

      In Jenkins version 2.60.1 we have a number of Checkamarx jobs performing security scans on a number of repositories. Most of them work just fine but the long running ones (due to the size of the code repositories) that run for more than 4 hours are crashing right during report creation. After failure the job run details are not even available in Jenkins and the Jenkins box is going down too.

      Here is the log found in Manage Jenkins/ System Log around the failure:

      Finished EC2 alive slaves monitor. 162 ms
      Jul 31, 2017 12:36:14 PM WARNING hudson.init.impl.InstallUncaughtExceptionHandler$1 reportException
      null
      java.io.IOException: Broken pipe
      at sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
      at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
      at sun.nio.ch.IOUtil.write(IOUtil.java:148)
      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:504)
      at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:172)
      Caused: org.eclipse.jetty.io.EofException
      at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:192)
      at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:408)
      at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:302)
      at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:129)

      (...)

       

      Also, in case it is useful, I was watching the log or the job as it run and this is what it looked like at the end (the the job altogether become unavailable):

      (Current stage progress: 99%, Total progress: 99%) [Checkmarx] - [info] - Scan Finished Successfully - RunID: 687 ScanID:1000601 [Checkmarx] - [info] - Requesting XML Scan Report Generation [Checkmarx] - [info] - XML Report generation in progress [Checkmarx] - [info] - Scan report generated on Checkmarx server

      Thanks,

      Mihai

          [JENKINS-45928] Jenkins job crashes and dissapears when long running a checkmark security scan

          zhang kunming added a comment -

          how to fix this bug? I met the same exception in system logs

          zhang kunming added a comment - how to fix this bug? I met the same exception in system logs

          Colm O added a comment -

          Slightly different context (different Jenkins instance), but this is still the same issue being manifested.

          We have a Jenkins pipeline submitting a Checkmarx task for the scannage of 3 related source code projects.  The Checkmarx scan completes (within Checkmarx), but on the Jenkins side, the task locks up after the following output:

          [Checkmarx] - [info] - Scan Finished Successfully - RunID: 12650 ScanID:1001884
          [Checkmarx] - [info] - Requesting XML Scan Report Generation
          [Checkmarx] - [info] - XML Report generation in progress
          [Checkmarx] - [info] - Scan report generated on Checkmarx server

          For comparison, in a different, but almost identically configured task, which is scans a different codebase, we see the following output:

          [Checkmarx] - [info] - XML Report generation in progress
          [Checkmarx] - [info] - Scan report generated on Checkmarx server
          [Checkmarx] - [info] - Scan report written to: /local/jenkins/jobs/<<censored>>/builds/40/checkmarx/ScanReport.xml
          [Checkmarx] - [info] - Requesting PDF Scan Report Generation
          [Checkmarx] - [info] - PDF Report generation in progress
          [Checkmarx] - [info] - Scan report generated on Checkmarx server
          [Checkmarx] - [info] - Scan report written to: /local/jenkins/jobs/<<censored>>/builds/40/checkmarx/ScanReport.pdf

          Note that in this example, the Jenkins instance does not fall over, but eventually all build executors are consumed by multiple instances of the same task, and subsequent tasks end up in the queue, instead of executing.

           

          Jenkins version: 2.109

          Checkmarx plugin version: 8.60.0

          Checkmarx version: 8.5.0

          Colm O added a comment - Slightly different context (different Jenkins instance), but this is still the same issue being manifested. We have a Jenkins pipeline submitting a Checkmarx task for the scannage of 3 related source code projects.  The Checkmarx scan completes (within Checkmarx), but on the Jenkins side, the task locks up after the following output: [Checkmarx] - [info] - Scan Finished Successfully - RunID: 12650 ScanID:1001884 [Checkmarx] - [info] - Requesting XML Scan Report Generation [Checkmarx] - [info] - XML Report generation in progress [Checkmarx] - [info] - Scan report generated on Checkmarx server For comparison, in a different, but almost identically configured task, which is scans a different codebase, we see the following output: [Checkmarx] - [info] - XML Report generation in progress [Checkmarx] - [info] - Scan report generated on Checkmarx server [Checkmarx] - [info] - Scan report written to: /local/jenkins/jobs/<<censored>>/builds/40/checkmarx/ScanReport.xml [Checkmarx] - [info] - Requesting PDF Scan Report Generation [Checkmarx] - [info] - PDF Report generation in progress [Checkmarx] - [info] - Scan report generated on Checkmarx server [Checkmarx] - [info] - Scan report written to: /local/jenkins/jobs/<<censored>>/builds/40/checkmarx/ScanReport.pdf Note that in this example, the Jenkins instance does not fall over, but eventually all build executors are consumed by multiple instances of the same task, and subsequent tasks end up in the queue, instead of executing.   Jenkins version: 2.109 Checkmarx plugin version: 8.60.0 Checkmarx version: 8.5.0

            checkmarxsupport Checkmarx Support
            mvilcu Mihai Vilcu
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: