Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-69585

Segmentation fault in pipeline shell command during build

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Major
    • pipeline
    • Jenkins 2.367
      Debian GNU/Linux 11 (bullseye)
      perl 5, version 32, subversion 1 (v5.32.1)

    Description

      One of my jobs is running a long make procedure, which usually takes about 20 minutes to complete. Running the job using shell commands in a freestyle job works fine. However, when I use the exact same command in a Jenkinsfile pipeline stage, after 15 Minutes I get a perl segmentation fault in syslog and the job crashes:

       

      23:28:14 jenkins kernel: [45626.883991] show_signal_msg: 2 callbacks suppressed
      Sep 11 23:28:14 jenkins kernel: [45626.883993] perl[3368371]: segfault at 7fff8686aff8 ip 00005616a2952609 sp 00007fff8686b000 error 6 in perl[5616a28cc000+239000]
      Sep 11 23:28:14 jenkins kernel: [45626.884002] Code: 90 41 56 41 55 41 54 49 89 f4 55 48 89 fd 53 48 83 ec 20 4c 8b 6f 08 64 48 8b 04 25 28 00 00 00 48 89 44 24 18 31 c0 4c 89 ef <e8> 22 03 12 00 48 85 c0 0f 84 f9 01 00 00 48 8d 15 5a d9 89 00 49
      Sep 11 23:28:14 jenkins kernel: [45626.912157] perl[3368373]: segfault at 7ffddd528ff8 ip 00007fed602a5559 sp 00007ffddd529000 error 6 in libc-2.31.so[7fed60242000+15a000]
      Sep 11 23:28:14 jenkins kernel: [45626.912167] Code: 00 00 be 81 00 00 00 b8 ca 00 00 00 0f 05 e9 68 ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 41 57 41 56 41 55 41 54 55 <53> 48 81 ec 88 00 00 00 48 85 f6 0f 88 ae 05 00 00 48 8d 46 17 49
      Sep 11 23:28:14 jenkins kernel: [45626.979091] perl[3368379]: segfault at 7ffc51feeff8 ip 00007f1c609e0956 sp 00007ffc51feeff0 error 6 in libc-2.31.so[7f1c6097d000+15a000]
      Sep 11 23:28:14 jenkins kernel: [45626.979098] Code: 00 4c 8b 35 44 94 14 00 48 8d 45 ef 45 31 db 48 c1 e8 04 48 89 44 24 10 64 49 83 3e 00 74 0b 48 39 05 7e 99 14 00 4c 0f 47 dd <48> c7 44 24 08 00 00 00 00 4d 8d 4c 24 60 4c 89 5c 24 20 44 89 6c
      Sep 11 23:28:14 jenkins kernel: [45627.056773] perl[3368385]: segfault at 7ffefe913ff8 ip 00007fc51aef40f9 sp 00007ffefe914000 error 6 in libc-2.31.so[7fc51ae8f000+15a000]
      Sep 11 23:28:14 jenkins kernel: [45627.056780] Code: 10 48 8b 05 21 7e 14 00 48 89 08 48 8b 05 ff 7e 14 00 48 89 30 48 8b 05 f5 7d 14 00 48 89 38 c3 90 48 8b 05 f9 7d 14 00 41 54 <55> 48 89 fd 53 48 8b 00 48 85 c0 0f 85 46 01 00 00 48 85 ff 0f 88
      Sep 11 23:28:14 jenkins kernel: [45627.081213] perl[3368387]: segfault at 7ffcf05d2ff8 ip 00007ff2754a40f9 sp 00007ffcf05d3000 error 6 in libc-2.31.so[7ff27543f000+15a000]
      Sep 11 23:28:14 jenkins kernel: [45627.081221] Code: 10 48 8b 05 21 7e 14 00 48 89 08 48 8b 05 ff 7e 14 00 48 89 30 48 8b 05 f5 7d 14 00 48 89 38 c3 90 48 8b 05 f9 7d 14 00 41 54 <55> 48 89 fd 53 48 8b 00 48 85 c0 0f 85 46 01 00 00 48 85 ff 0f 88
      Sep 11 23:28:14 jenkins kernel: [45627.105686] perl[3368389]: segfault at 7fffcafeaff8 ip 000055e6e8fa4879 sp 00007fffcafeb000 error 6 in perl[55e6e8df7000+239000]
      Sep 11 23:28:14 jenkins kernel: [45627.105695] Code: c0 48 8d 05 7c 7d 25 00 4d 85 c0 0f 85 07 ff ff ff 8b 45 0c e9 9b fe ff ff 0f 1f 84 00 00 00 00 00 41 57 48 8d 15 27 96 25 00 <41> 56 41 55 41 54 55 53 48 83 ec 28 8b 47 0c 44 0f b6 f0 4e 8d 3c
      Sep 11 23:28:14 jenkins kernel: [45627.153470] perl[3368393]: segfault at 7ffd055b5ff8 ip 00007fcdb2991f3b sp 00007ffd055b6000 error 6 in libc-2.31.so[7fcdb292e000+15a000]
      Sep 11 23:28:14 jenkins kernel: [45627.153479] Code: 89 ff 40 0f 95 c6 40 0f b6 f6 48 c1 e6 02 48 09 ee 48 83 ce 01 48 89 70 08 48 89 ce 48 83 ce 01 48 89 72 08 48 89 de 49 89 0a <e8> 30 cd ff ff e9 0d f7 ff ff 0f 1f 00 48 8b 71 18 48 83 4c 29 08
      Sep 11 23:28:14 jenkins kernel: [45627.178268] perl[3368395]: segfault at 7ffcba345ff8 ip 00007f2ae701215f sp 00007ffcba346000 error 6 in libc-2.31.so[7f2ae6fad000+15a000]
      Sep 11 23:28:14 jenkins kernel: [45627.178275] Code: 00 00 00 48 3b 1d 91 81 14 00 0f 82 ab 00 00 00 64 8b 04 25 18 00 00 00 85 c0 0f 85 0b 01 00 00 48 89 ee 48 8d 3d 21 8a 14 00 <e8> ec e3 ff ff 49 89 c0 48 85 c0 0f 84 b0 00 00 00 48 8b 40 f8 a8
      Sep 11 23:28:14 jenkins kernel: [45627.201212] perl[3368397]: segfault at 7ffe2a91aff8 ip 00007f8dc4f26556 sp 00007ffe2a91b000 error 6 in libc-2.31.so[7f8dc4ec3000+15a000]
      Sep 11 23:28:14 jenkins kernel: [45627.201220] Code: ba 01 00 00 00 be 81 00 00 00 b8 ca 00 00 00 0f 05 e9 68 ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 41 57 41 56 41 55 <41> 54 55 53 48 81 ec 88 00 00 00 48 85 f6 0f 88 ae 05 00 00 48 8d
      Sep 11 23:28:14 jenkins kernel: [45627.222736] perl[3368399]: segfault at 7ffc83d1dff8 ip 000055f274522879 sp 00007ffc83d1e000 error 6 in perl[55f274375000+239000]
      Sep 11 23:28:14 jenkins kernel: [45627.222746] Code: c0 48 8d 05 7c 7d 25 00 4d 85 c0 0f 85 07 ff ff ff 8b 45 0c e9 9b fe ff ff 0f 1f 84 00 00 00 00 00 41 57 48 8d 15 27 96 25 00 <41> 56 41 55 41 54 55 53 48 83 ec 28 8b 47 0c 44 0f b6 f0 4e 8d 3c
      Sep 11 23:28:52 jenkins jenkins[2268949]: 2022-09-11 21:28:52.748+0000 [id=791]#011INFO#011o.j.p.workflow.job.WorkflowRun#finish: ffkbu-multihood-stable-wireguard-gluon2021.X-ssidchanger #23 completed: FAILURE
      
      

      I have no clue why the shell command in the second context is failing. Here is my Jenkinsfile:

       

      properties([buildDiscarder(logRotator(artifactDaysToKeepStr: '', artifactNumToKeepStr: '', daysToKeepStr: '', numToKeepStr: '3')), [$class: 'ScannerJobProperty', doNotScan: false], disableConcurrentBuilds(), disableResume(), durabilityHint('PERFORMANCE_OPTIMIZED'), [$class: 'EnvInjectJobProperty', info: [loadFilesFromMaster: false, propertiesContent: '''GLUON_AUTOUPDATER_BRANCH="stable"
      GLUON_AUTOUPDATER_ENABLED=1
      MAKEARGS=-j8
      V="sc"''', secureGroovyScript: [classpath: [], oldScript: '', sandbox: false, script: '']], keepBuildVariables: true, keepJenkinsSystemVariables: true, on: true]])
      
      pipeline {
        agent any
        stages {
          stage('Download site configuration') {
            steps {
              ansiColor('xterm') {
                checkout([$class: 'GitSCM', branches: [[name: 'v2021.1.2']], extensions: [], userRemoteConfigs: [[url: 'https://github.com/freifunk-gluon/gluon.git']]])
                sh 'rm -rf site/ || true'
                sh 'git clone https://github.com/ff-kbu/site-ffkbu-multidomain -b wireguard-offlinessid site'
                sh 'cp /var/lib/jenkins/workspace/ar71xx-generic ./targets/ar71xx-generic'
              }
            }
          }
      
          stage('Update gluon packages') {
            steps {
              ansiColor('xterm') {
                sh 'make $MAKEARGS update'
              }
            }
          }
      
          stage('Build x86') {
            steps {
              ansiColor('xterm') {
                sh 'make $MAKEARGS GLUON_TARGET=x86-64'
                sh 'make $MAKEARGS GLUON_TARGET=x86-generic'
                sh 'make $MAKEARGS GLUON_TARGET=x86-geode'
              }
            }
          }
      
          stage('Build ar71xx') {
            steps {
              ansiColor('xterm') {
                sh 'make $MAKEARGS GLUON_TARGET=ar71xx-generic'
                sh 'make $MAKEARGS GLUON_TARGET=ar71xx-generic GLUON_DEVICES=tp-link-archer-c25-v1 BROKEN=1'
                sh 'make $MAKEARGS GLUON_TARGET=ar71xx-nand'
              }
            }
          }
      
          stage('Build ath79') {
            steps {
              ansiColor('xterm') {
                sh 'make $MAKEARGS GLUON_TARGET=ath79-generic'
              }
            }
          }
      
          stage('Build brcm2708') {
            steps {
              ansiColor('xterm') {
                sh 'make $MAKEARGS GLUON_TARGET=brcm2708-bcm2708'
                sh 'make $MAKEARGS GLUON_TARGET=brcm2708-bcm2709'
              }
            }
          }
      
          stage('Build ipq40xx') {
            steps {
              ansiColor('xterm') {
                sh 'make $MAKEARGS GLUON_TARGET=ipq40xx-generic'
              }
            }
          }
      
          stage('Build ipq806x') {
            steps {
              ansiColor('xterm') {
                sh 'make $MAKEARGS GLUON_TARGET=ipq806x-generic'
              }
            }
          }
      
          stage('Build lantiq') {
            steps {
              ansiColor('xterm') {
                sh 'make $MAKEARGS GLUON_TARGET=lantiq-xrx200'
                sh 'make $MAKEARGS GLUON_TARGET=lantiq-xway'
              }
            }
          }
      
          stage('Build mpc85xx') {
            steps {
              ansiColor('xterm') {
                sh 'make $MAKEARGS GLUON_TARGET=mpc85xx-generic'
                sh 'make $MAKEARGS GLUON_TARGET=mpc85xx-p1020'
              }
            }
          }
      
          stage('Build ramips') {
            steps {
              ansiColor('xterm') {
                sh 'make $MAKEARGS GLUON_TARGET=ramips-mt7620'
                sh 'make $MAKEARGS GLUON_TARGET=ramips-mt7621'
                sh 'make $MAKEARGS GLUON_TARGET=ramips-mt76x8'
              }
            }
          }
      
          stage('Manifest') {
            steps {
              ansiColor('xterm') {
                sh 'make $MAKEARGS manifest GLUON_PRIORITY=1'
              }
            }
          }
        }
      }
       

      Any suggestions or clues on this issue?

      Attachments

        Activity

          People

            Unassigned Unassigned
            jzielke jzielke
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: