[JENKINS-48990] Long builds with no logs fail at Ubuntu after 1-2 hours

Type: Bug
Resolution: Unresolved
Priority: Blocker
Component/s: durable-task-plugin
Labels:
None
Environment:
Jenkins ver. 2.89.2
Remoting Version 3.14
Latest plugins for 01/17/2018
Agent ubuntu.1, launched via ssh
Ubuntu 14.04.5 LTS

Similar Issues:
Powered by SuggestiMate

Show

The simplest Jenkinsfile to reproduce:

pipeline {
    agent { label "ubuntu.1" }

    options {
        disableConcurrentBuilds()
        ansiColor('xterm')
        timestamps()
    }

    stages {
        stage('Sleep') {
            steps {
                sh "sleep 99999999"
            }
        }
    }
}

Log ends with:

08:54:24 [Org_repository-5BKEJ4KU7KWRDM4GMA5EGH4UVHK4U74AML3VWSVHEXZJBWCI2QTQ] Running shell script
08:54:25 + sleep 99999999
11:30:55 Cannot contact ubuntu.1: java.lang.InterruptedException
Post stage
[Pipeline] archiveArtifacts
11:31:11 Archiving artifacts
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // timeout
[Pipeline] End of Pipeline

GitHub has been notified of this commit’s build result

ERROR: script returned exit code -1
Finished: FAILURE

With All trace from durable-task-plugin I see "heartbeat touches apparently not running in ..."

Since failures started about a month ago or so, it seem to be a regression of https://github.com/jenkinsci/durable-task-plugin/commit/5c98ca855a9a2fb0043888c1bab9cc5f41c8773a

I did also try to run 'ps' once a minute at the same agent host in parallel with this job. The sh processes with heartbeat and with sleep disappear right after the build fails. These are processes defined in https://github.com/jenkinsci/durable-task-plugin/blob/5c98ca855a9a2fb0043888c1bab9cc5f41c8773a/src/main/java/org/jenkinsci/plugins/durabletask/BourneShellScript.java#L154

Looking to the workspace folder and @tmp folder I see no pid file - is it expected?

Alexander Vorobiev created issue - 2018-01-17 10:15

Alexander Vorobiev made changes - 2018-01-17 10:35

Description

Original: The simplest Jenkinsfile to reproduce:
{code:java}
pipeline {
    agent { label "ubuntu.1" }

    options {
        disableConcurrentBuilds()
        ansiColor('xterm')
        timestamps()
    }

    stages {
        stage('Sleep') {
            steps {
                sh "sleep 99999999"
            }
        }
    }
}{code}
Log ends with:
{code:java}
08:54:24 [Org_repository-5BKEJ4KU7KWRDM4GMA5EGH4UVHK4U74AML3VWSVHEXZJBWCI2QTQ] Running shell script
08:54:25 + sleep 99999999
11:30:55 Cannot contact ubuntu.1: java.lang.InterruptedException
Post stage
[Pipeline] archiveArtifacts
11:31:11 Archiving artifacts
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // timeout
[Pipeline] End of Pipeline

GitHub has been notified of this commit’s build result

ERROR: script returned exit code -1
Finished: FAILURE
{code}
With All trace from durable-task-plugin I see "heartbeat touches apparently not running in ..."

Since failures started about a month ago or so, it seem to be a regression of [https://github.com/jenkinsci/durable-task-plugin/commit/5c98ca855a9a2fb0043888c1bab9cc5f41c8773a]

I did also try to run 'ps' once a minute at the same agent host in parallel with this job. The sh processes with heartbeat and with sleep disappear right after the build fails. These are processes defined in [https://github.com/jenkinsci/durable-task-plugin/blob/5c98ca855a9a2fb0043888c1bab9cc5f41c8773a/src/main/java/org/jenkinsci/plugins/durabletask/BourneShellScript.java#L154]

New:

The simplest Jenkinsfile to reproduce:
{code:java}
pipeline {
    agent { label "ubuntu.1" }

    options {
        disableConcurrentBuilds()
        ansiColor('xterm')
        timestamps()
    }

    stages {
        stage('Sleep') {
            steps {
                sh "sleep 99999999"
            }
        }
    }
}{code}
Log ends with:
{code:java}
08:54:24 [Org_repository-5BKEJ4KU7KWRDM4GMA5EGH4UVHK4U74AML3VWSVHEXZJBWCI2QTQ] Running shell script
08:54:25 + sleep 99999999
11:30:55 Cannot contact ubuntu.1: java.lang.InterruptedException
Post stage
[Pipeline] archiveArtifacts
11:31:11 Archiving artifacts
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // timeout
[Pipeline] End of Pipeline

GitHub has been notified of this commit’s build result

ERROR: script returned exit code -1
Finished: FAILURE
{code}
With All trace from durable-task-plugin I see "heartbeat touches apparently not running in ..."

Since failures started about a month ago or so, it seem to be a regression of [https://github.com/jenkinsci/durable-task-plugin/commit/5c98ca855a9a2fb0043888c1bab9cc5f41c8773a]

I did also try to run 'ps' once a minute at the same agent host in parallel with this job. The sh processes with heartbeat and with sleep disappear right after the build fails. These are processes defined in [https://github.com/jenkinsci/durable-task-plugin/blob/5c98ca855a9a2fb0043888c1bab9cc5f41c8773a/src/main/java/org/jenkinsci/plugins/durabletask/BourneShellScript.java#L154]

Alexander Vorobiev made changes - 2018-01-17 10:38

Environment

New: Jenkins ver. 2.89.2
Latest plugins for 01/17/2018
Agent ubuntu.1, launched via ssh
Ubuntu 14.04.5 LTS

Alexander Vorobiev made changes - 2018-01-17 10:43

Description

Original:

The simplest Jenkinsfile to reproduce:
{code:java}
pipeline {
    agent { label "ubuntu.1" }

    options {
        disableConcurrentBuilds()
        ansiColor('xterm')
        timestamps()
    }

    stages {
        stage('Sleep') {
            steps {
                sh "sleep 99999999"
            }
        }
    }
}{code}
Log ends with:
{code:java}
08:54:24 [Org_repository-5BKEJ4KU7KWRDM4GMA5EGH4UVHK4U74AML3VWSVHEXZJBWCI2QTQ] Running shell script
08:54:25 + sleep 99999999
11:30:55 Cannot contact ubuntu.1: java.lang.InterruptedException
Post stage
[Pipeline] archiveArtifacts
11:31:11 Archiving artifacts
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // timeout
[Pipeline] End of Pipeline

GitHub has been notified of this commit’s build result

ERROR: script returned exit code -1
Finished: FAILURE
{code}
With All trace from durable-task-plugin I see "heartbeat touches apparently not running in ..."

Since failures started about a month ago or so, it seem to be a regression of [https://github.com/jenkinsci/durable-task-plugin/commit/5c98ca855a9a2fb0043888c1bab9cc5f41c8773a]

I did also try to run 'ps' once a minute at the same agent host in parallel with this job. The sh processes with heartbeat and with sleep disappear right after the build fails. These are processes defined in [https://github.com/jenkinsci/durable-task-plugin/blob/5c98ca855a9a2fb0043888c1bab9cc5f41c8773a/src/main/java/org/jenkinsci/plugins/durabletask/BourneShellScript.java#L154]

New: The simplest Jenkinsfile to reproduce:
{code:java}
pipeline {
    agent { label "ubuntu.1" }

    options {
        disableConcurrentBuilds()
        ansiColor('xterm')
        timestamps()
    }

    stages {
        stage('Sleep') {
            steps {
                sh "sleep 99999999"
            }
        }
    }
}{code}
Log ends with:
{code:java}
08:54:24 [Org_repository-5BKEJ4KU7KWRDM4GMA5EGH4UVHK4U74AML3VWSVHEXZJBWCI2QTQ] Running shell script
08:54:25 + sleep 99999999
11:30:55 Cannot contact ubuntu.1: java.lang.InterruptedException
Post stage
[Pipeline] archiveArtifacts
11:31:11 Archiving artifacts
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // timeout
[Pipeline] End of Pipeline

GitHub has been notified of this commit’s build result

ERROR: script returned exit code -1
Finished: FAILURE
{code}
With All trace from durable-task-plugin I see "heartbeat touches apparently not running in ..."

Since failures started about a month ago or so, it seem to be a regression of [https://github.com/jenkinsci/durable-task-plugin/commit/5c98ca855a9a2fb0043888c1bab9cc5f41c8773a]

I did also try to run 'ps' once a minute at the same agent host in parallel with this job. The sh processes with heartbeat and with sleep disappear right after the build fails. These are processes defined in [https://github.com/jenkinsci/durable-task-plugin/blob/5c98ca855a9a2fb0043888c1bab9cc5f41c8773a/src/main/java/org/jenkinsci/plugins/durabletask/BourneShellScript.java#L154]

Looking to the workspace folder and @tmp folder I see no pid file - is it expected?

Alexander Vorobiev made changes - 2018-01-22 10:53

Environment

Original: Jenkins ver. 2.89.2
Latest plugins for 01/17/2018
Agent ubuntu.1, launched via ssh
Ubuntu 14.04.5 LTS

New: Jenkins ver. 2.89.2
Remoting Version 3.14
Latest plugins for 01/17/2018
Agent ubuntu.1, launched via ssh
Ubuntu 14.04.5 LTS

Jeff Thompson made changes - 2018-10-29 22:31

Component/s

Original: remoting [ 15489 ]

Assignee:: Unassigned

Reporter:: Alexander Vorobiev

Votes:: 0 Vote for this issue

Watchers:: 6 Start watching this issue

Created:: 2018-01-17 10:15

Updated:: 2018-10-29 22:31

Jenkins

Details

Description

Attachments

Activity

People

Dates