Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44427

p4publish and unexpected timeout - 300 seconds

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • p4-plugin

      In case long publish (more than 300s) - pipeline failed with error:
      ERROR: java.lang.InterruptedException
      Short/small submits - no errors

      Network activity show normal submit (200+ mbit/s)

      p4d logs - brocken pipe

      Timeout period looks like not on p4 submit, but on all stack of p4publish groovy commands (ex: 21:02:56 - 21:07:56)

      [Pipeline] p4publish
21:02:56 ... p4 client -o XXX +
21:02:56 ... p4 info +
21:02:56 ... p4 client -o XXX +
21:02:56 ... p4 client -i +
21:02:56 ... client: XXX
21:02:56 ... p4 client -o XXX  +
21:02:56 ... p4 info +
21:02:56 
21:02:56 P4 Task: establishing connection.
21:02:56 ... server: XXX:1666
21:02:56 ... node: XXX
21:02:56 
21:02:56 P4 Task: reconcile files to changelist.
21:02:56 ... p4 revert -k //XXX/... +
21:02:56 ... p4 sync -k //XXX/... +
21:02:56 ... p4 reconcile -e -a -f -d //XXX/... +
21:02:59 ... p4 opened -CXXX //XXX/... +
21:02:59 duration: 0m 2s
21:02:59 
21:02:59 P4 Task: publish files to Perforce.
21:02:59 ... p4 change -i +
21:02:59 ... p4 change -o 343 +
21:02:59 ... pending change: 343
21:02:59 ... p4 reopen -c343 -t+S1 //XXX/... +
21:02:59 ... p4 opened -CXXX //XXX/... +
21:02:59 ... ... add //XXX
...1500+ files
21:02:59 ... ... add //XXX
21:03:00 ... p4 describe -s 343 +
21:03:00 ... p4 fixes -c343 +
21:03:00 ... p4 submit -i +
21:03:00 
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] node
21:07:56 Running on node_name in c:\jenkins-web\workspace\main_build
[Pipeline] {
[Pipeline] echo
[Pipeline] error
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] End of Pipeline
ERROR: java.lang.InterruptedException
Finished: FAILURE

      Pipeline(fragment):

       

         try {
        stage('Push results'){
            node(WN_NODE) {
                def p4_spec_name = "jenkins-${env.NODE_NAME}-${env.JOB_NAME}-bin"
                def p4_mapping = """XXX"""
                def p4_description = "${build_desc} ${job_tag}"
    p4publish credential: p4_credentials,
        publish: [$class: 'SubmitImpl', delete: true, description: p4_description, onlyOnSuccess: false, reopen: false, purge: '1'],
        workspace: [$class: 'ManualWorkspaceImpl', charset: 'auto', name: p4_spec_name, pinHost: false,
            spec: [allwrite: true, clobber: false, compress: false, line: 'LOCAL', locked: false, modtime: false, rmdir: true, streamName: '', view: p4_mapping]]

            }

        }
    }
    catch (err){
        node(WN_NODE){
                error "${err}"
        }
    }

       

      p4sync - no error on long/big sync > 300s

      After failure - in perforce remain good changelist, p4v submit his without problem and timeout limits.

       

      In p4 credentials RPC_SOCKET_SO_TIMEOUT_NICK == 0

       

      Jenkins docker-compose.yml

       

      version: '2'
services:
  jenkins:
    image: "jenkins:latest"
    restart: always
    ports:
      - 50000:50000
      - 8080:8080
    volumes:
      - ./jenkins_home:/var/jenkins_home
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"
    environment:
      - JAVA_OPTS="-Duser.timezone=Europe/Kiev"

       

       

      docker logs at this period:

      May 22, 2017 9:00:35 PM org.jenkinsci.plugins.p4.PerforceScm guessBrowser
INFO: Unable to guess repository browser.
May 22, 2017 9:07:56 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish
INFO: main_build #29 completed: FAILURE

       

       

        1. jenkins 300.png
          jenkins 300.png
          26 kB

          [JENKINS-44427] p4publish and unexpected timeout - 300 seconds

          Oleksii Trekhov created issue -

          Oleksii Trekhov added a comment - - edited

          looks like core problem - JNLP-slave ping:

          https://wiki.jenkins-ci.org/display/JENKINS/Features+controlled+by+system+properties

          hudson.slaves.ChannelPinger.pingIntervalSeconds 300 2.37 Frequency of pings between the master and slaves, in seconds

          reproduce on Windows slaves with

          • Launch agent via Java Web Start
          • Let Jenkins control this Windows slave as a Windows service

          Oleksii Trekhov added a comment - - edited looks like core problem - JNLP-slave ping: https://wiki.jenkins-ci.org/display/JENKINS/Features+controlled+by+system+properties hudson.slaves.ChannelPinger.pingIntervalSeconds 300 2.37 Frequency of  pings between the master and slaves , in seconds reproduce on Windows slaves with Launch agent via Java Web Start Let Jenkins control this Windows slave as a Windows service
          Oleksii Trekhov made changes -
          Environment Original: Jenkins ver. 2.46.2 (inside official docker image jenkins:latest)
          p4d Server version: P4D/LINUX26X86_64/2016.2/1509269 (2017/04/27)
          p4 Plugin ver: 1.7.0 + 1.6.2           		New: Jenkins ver. 2.46.2 (inside official docker image jenkins:latest)
          p4d Server version: P4D/LINUX26X86_64/2016.2/1509269 (2017/04/27)
          p4 Plugin ver: 1.7.0 + 1.6.2
          windows slaves (win10x64, java x64 ver. 8u131)
          Oleksii Trekhov made changes -
          Attachment New: jenkins 300.png [ 38114 ]

          Paul Allen added a comment -

          Have you been able to lengthen the time or disable it?  Did this resolve the issue?

          https://wiki.jenkins-ci.org/display/JENKINS/Ping+Thread

          Paul Allen added a comment - Have you been able to lengthen the time or disable it?  Did this resolve the issue? https://wiki.jenkins-ci.org/display/JENKINS/Ping+Thread

          Oleksii Trekhov added a comment -

          > Did this resolve the issue?

          No. Also in slave log

          May 23, 2017 11:04:47 PM FINE hudson.remoting.Channel
Received hudson.remoting.Request$Cancel@62a4211d
May 23, 2017 11:04:48 PM FINE hudson.remoting.Channel
Received UserRequest:hudson.Launcher$RemoteLauncher$KillTask@e61e62
May 23, 2017 11:04:48 PM FINER hudson.remoting.RemoteClassLoader
fetch3(hudson.Launcher$RemoteLauncher$KillTask)

          +All p4submit pipeline logs print in jenkins web only after submit success or submit error - looks like logs stuck on slave... like submit block slave thread or master-slave communication

          21:02:56 ... p4 client -o XXX +
21:02:56 ... p4 info +
21:02:56 ... p4 client -o XXX +
21:02:56 ... p4 client -i +
21:02:56 ... p4 client -o XXX  +
21:02:56 ... p4 info +
21:02:56 ... p4 revert -k //XXX/... +
21:02:56 ... p4 sync -k //XXX/... +
21:02:56 ... p4 reconcile -e -a -f -d //XXX/... +
21:02:59 ... p4 opened -CXXX //XXX/... +
21:02:59 ... p4 change -i +
21:02:59 ... p4 change -o 343 +
21:02:59 ... pending change: 343
21:02:59 ... p4 reopen -c343 -t+S1 //XXX/... +
21:02:59 ... p4 opened -CXXX //XXX/... +
21:03:00 ... p4 describe -s 343 +
21:03:00 ... p4 fixes -c343 +
21:03:00 ... p4 submit -i +

          Oleksii Trekhov added a comment - > Did this resolve the issue? No. Also in slave log May 23, 2017 11:04:47 PM FINE hudson.remoting.Channel Received hudson.remoting.Request$Cancel@62a4211d May 23, 2017 11:04:48 PM FINE hudson.remoting.Channel Received UserRequest:hudson.Launcher$RemoteLauncher$KillTask@e61e62 May 23, 2017 11:04:48 PM FINER hudson.remoting.RemoteClassLoader fetch3(hudson.Launcher$RemoteLauncher$KillTask) +All p4submit pipeline logs print in jenkins web only after submit success or submit error - looks like logs stuck on slave... like submit block slave thread or master-slave communication 21:02:56 ... p4 client -o XXX + 21:02:56 ... p4 info + 21:02:56 ... p4 client -o XXX + 21:02:56 ... p4 client -i + 21:02:56 ... p4 client -o XXX + 21:02:56 ... p4 info + 21:02:56 ... p4 revert -k //XXX/... + 21:02:56 ... p4 sync -k //XXX/... + 21:02:56 ... p4 reconcile -e -a -f -d //XXX/... + 21:02:59 ... p4 opened -CXXX //XXX/... + 21:02:59 ... p4 change -i + 21:02:59 ... p4 change -o 343 + 21:02:59 ... pending change: 343 21:02:59 ... p4 reopen -c343 -t+S1 //XXX/... + 21:02:59 ... p4 opened -CXXX //XXX/... + 21:03:00 ... p4 describe -s 343 + 21:03:00 ... p4 fixes -c343 + 21:03:00 ... p4 submit -i +

          Oleksii Trekhov added a comment -

          We try another new windows slave and new fresh jenkins master w/o unnecessary plugins - same fail after 300 secs...

          Oleksii Trekhov added a comment - We try another new windows slave and new fresh jenkins master w/o unnecessary plugins - same fail after 300 secs...

          Oleksii Trekhov added a comment -

          maybe https://www.jfrog.com/jira/browse/HAP-893 same bug?

          Oleksii Trekhov added a comment - maybe https://www.jfrog.com/jira/browse/HAP-893 same bug?

          Paul Allen added a comment -

          The submit is blocking.  It might be possible to make submit use the Streaming Asynchronous Callback mechanism used by sync.

          Paul Allen added a comment - The submit is blocking.  It might be possible to make submit use the Streaming Asynchronous Callback mechanism used by sync.

          Oleksii Trekhov added a comment -

          any terms for async?

          reproduce and on linux slave

          without master-slave communication:

          May 25, 2017 1:53:15 PM org.jenkinsci.plugins.p4.tasks.PublishTask task
          WARNING: Unable to publish workspace: com.perforce.p4java.exception.P4JavaException: Unable to submit change.
          May 25, 2017 1:53:15 PM org.jenkinsci.plugins.p4.tasks.AbstractTask tryTask
          WARNING: P4: Task Aborted!

          Oleksii Trekhov added a comment - any terms for async? reproduce and on linux slave without master-slave communication: May 25, 2017 1:53:15 PM org.jenkinsci.plugins.p4.tasks.PublishTask task WARNING: Unable to publish workspace: com.perforce.p4java.exception.P4JavaException: Unable to submit change. May 25, 2017 1:53:15 PM org.jenkinsci.plugins.p4.tasks.AbstractTask tryTask WARNING: P4: Task Aborted!

            p4karl Karl Wirth
            t_rex Oleksii Trekhov
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: