Uploaded image for project: 'Infrastructure'
  1. Infrastructure
  2. INFRA-3106

[ci.jenkins.io] Build using the label {node} are blocked in the queue

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      The repository https://github.com/jenkinsci/.github/tree/master has its builds on ci.jenkins.io that are blocked, waiting in the queue.

      The build messages states that it's aiting for agent with the label

      {node}

      to be allocated.

      The agents with label "nodes" are pod templates (as per https://github.com/jenkins-infra/jenkins-infra/blob/production/hieradata/clients/azure.ci.jenkins.io.yaml#L98-L102) that execute a container based on the Docker image specified here: https://github.com/jenkins-infra/jenkins-infra/blob/production/hieradata/common.yaml#L308.

      We have to enable the pod retention (when on failure, instead of "never" by default) and determine what is the error (is it a pod event or in the logs during the entrypoint startup for container?).

        Attachments

          Activity

          Hide
          dduportal Damien Duportal added a comment -

          cc Oleg Nenashev for information (as you seem to have interacted with the repository recently)

          Show
          dduportal Damien Duportal added a comment - cc Oleg Nenashev for information (as you seem to have interacted with the repository recently)
          Hide
          dduportal Damien Duportal added a comment -
          • Enabling retention on failed container for {jnlp-node}

            pod template (+ disabling puppet agent to keep this setting for now).

          • Got the following log: the error come from the image itself (java not found)
            ➜  jenkins-infra git:(fix/ci-agent-jnlp-nodejs) kubectl get pod -n jenkins-agents                                               
            NAME                  READY   STATUS              RESTARTS   AGE
            jnlp-maven-11-0hjm8   0/1     ContainerCreating   0          4s
            jnlp-maven-11-4xkjq   1/1     Running             0          2m2s
            jnlp-maven-11-6626h   1/1     Running             0          2m2s
            jnlp-maven-11-cr018   1/1     Running             0          2m2s
            jnlp-maven-11-d1p0k   1/1     Running             0          78m
            jnlp-maven-11-w2b5s   0/1     ContainerCreating   0          4s
            jnlp-maven-11-wxsv7   1/1     Running             0          2m2s
            jnlp-maven-8-gr3sh    1/1     Running             0          78m
            jnlp-node-4r4g4       0/1     Error               0          14s
            jnlp-node-6dmpj       0/1     Error               0          14s
            jnlp-node-d0c07       0/1     ContainerCreating   0          14s
            jnlp-node-j98vs       0/1     ContainerCreating   0          14s
            jnlp-node-q9j6t       0/1     ContainerCreating   0          14s
            ➜  jenkins-infra git:(fix/ci-agent-jnlp-nodejs) kubectl -n jenkins-agents logs jnlp-node-4r4g4
            /usr/local/bin/jenkins-agent: exec: line 121: /opt/jdk-11/bin/java: not found
            
          Show
          dduportal Damien Duportal added a comment - Enabling retention on failed container for {jnlp-node} pod template (+ disabling puppet agent to keep this setting for now). Got the following log: the error come from the image itself (java not found) ➜ jenkins-infra git:(fix/ci-agent-jnlp-nodejs) kubectl get pod -n jenkins-agents NAME READY STATUS RESTARTS AGE jnlp-maven-11-0hjm8 0/1 ContainerCreating 0 4s jnlp-maven-11-4xkjq 1/1 Running 0 2m2s jnlp-maven-11-6626h 1/1 Running 0 2m2s jnlp-maven-11-cr018 1/1 Running 0 2m2s jnlp-maven-11-d1p0k 1/1 Running 0 78m jnlp-maven-11-w2b5s 0/1 ContainerCreating 0 4s jnlp-maven-11-wxsv7 1/1 Running 0 2m2s jnlp-maven-8-gr3sh 1/1 Running 0 78m jnlp-node-4r4g4 0/1 Error 0 14s jnlp-node-6dmpj 0/1 Error 0 14s jnlp-node-d0c07 0/1 ContainerCreating 0 14s jnlp-node-j98vs 0/1 ContainerCreating 0 14s jnlp-node-q9j6t 0/1 ContainerCreating 0 14s ➜ jenkins-infra git:(fix/ci-agent-jnlp-nodejs) kubectl -n jenkins-agents logs jnlp-node-4r4g4 /usr/local/bin/jenkins-agent: exec: line 121: /opt/jdk-11/bin/java: not found
          Hide
          dduportal Damien Duportal added a comment -
          Show
          dduportal Damien Duportal added a comment - Issue seems to be related to the Docker image/entrypoint and its java installation Docker image is defined here: https://github.com/jenkins-infra/docker-inbound-agents/blob/main/node/Dockerfile
          Hide
          dduportal Damien Duportal added a comment -

          Issue found: it's related to the JDK in Alpine. Let's use debian base. PR: https://github.com/jenkins-infra/docker-inbound-agents/pull/8

          Show
          dduportal Damien Duportal added a comment - Issue found: it's related to the JDK in Alpine. Let's use debian base. PR: https://github.com/jenkins-infra/docker-inbound-agents/pull/8
          Hide
          dduportal Damien Duportal added a comment -
          • PR merged: Docker image built and deployed successfully
          • Applied to ci.j and trusted.ci.j with https://github.com/jenkins-infra/jenkins-infra/pull/1935
          • Puppet agent enabled again: ran successfully and applied the change
          • Builds are un-queued, with jnlp-node pods starting successfully
          • Errored pods cleaned up
          Show
          dduportal Damien Duportal added a comment - PR merged: Docker image built and deployed successfully Applied to ci.j and trusted.ci.j with https://github.com/jenkins-infra/jenkins-infra/pull/1935 Puppet agent enabled again: ran successfully and applied the change Builds are un-queued, with jnlp-node pods starting successfully Errored pods cleaned up
          Hide
          dduportal Damien Duportal added a comment -

          Thanks for the 2nd pair of eye Hervé!

          Show
          dduportal Damien Duportal added a comment - Thanks for the 2nd pair of eye Hervé !

            People

            Assignee:
            dduportal Damien Duportal
            Reporter:
            dduportal Damien Duportal
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: