Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52564

Plugin usage documentation may be incomplete/confusing

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Minor
    • Resolution: Fixed
    • Component/s: debian-pbuilder-plugin
    • Labels:
      None
    • Environment:
      debian-pbuilder-plugin - version 1.4

      Jenkins ver. 2.121.1

      All other plugins up to date (or updated to latest within a week)
    • Similar Issues:

      Description

       

      Sorry - I understand this may be a support call masquerading as an issue.  I'm unable to work out from first-principles and the documentation on plugins.jenkins.io and wiki.jenkins.io how to use this plugin.

      Where in a packaging workflow does this plugin sit?

      The references in the documentation to debian-jenkins-glue may be confusing or clarifying - I cannot tell which.   The example pipeline does not include any source data - is it supposed to be attached to the debian-jenkins-glue workflow somewhere?

      node(){
        ws{
          stage( "clean" ){
            cleanWs()
          }
          stage("build"){
            //Add whatever parameters you need to the class
            debianPbuilder()
          }
        }
      }
      

       

      Using the "baby steps" technique I tried the following to a package of moderate complexity direct from debian upstream source to exactly one pbuilder configuration (xenial, amd64):

      node('pbuilder') {
        stage("checkout") {
          cleanWs();
          checkout(changelog: true, poll: false,
          scm: [$class: 'GitSCM', branches: [[name: '*/debian/sid']],
                             doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'CleanBeforeCheckout']],
                             extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: 'source']], submoduleCfg: [],
                             userRemoteConfigs: [[url: 'https://salsa.debian.org/crosstoolchain-team/openocd.git']]])
          }
        stage("build") {
          debianPbuilder additionalBuildResults: '', architecture: 'amd64',
            distribution: 'xenial', keyring: '', mirrorSite: ''
        }
      }
      

      This fails as follows:

       

      ...
      [openocd-test] $ dpkg-source -b source
      dpkg-source: error: can't build with source format '3.0 (quilt)': no upstream tarball found at ../openocd_0.9.0.orig.tar.{bz2,gz,lzma,xz}
      

       

       The failure makes perfect sense "there is no source tarball" - because it is being built directly from git.

       

      Yet your example code in CSerial's Jenkinsfiles cleans the workspace after downloading the source (thus we lose the source).  I'm missing something obvious here.

       

      I then tried to replicate the debian-jenkins-glue configuration to generate the source tarballs (using stashing instead of separate pipelines until I figure this out):

       

      node('pbuilder') {
        stage("checkout") {
          cleanWs();
          scm_env = checkout(changelog: true, poll: false,
          scm: [$class: 'GitSCM', branches: [[name: '*/debian/sid']],
                             doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'CleanBeforeCheckout']],
                             extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: 'source']], submoduleCfg: [],
                             userRemoteConfigs: [[url: 'https://salsa.debian.org/crosstoolchain-team/openocd.git']]])
          }
        stage("generate") {
          withEnv(["GIT_BRANCH=${scm_env.GIT_BRANCH}","GIT_COMMIT=${scm_env.GIT_COMMIT}"]) {
            sh "generate-git-snapshot"
            stash includes: '*.gz,*.bz2,*.xz,*.deb,*.dsc,*.changes,*.buildinfo,lintian.txt', name: 'deb-source', useDefaultExcludes: false
          }
        }
        stage("build") {
          sh "rm ./* || true"
          unstash 'deb-source'
          // sh "ls -la"
          debianPbuilder additionalBuildResults: '', architecture: 'amd64',
            distribution: 'xenial', keyring: '', mirrorSite: ''
        }
      }
      

      Which gets a little further and fails here:

       

       

      [openocd-test] $ dpkg-genchanges -u. source
      dpkg-genchanges: unknown option 'source'
      

       

      My best interpretation of what is happening is that I may not have integrated this properly into the pipeline, since the failure looks like some code or parameters have turned into null strings, thus invoking the underlying tools incorrectly.

       

      Could you fill out the pipeline example so that it actually builds code? or perhaps further document how this build step is intended to be integrated?

      1. What input requirements are there?  source files? tarballs etc?
      2. Are there restrictions on what packages can be built?  Does it fail with 'quilt' packages built from git?  Does it require non-git packages (ie tarball + debian directories)

       

        Attachments

          Activity

          Hide
          rm5248 Robert Middleton added a comment -

          Thanks for the report.

          I've updated the documentation to have a better example for pipeline.  You can see an example of the Jenkinsfile here: https://github.com/rm5248/CSerial/blob/8e61cbf4dce9a9e8b47f026e3749b829198b13e9/Jenkinsfile

          I thought it was in the documentation, but at the moment the plugin doesn't create the orig.tar.gz files automatically, if the source format is anything besides 3.0(native).  As far as I can tell, the proper way to make the orig.tar.gz is to use pristine-tar to generate it; the documentation that I have been able to find is not very clear.

          The workaround for now is to use a shell script to either re-generate the orig.tar.gz or download it.  Here's your setup working: https://jenkins.rm5248.com/job/jenkins-52564/12/

          And here's the script that I used:

           

          node('master') {
            stage("checkout") {
              cleanWs();
              checkout(changelog: true, poll: false,
              scm: [$class: 'GitSCM', branches: [[name: '*/debian/sid']],
                                 doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'CleanBeforeCheckout']],
                                 extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: 'source']], submoduleCfg: [],
                                 userRemoteConfigs: [[url: 'https://salsa.debian.org/crosstoolchain-team/openocd.git']]])
              }
              stage("original"){
                  //This is where you would either re-create orig.tar.gz or run something like pristine-tar to get it
                  sh '''cp /tmp/openocd_0.9.0.orig.tar.gz .'''
              }
            stage("build") {
                //G01-enable-universe: echo "deb http://us.archive.ubuntu.com/ubuntu/ xenial universe" >> /etc/apt/sources.list
                configFileProvider([configFile(fileId: '01198bc4-beb9-4ee1-84e1-8f39ccae9971', targetLocation: 'hookdir/G01-enable-universe')]) {
              // some block
          }
              debianPbuilder additionalBuildResults: '', architecture: 'amd64',
                distribution: 'xenial', keyring: '/usr/share/keyrings/ubuntu-archive-keyring.gpg', 
                mirrorSite: 'http://archive.ubuntu.com/ubuntu'
            }
          }
          

           

           

          Show
          rm5248 Robert Middleton added a comment - Thanks for the report. I've updated the documentation to have a better example for pipeline.  You can see an example of the Jenkinsfile here: https://github.com/rm5248/CSerial/blob/8e61cbf4dce9a9e8b47f026e3749b829198b13e9/Jenkinsfile I thought it was in the documentation, but at the moment the plugin doesn't create the orig.tar.gz files automatically, if the source format is anything besides 3.0(native).  As far as I can tell, the proper way to make the orig.tar.gz is to use pristine-tar to generate it; the documentation that I have been able to find is not very clear. The workaround for now is to use a shell script to either re-generate the orig.tar.gz or download it.  Here's your setup working:  https://jenkins.rm5248.com/job/jenkins-52564/12/ And here's the script that I used:   node( 'master' ) { stage( "checkout" ) { cleanWs(); checkout(changelog: true , poll: false , scm: [$class: 'GitSCM' , branches: [[name: '*/debian/sid' ]], doGenerateSubmoduleConfigurations: false , extensions: [[$class: 'CleanBeforeCheckout' ]], extensions: [[$class: 'RelativeTargetDirectory' , relativeTargetDir: 'source' ]], submoduleCfg: [], userRemoteConfigs: [[url: 'https: //salsa.debian.org/crosstoolchain-team/openocd.git' ]]]) } stage( "original" ){ //This is where you would either re-create orig.tar.gz or run something like pristine-tar to get it sh '''cp /tmp/openocd_0.9.0.orig.tar.gz .' '' } stage( "build" ) { //G01-enable-universe: echo "deb http://us.archive.ubuntu.com/ubuntu/ xenial universe" >> /etc/apt/sources.list configFileProvider([configFile(fileId: '01198bc4-beb9-4ee1-84e1-8f39ccae9971' , targetLocation: 'hookdir/G01-enable-universe' )]) { // some block } debianPbuilder additionalBuildResults: '', architecture: ' amd64', distribution: 'xenial' , keyring: '/usr/share/keyrings/ubuntu-archive-keyring.gpg' , mirrorSite: 'http: //archive.ubuntu.com/ubuntu' } }    
          Hide
          whitty Greg Whiteley added a comment -

          Thanks - you've gone above and beyond, and also given me a tonne of info to work from .

          In the wiki update the single arch pipeline setup still has the confusing "cleanupWs before build" (ie delete the checkout).

          Pipeline Setup - single arch
          **
                      stage( }}{{"checkout" ){
                          checkout([$class: }}{{'GitSCM'{{, branches: }}
                                   {{[[name: }}{{'*/jenkinsfile-updates'}}{{]], }}
                                   extensions: [[$}}{{class}}{{: }}{{'RelativeTargetDirectory'}}{{, relativeTargetDir: }}{{'source'}}{{]], }}{{...{{}}
                      }
            
                      stage( }}{{"clean" ){
                          cleanWs()
                      }
           

          Is that intentional - is there something I'm missing?  Will the source directory somehow be bypassed (I haven't observed this previously).

           

          Show
          whitty Greg Whiteley added a comment - Thanks - you've gone above and beyond, and also given me a tonne of info to work from . In the wiki update the single arch pipeline setup still has the confusing "cleanupWs before build" (ie delete the checkout). Pipeline Setup - single arch **              stage( }}{{"checkout" ){                  checkout([$ class : }}{{'GitSCM' {{, branches: }}                           {{[ [name: }}{{'*/jenkinsfile-updates'}}{{] ], }}                           extensions: [ [$}}{{class}}{{: }}{{'RelativeTargetDirectory'}}{{, relativeTargetDir: }}{{'source'}}{{] ], }}{{... {{}}              }                  stage( }}{{"clean" ){                  cleanWs()              }   Is that intentional - is there something I'm missing?  Will the source directory somehow be bypassed (I haven't observed this previously).  
          Hide
          rm5248 Robert Middleton added a comment -

          That must be in the wrong place then.  The cleanWs() is needed, but it should happen before the checkout.  I've updated the page to reflect that.

          Show
          rm5248 Robert Middleton added a comment - That must be in the wrong place then.  The cleanWs() is needed, but it should happen before the checkout.  I've updated the page to reflect that.
          Hide
          rm5248 Robert Middleton added a comment -

          Marking this as fixed as no new problems have been reported.

           

          JENKINS-52645 was created as a result to track the current bug of not creating the orig.tar.gz automatically.

          Show
          rm5248 Robert Middleton added a comment - Marking this as fixed as no new problems have been reported.   JENKINS-52645 was created as a result to track the current bug of not creating the orig.tar.gz automatically.

            People

            Assignee:
            rm5248 Robert Middleton
            Reporter:
            whitty Greg Whiteley
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: