Event-based Dynamic Job Creation

XMLWordPrintable

      #### Proposal Summary
      
      This proposal outlines a native integration in the JobDSL plugin to automate the detection and processing of changes in
      the `.jobdsl` folder (located at the root directory of the target repository) during SCM merge events from providers
      like Gerrit, GitHub, BitBucket, and GitLab. By preprocessing scripts with dynamic placeholders and executing them
      directly, the plugin would enable seamless, real-time job updates in Jenkins, reducing manual intervention and aligning
      with GitOps principles. This eliminates reliance on external tools or scripts, leveraging Jenkins' built-in
      configurations for scalability across multi-repo environments.
      
      #### Key Objectives
      
      - **Automation:** Respond to merges by scanning for changes in the `.jobdsl` folder at the repository root (including
        any subpaths recursively), preprocessing `.groovy` scripts (e.g., substituting placeholders like `%%BRANCH%%` and
        deriving job names from folder paths), and invoking JobDSL execution.
      - **Integration:** Use Jenkins global and job-level configs for SCM credentials, processing rules, and multi-project
        handling.
      - **Outcomes:** Instant job/folder sync post-merge, with built-in validation, logging, and security.
      
      #### Detailed Proposal
      
      **Problem Statement:**  
      In organizations with high-velocity development, JobDSL scripts organized in the `.jobdsl` folder at the repository root
      require manual triggering after merges, disrupting workflows. Current setups demand custom scripts for placeholder
      resolution (e.g., branch-aware paths) and duplicate checks, adding complexity.
      
      **Proposed Solution:**  
      Extend the JobDSL plugin to include an SCM event listener that:
      
      - Registers webhooks (or polls) for merge events, parsing payloads to detect changes in the `.jobdsl` folder at the
        repository root (including any recursive subpaths).
      - Preprocesses affected `.groovy` files:
          - Resolves placeholders (e.g., `%%BRANCH%%` from merge target; `%%JOB_NAME%%` derived as "
            parent-project/repo/branch/script").
          - Applies configurable regex replacements and duplicate name prevention.
      - Executes via `GroovyDSLManager`, using internals to auto-generate nested Jenkins folders from paths (e.g., "
        payment/checkout-svc/develop/job-script" → folders "payment > checkout-svc > develop" with job "job-script").
      - Supports no-op for unchanged files, with console logging for traceability.
      
      **Illustrative Script Example:**  
      A preprocessed snippet might resolve as:
      
      ```groovy
      pipelineJob('payment/checkout-svc/develop/job-script') {  // Derived from path
          description('')
          disabled(false)
          parameters {
              stringParam {
                  trim(true); name('GERRIT_BRANCH'); defaultValue('develop')  // %%BRANCH%% resolved
                  description('The branch or the target branch of the change.')
              }
              stringParam {
                  trim(true); name('GERRIT_REFSPEC'); defaultValue('refs/heads/develop')
                  description('Change refspec')
              }
          }
          // ... (triggers, definition, etc.)
      }
      ```
      
      **Configuration Approach:**
      
      - **Global (Manage Jenkins > Configure System):** Define shared SCM servers (e.g., Gerrit API URLs, credential IDs),
        placeholder mappings, regex rules, and project scopes (included/excluded lists).
      
      **Implementation Considerations:**
      
      - Leverage existing Jenkins plugins (e.g., Gerrit Trigger) for auth and payload handling.
      - Focus on delta changes for performance; async processing for scale.
      
      **Acceptance Criteria:**
      
      - Successful end-to-end flow: Merge → Detection → Preprocess → Execute → Updated jobs/folders.
      - Handles edge cases: No changes, errors, duplicates (fail-fast with logs).
      - Secure: Webhook validation, credential isolation.
      - Testable: Via mocks and e2e scenarios.
      
      **Background/Context:**  
      Our setup uses hierarchical `.jobdsl` structures (a folder at the root of each repository) for branch-agnostic
      automation:
      
      ```
      .jobdsl/  # Root-level folder
      ├── parent-project/  # e.g., "payment"
      │   └── repo/        # e.g., "checkout-svc"
      │       └── %%BRANCH%%/  # e.g., resolves to "develop"
      │           └── job-script.groovy  # Job name derived from full path
      ```
      
      This enables zero-touch scaling, but manual post-merge runs hinder efficiency. Native support would consolidate this
      into the plugin.
      
      **Benefits:**
      
      - End-to-end automation from merge to job sync.
      - Scales for multi-repo/branch velocity.
      - Enforces org standards with minimal config. 

            Assignee:
            Unassigned
            Reporter:
            Gabriel
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: