[JENKINS-72489] Not able to push pipeline projects data to InfluxDB2.7

Aleksi Simell added a comment - 2024-01-08 06:29

Hi, can you share your pipeline snippet? And there is no Jenkins 3.4 version; the latest version is 2.439.

Aleksi Simell added a comment - 2024-01-08 06:29 Hi, can you share your pipeline snippet? And there is no Jenkins 3.4 version; the latest version is 2.439.

Mehak added a comment - 2024-01-09 19:02 - edited

Hi,

correction:

InfluxDB Plugin Version: 3.4

InfluxDB Version: 2.7

I need to fetch change numbers against each job and push the data to influxdb2.7
Below is the code written to achieve this , but pipeline projects data is not getting written to influxdb2.7

pipeline {
    agent { label '<my node name>' }
    stages {
        stage('Print Echo') {
            steps {
                echo "hello"
            }
        }
    }
    post {
        always {
            script {
                try {
                    // Replace 'https://jenkins-url/' with the actual URL of your Jenkins instance
                    def jenkinsUrl = 'https://xxxxx/'
                    // Fetching the job names dynamically
                    def folderNamePrefix = 'PROD'
                    def jobNames = Jenkins.instance.items.findAll { it.fullName.startsWith(folderNamePrefix) }
                                       .collectMany { it.getAllJobs().collect { job -> "${it.fullName}/${job.name}" } }
                    // Map to store change numbers for each service
                    def changeNumbersMap = [:]
                    jobNames.each { jobName ->
                        // Constructing the job URL
                        def jobUrl = "${jenkinsUrl}/job/${jobName}"
                        // Fetching information specific to each job (changeManagement and jiraTicketId)
                        def changeManagement = fetchKeyValueFromLogs(jobUrl, 'Change_Management')
                        def jiraTicketId = fetchKeyValueFromLogs(jobUrl, 'jiraticket_id')
                        // Adding change number to the map for the current service
                        if (changeManagement && jiraTicketId) {
                            changeNumbersMap.computeIfAbsent(jobName, { [] }) << "${changeManagement}-${jiraTicketId}"
                        }
                        // Constructing InfluxDB data
                        def influxDbData = [
                            selectedTarget: 'Influx_DB_Grafana_Dashboard', // Match with your configured target name
                            customData: [
                                measurementName: 'jenkins_pipeline_data',
                                tags: [
                                    'service_name': jobName, // Assuming jobName is the service name
                                    'build_number': env.BUILD_NUMBER,
                                    'result': currentBuild.currentResult == 'SUCCESS' ? 'SUCCESS' : 'FAILURE',
                                    'change_management': changeManagement,
                                    'jiraticket_id': jiraTicketId,
                                    // Add more tags as needed
                                ],
                                fields: [
                                    duration: currentBuild.duration,
                                    timestamp: System.currentTimeMillis()
                                ]
                            ]
                        ]
                        // Print messages for debugging
                        echo "[InfluxDB Plugin] Collecting data..."
                        echo "[InfluxDB Plugin] Custom data found. Writing to InfluxDB..."
                        echo "[InfluxDB Plugin] Git data found. Writing to InfluxDB..."
                        echo "[InfluxDB plugin] Metrics plugin data found. Writing to InfluxDB..."
                        // Print the InfluxDB data for debugging
                        echo "InfluxDB Data: ${influxDbData}"
                        // Assuming InfluxDB write command
                        def influxDbWriteCommand = """
                            influx write --host https://xxx.com:8086 --token <influxdb token> --org <org name> --bucket <influx-bucket-name> --precision ns
                        """
                        // Assuming InfluxDB write command
                        def influxDbExecution = sh(script: "echo ${influxDbData} | ${influxDbWriteCommand}", returnStatus: true)
                        // Print the output and status of the InfluxDB execution
                        echo "InfluxDB Output: ${influxDbExecution}"
                        echo "InfluxDB Status Code: ${influxDbExecution}"
                        // Move the "Data sent to InfluxDB successfully!" message here
                        echo "[InfluxDB Plugin] Publishing data to target 'Influx_DB_Grafana_Dashboard' (url='https:xxxx:8086/', database='jenkins')"
                        // Check the status code and terminate if not successful
                        if (influxDbExecution != 0) {
                            error "[InfluxDB plugin] failed with status code: ${influxDbExecution}"
                        }
                    }
                    // Additional debug information
                    echo "Jenkins Job Name: ${env.JOB_NAME}"
                    echo "Jenkins Build Number: ${env.BUILD_NUMBER}"
                    echo "[InfluxDB Plugin] Data sent to InfluxDB successfully!"
                } catch (Exception e) {
                    echo "[InfluxDB Plugin] Error sending data to InfluxDB: ${e.message}"
                    currentBuild.result = 'FAILURE'
                    error "[InfluxDB Plugin] Failed to send data to InfluxDB"
                }
            }
        }
    }
}

======================

I am running this code from jenkins pipeline project.
below is the output.

when i logged in to influxdb , there is no data found.

03:31:07  hello
03:31:07  [Pipeline] }
03:31:07  [Pipeline] // stage
03:31:07  [Pipeline] stage
03:31:07  [Pipeline] { (Declarative: Post Actions)
03:31:07  [Pipeline] script
03:31:07  [Pipeline] {
03:31:07  [Pipeline] echo
03:31:07  Jenkins Job Name: Central_DevOps_Team/secondtest-pipeline
03:31:07  [Pipeline] echo
03:31:07  Jenkins Build Number: 75
03:31:07  [Pipeline] echo
03:31:07  [InfluxDB Plugin] Data sent to InfluxDB successfully!
03:31:07  [Pipeline] }
03:31:07  [Pipeline] // script
03:31:07  [Pipeline] }
03:31:07  [Pipeline] // stage
03:31:07  [Pipeline] }
03:31:08  [Pipeline] // node
03:31:08  [Pipeline] End of Pipeline
03:31:08  [InfluxDB Plugin] Collecting data...
03:31:08  [InfluxDB Plugin] Git data found. Writing to InfluxDB...
03:31:08  Finished: SUCCESS

Mehak added a comment - 2024-01-09 19:02 - edited Hi, correction: InfluxDB Plugin Version: 3.4 InfluxDB Version: 2.7 I need to fetch change numbers against each job and push the data to influxdb2.7 Below is the code written to achieve this , but pipeline projects data is not getting written to influxdb2.7 pipeline { agent { label '<my node name>' } stages { stage( 'Print Echo' ) { steps { echo "hello" } } } post { always { script { try { // Replace 'https://jenkins-url/' with the actual URL of your Jenkins instance def jenkinsUrl = 'https: //xxxxx/' // Fetching the job names dynamically def folderNamePrefix = 'PROD' def jobNames = Jenkins.instance.items.findAll { it.fullName.startsWith(folderNamePrefix) } .collectMany { it.getAllJobs().collect { job -> "${it.fullName}/${job.name}" } } // Map to store change numbers for each service def changeNumbersMap = [:] jobNames.each { jobName -> // Constructing the job URL def jobUrl = "${jenkinsUrl}/job/${jobName}" // Fetching information specific to each job (changeManagement and jiraTicketId) def changeManagement = fetchKeyValueFromLogs(jobUrl, 'Change_Management' ) def jiraTicketId = fetchKeyValueFromLogs(jobUrl, 'jiraticket_id' ) // Adding change number to the map for the current service if (changeManagement && jiraTicketId) { changeNumbersMap.computeIfAbsent(jobName, { [] }) << "${changeManagement}-${jiraTicketId}" } // Constructing InfluxDB data def influxDbData = [ selectedTarget: 'Influx_DB_Grafana_Dashboard' , // Match with your configured target name customData: [ measurementName: 'jenkins_pipeline_data' , tags: [ 'service_name' : jobName, // Assuming jobName is the service name 'build_number' : env.BUILD_NUMBER, 'result' : currentBuild.currentResult == 'SUCCESS' ? 'SUCCESS' : 'FAILURE' , 'change_management' : changeManagement, 'jiraticket_id' : jiraTicketId, // Add more tags as needed ], fields: [ duration: currentBuild.duration, timestamp: System .currentTimeMillis() ] ] ] // Print messages for debugging echo "[InfluxDB Plugin] Collecting data..." echo "[InfluxDB Plugin] Custom data found. Writing to InfluxDB..." echo "[InfluxDB Plugin] Git data found. Writing to InfluxDB..." echo "[InfluxDB plugin] Metrics plugin data found. Writing to InfluxDB..." // Print the InfluxDB data for debugging echo "InfluxDB Data: ${influxDbData}" // Assuming InfluxDB write command def influxDbWriteCommand = """ influx write --host https: //xxx.com:8086 --token <influxdb token> --org <org name> --bucket <influx-bucket-name> --precision ns """ // Assuming InfluxDB write command def influxDbExecution = sh(script: "echo ${influxDbData} | ${influxDbWriteCommand}" , returnStatus: true ) // Print the output and status of the InfluxDB execution echo "InfluxDB Output: ${influxDbExecution}" echo "InfluxDB Status Code: ${influxDbExecution}" // Move the "Data sent to InfluxDB successfully!" message here echo "[InfluxDB Plugin] Publishing data to target 'Influx_DB_Grafana_Dashboard' (url= 'https:xxxx:8086/' , database= 'jenkins' )" // Check the status code and terminate if not successful if (influxDbExecution != 0) { error "[InfluxDB plugin] failed with status code: ${influxDbExecution}" } } // Additional debug information echo "Jenkins Job Name: ${env.JOB_NAME}" echo "Jenkins Build Number : ${env.BUILD_NUMBER}" echo "[InfluxDB Plugin] Data sent to InfluxDB successfully!" } catch (Exception e) { echo "[InfluxDB Plugin] Error sending data to InfluxDB: ${e.message}" currentBuild.result = 'FAILURE' error "[InfluxDB Plugin] Failed to send data to InfluxDB" } } } } } ====================== I am running this code from jenkins pipeline project. below is the output. when i logged in to influxdb , there is no data found. 03:31:07 hello 03:31:07 [Pipeline] } 03:31:07 [Pipeline] // stage 03:31:07 [Pipeline] stage 03:31:07 [Pipeline] { (Declarative: Post Actions) 03:31:07 [Pipeline] script 03:31:07 [Pipeline] { 03:31:07 [Pipeline] echo 03:31:07 Jenkins Job Name: Central_DevOps_Team/secondtest-pipeline 03:31:07 [Pipeline] echo 03:31:07 Jenkins Build Number : 75 03:31:07 [Pipeline] echo 03:31:07 [InfluxDB Plugin] Data sent to InfluxDB successfully! 03:31:07 [Pipeline] } 03:31:07 [Pipeline] // script 03:31:07 [Pipeline] } 03:31:07 [Pipeline] // stage 03:31:07 [Pipeline] } 03:31:08 [Pipeline] // node 03:31:08 [Pipeline] End of Pipeline 03:31:08 [InfluxDB Plugin] Collecting data... 03:31:08 [InfluxDB Plugin] Git data found. Writing to InfluxDB... 03:31:08 Finished: SUCCESS

Aleksi Simell added a comment - 2024-01-10 06:01 - edited

Looks like you're not even calling the plugin. You're implementing your own write command and adding "[InfluxDB Plugin]" into your own echo commands to match what the plugin should be printing and a few extras, which the plugin does not print at all.

You're fetching data from other jobs. The easiest solution would be to just call the plugin from each job individually. If you insist on sending data from all jobs in this single job you can do that by calling

influxDbPublisher(selectedTarget: 'my-target', customDataMap: myCustomMeasurementFields, customDataMapTags: myTags)

as explained in the plugin documentation.

Aleksi Simell added a comment - 2024-01-10 06:01 - edited Looks like you're not even calling the plugin. You're implementing your own write command and adding " [InfluxDB Plugin] " into your own echo commands to match what the plugin should be printing and a few extras, which the plugin does not print at all. You're fetching data from other jobs. The easiest solution would be to just call the plugin from each job individually. If you insist on sending data from all jobs in this single job you can do that by calling influxDbPublisher(selectedTarget: 'my-target' , customDataMap: myCustomMeasurementFields, customDataMapTags: myTags) as explained in the plugin documentation .

Mehak added a comment - 2024-01-10 06:36

I tried one more snippet , please have a look.

def fetchKeyValueFromLogs(key) {
    def logs = getJobLogs() 
    def pattern = "${key}=(\\S+)"
    def matcher = (logs =~ pattern)    return matcher ? matcher[0][1] : ''
}def getJobLogs() {
    // Implement logic to fetch job logs
    // Example: return someFunctionToFetchJobLogs()
    return ''
}pipeline {
    agent { label 'xxxx' }    stages {
        stage('Print Echo') {
            steps {
                echo "hello"
            }
        }
    }    post {
        always {
            script {
                try {
                    // Convert boolean to string
                    //def resultString = currentBuild.currentResult == 'SUCCESS' ? 'SUCCESS' : 'FAILURE'                    def changeManagement = fetchKeyValueFromLogs('Change_Management')
                    def jiraTicketId = fetchKeyValueFromLogs('jiraticket_id')                    influxDbPublisher(
                        selectedTarget: 'Influx_DB_Grafana_Dashboard',
                        customData: [
                            measurementName: 'jenkins_pipeline_data_spm',
                            tags: [
                                //'job_name': env.JOB_NAME,
                                //'service_name': jobName, // Assuming jobName is the service name
                                'build_number': env.BUILD_NUMBER,
                                'result': currentBuild.currentResult == 'SUCCESS' ? 'SUCCESS' : 'FAILURE',
                                'description': 'Influx_DB_Grafana_Dashboard',
                                'url': 'https://xxxx:8086/',
                                'change_management': changeManagement,
                                'jiraticket_id': jiraTicketId
                            ],
                            fields: [
                                duration: currentBuild.duration,
                                timestamp: System.currentTimeMillis()
                            ]
                        ]
                    )                    echo "Data sent to InfluxDB successfully!"                    // Print Change Management and Jira Ticket ID
                    echo "Change Management: ${changeManagement}"
                    echo "Jira Ticket ID: ${jiraTicketId}"                } catch (Exception e) {
                    echo "Error sending data to InfluxDB: ${e.message}"
                }
            }
        }
    }
}

output is - error - invalid boolean

 [InfluxDB Plugin] Collecting data...
04:05:48  [InfluxDB Plugin] Custom data found. Writing to InfluxDB...
04:05:48  [InfluxDB Plugin] Git data found. Writing to InfluxDB...
04:05:48  [InfluxDB plugin] Metrics plugin data found. Writing to InfluxDB...
04:05:48  [InfluxDB Plugin] Publishing data to target 'Influx_DB_Grafana_Dashboard' (url='https:xxx:8086/', database='jenkins')
04:05:48  [Pipeline] echo
04:05:48  Error sending data to InfluxDB: com.influxdb.exceptions.BadRequestException: unable to parse 'jenkins_custom_data,instance=https:xxxx/,project_name=secondtest-pipeline,project_namespace=Central_DevOps_Team,project_path=Central_DevOps_Team/secondtest-pipeline build_number=61i,build_time=2037i,fields={duration=2019, timestamp=1704321348318},measurementName="jenkins_pipeline_data_spm",project_name="secondtest-pipeline",project_path="Central_DevOps_Team/secondtest-pipeline",tags={build_number=61, result=SUCCESS, description=Influx_DB_Grafana_Dashboard, url=xxx:8086/, change_management=, jiraticket_id=} 1704321346294000000': invalid boolean

Mehak added a comment - 2024-01-10 06:36 I tried one more snippet , please have a look. def fetchKeyValueFromLogs(key) { def logs = getJobLogs() def pattern = "${key}=(\\S+)" def matcher = (logs =~ pattern) return matcher ? matcher[0][1] : '' }def getJobLogs() { // Implement logic to fetch job logs // Example: return someFunctionToFetchJobLogs() return '' }pipeline { agent { label 'xxxx' } stages { stage( 'Print Echo' ) { steps { echo "hello" } } } post { always { script { try { // Convert boolean to string //def resultString = currentBuild.currentResult == 'SUCCESS' ? 'SUCCESS' : 'FAILURE' def changeManagement = fetchKeyValueFromLogs( 'Change_Management' ) def jiraTicketId = fetchKeyValueFromLogs( 'jiraticket_id' ) influxDbPublisher( selectedTarget: 'Influx_DB_Grafana_Dashboard' , customData: [ measurementName: 'jenkins_pipeline_data_spm' , tags: [ // 'job_name' : env.JOB_NAME, // 'service_name' : jobName, // Assuming jobName is the service name 'build_number' : env.BUILD_NUMBER, 'result' : currentBuild.currentResult == 'SUCCESS' ? 'SUCCESS' : 'FAILURE' , 'description' : 'Influx_DB_Grafana_Dashboard' , 'url' : 'https: //xxxx:8086/' , 'change_management' : changeManagement, 'jiraticket_id' : jiraTicketId ], fields: [ duration: currentBuild.duration, timestamp: System .currentTimeMillis() ] ] ) echo "Data sent to InfluxDB successfully!" // Print Change Management and Jira Ticket ID echo "Change Management: ${changeManagement}" echo "Jira Ticket ID: ${jiraTicketId}" } catch (Exception e) { echo "Error sending data to InfluxDB: ${e.message}" } } } } } output is - error - invalid boolean [InfluxDB Plugin] Collecting data... 04:05:48 [InfluxDB Plugin] Custom data found. Writing to InfluxDB... 04:05:48 [InfluxDB Plugin] Git data found. Writing to InfluxDB... 04:05:48 [InfluxDB plugin] Metrics plugin data found. Writing to InfluxDB... 04:05:48 [InfluxDB Plugin] Publishing data to target 'Influx_DB_Grafana_Dashboard' (url= 'https:xxx:8086/' , database= 'jenkins' ) 04:05:48 [Pipeline] echo 04:05:48 Error sending data to InfluxDB: com.influxdb.exceptions.BadRequestException: unable to parse 'jenkins_custom_data,instance=https:xxxx/,project_name=secondtest-pipeline,project_namespace=Central_DevOps_Team,project_path=Central_DevOps_Team/secondtest-pipeline build_number=61i,build_time=2037i,fields={duration=2019, timestamp=1704321348318},measurementName= "jenkins_pipeline_data_spm" ,project_name= "secondtest-pipeline" ,project_path= "Central_DevOps_Team/secondtest-pipeline" ,tags={build_number=61, result=SUCCESS, description=Influx_DB_Grafana_Dashboard, url=xxx:8086/, change_management=, jiraticket_id=} 1704321346294000000' : invalid boolean

Aleksi Simell added a comment - 2024-01-10 07:19

The keys for customData need to be strings, so for example "measurementName" instead of measurementName.

Also, your custom data doesn't seem to make any sense. I assume you're trying to add a custom measurement with some tags and obviously with some custom fields. If that's the case, you need to use customDataMap instead of customData. But, it looks like the only field you're trying to add is the build time, which is already available in the standard jenkins_data measurement as listed in available metrics documentation. Also, build_number is automatically added to all measurements, as explained in the available metrics documentation.

Aleksi Simell added a comment - 2024-01-10 07:19 The keys for customData need to be strings, so for example "measurementName" instead of measurementName . Also, your custom data doesn't seem to make any sense. I assume you're trying to add a custom measurement with some tags and obviously with some custom fields. If that's the case, you need to use customDataMap instead of customData . But, it looks like the only field you're trying to add is the build time, which is already available in the standard jenkins_data measurement as listed in available metrics documentation . Also, build_number is automatically added to all measurements, as explained in the available metrics documentation.

Mehak added a comment - 2024-01-10 11:16

I have tried multiple snippets , but not able to achieve the desired output,these are just 2 code snippets which I shared with you.

I want to fetch job name/ pipeline data of each project and all change requests numbers associated with each job.

Could you please help me with the snippet?

Mehak added a comment - 2024-01-10 11:16 I have tried multiple snippets , but not able to achieve the desired output,these are just 2 code snippets which I shared with you. I want to fetch job name/ pipeline data of each project and all change requests numbers associated with each job. Could you please help me with the snippet?

Aleksi Simell added a comment - 2024-01-10 12:15

As I mentioned, by far the easiest method would be to just add InfluxDB plugin to each job individually, so that the data is sent automatically.

However, if you persistently want to have this separate job to gather the data...

First, you need to gather your customDataMap (not customData).

def dataMap = [:]
def fields = [:]
fields["key"] = "value"
fields["another_key"] = "some other value"
dataMap["my_measurement"] = fields

In this simple example, I create a customDataMap called "dataMap" with some fields provided by the "fields" map. I then give my dataMap a new measurement called "my_measurement" and give it the fields-map as its value. In your case this would mean the job name and pipeline data of each project.

Next, if you want your custom measurement to have some tags, you need to add those to another map.

def tags = [:]
tags["my_measurement"] = ["tag1": "value 1", "tag2": "value"] # etc.

Note that tags-map must share the map keys with your dataMap.

Finally, you need to call InfluxDbPublisher with your custom data

influxDbPublisher(selectedTarget: "my-target", customDataMap: dataMap, customDataMapTags: tags)

Aleksi Simell added a comment - 2024-01-10 12:15 As I mentioned, by far the easiest method would be to just add InfluxDB plugin to each job individually, so that the data is sent automatically. However, if you persistently want to have this separate job to gather the data... First, you need to gather your customDataMap (not customData). def dataMap = [:] def fields = [:] fields[ "key" ] = "value" fields[ "another_key" ] = "some other value" dataMap[ "my_measurement" ] = fields In this simple example, I create a customDataMap called "dataMap" with some fields provided by the "fields" map. I then give my dataMap a new measurement called "my_measurement" and give it the fields-map as its value. In your case this would mean the job name and pipeline data of each project. Next, if you want your custom measurement to have some tags, you need to add those to another map. def tags = [:] tags[ "my_measurement" ] = [ "tag1" : "value 1" , "tag2" : "value" ] # etc. Note that tags-map must share the map keys with your dataMap. Finally, you need to call InfluxDbPublisher with your custom data influxDbPublisher(selectedTarget: "my-target" , customDataMap: dataMap, customDataMapTags: tags)

Details

Description

Attachments

Activity

Collapse comment: Aleksi Simell added a comment - 2024-01-08 06:29

Expand comment: Aleksi Simell added a comment - 2024-01-08 06:29

Collapse comment: Mehak added a comment - 2024-01-09 19:02, Edited by Mehak - 2024-01-09 19:03

Expand comment: Mehak added a comment - 2024-01-09 19:02, Edited by Mehak - 2024-01-09 19:03

Collapse comment: Aleksi Simell added a comment - 2024-01-10 06:01, Edited by Aleksi Simell - 2024-01-10 06:01

Expand comment: Aleksi Simell added a comment - 2024-01-10 06:01, Edited by Aleksi Simell - 2024-01-10 06:01

Collapse comment: Mehak added a comment - 2024-01-10 06:36

Expand comment: Mehak added a comment - 2024-01-10 06:36

Collapse comment: Aleksi Simell added a comment - 2024-01-10 07:19

Expand comment: Aleksi Simell added a comment - 2024-01-10 07:19

Collapse comment: Mehak added a comment - 2024-01-10 11:16

Expand comment: Mehak added a comment - 2024-01-10 11:16

Collapse comment: Aleksi Simell added a comment - 2024-01-10 12:15

Expand comment: Aleksi Simell added a comment - 2024-01-10 12:15

People

Dates