-
Bug
-
Resolution: Fixed
-
Blocker
-
None
-
Jenkins 2.440.1 Piugin version 0.2
I see this error from time on console output.
My environment is :
Jenkins controller on docker in AWS ECS/EC2
Jenkins workers are docker containers running as ECS tasks
com.amazonaws.services.logs.model.InvalidParameterException: 1 validation error detected: Value '[vmol-develop-apps@agent1, vmol-develop-apps@agent10, vmol-develop-apps@agent100, vmol-develop-apps@agent101, vmol-develop-apps@agent102, vmol-develop-apps@agent103, vmol-develop-apps@agent104, vmol-develop-apps@agent105, vmol-develop-apps@agent106, vmol-develop-apps@agent107, vmol-develop-apps@agent108, vmol-develop-apps@agent109, vmol-develop-apps@agent11, vmol-develop-apps@agent110, vmol-develop-apps@agent111, vmol-develop-apps@agent112, vmol-develop-apps@agent113, vmol-develop-apps@agent114, vmol-develop-apps@agent115, vmol-develop-apps@agent116, vmol-develop-apps@agent117, vmol-develop-apps@agent118, vmol-develop-apps@agent119, vmol-develop-apps@agent12, vmol-develop-apps@agent120, vmol-develop-apps@agent121, vmol-develop-apps@agent122, vmol-develop-apps@agent123, vmol-develop-apps@agent124, vmol-develop-apps@agent125, vmol-develop-apps@agent126, vmol-develop-apps@agent127, vmol-develop-apps@agent128, vmol-develop-apps@agent129, vmol-develop-apps@agent13, vmol-develop-apps@agent130, vmol-develop-apps@agent131, vmol-develop-apps@agent132, vmol-develop-apps@agent133, vmol-develop-apps@agent134, vmol-develop-apps@agent135, vmol-develop-apps@agent136, vmol-develop-apps@agent137, vmol-develop-apps@agent138, vmol-develop-apps@agent139, vmol-develop-apps@agent14, vmol-develop-apps@agent140, vmol-develop-apps@agent141, vmol-develop-apps@agent142, vmol-develop-apps@agent143, vmol-develop-apps@agent144, vmol-develop-apps@agent145, vmol-develop-apps@agent146, vmol-develop-apps@agent147, vmol-develop-apps@agent148, vmol-develop-apps@agent149, vmol-develop-apps@agent15, vmol-develop-apps@agent150, vmol-develop-apps@agent151, vmol-develop-apps@agent152, vmol-develop-apps@agent153, vmol-develop-apps@agent154, vmol-develop-apps@agent155, vmol-develop-apps@agent156, vmol-develop-apps@agent157, vmol-develop-apps@agent158, vmol-develop-apps@agent159, vmol-develop-apps@agent16, vmol-develop-apps@agent160, vmol-develop-apps@agent161, vmol-develop-apps@agent162, vmol-develop-apps@agent163, vmol-develop-apps@agent164, vmol-develop ...' at 'logStreamNames' failed to satisfy constraint: Member must have length less than or equal to 100 (Service: AWSLogs; Status Code: 400; Error Code: InvalidParameterException; Request ID: cb59caac-3fd0-4e2b-ba14-7d6d63fa3b29; Proxy: null) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)
Operating System (including version, and whether it’s 32/64 bit) of all systems involved (your client’s, the Jenkins server’s, all agent nodes'),
uname -a Linux ip-10-0-3-57.ap-southeast-2.compute.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 GNU/Linux
All relevant JRE JDK vendors and versions and the parameters set
JDK : lts-jdk11 ENV JAVA_OPTS='-Djenkins.install.runSetupWizard=false \ -Djava.net.preferIPv4Stack=true \ -Dorg.apache.commons.jelly.tags.fmt.timeZone=TZ=Australia/Melbourne \ -Duser.timezone=Australia/Melbourne \ -Dhudson.model.ParametersAction.safeParameters=sqs_body,sqs_message_id'
Jenkins and plugin versions
Jenkins: 2.440.1 OS: Linux - 4.14.336-257.562.amzn2.x86_64 Java: 11.0.22 - Eclipse Adoptium (OpenJDK 64-Bit Server VM) --- amazon-ecs:1.49 antisamy-markup-formatter:162.v0e6ec0fcfcf6 apache-httpcomponents-client-4-api:4.5.14-208.v438351942757 authentication-tokens:1.53.v1c90fd9191a_b_ aws-credentials:218.v1b_e9466ec5da_ aws-global-configuration:130.v35b_7b_96f53c3 aws-java-sdk:1.12.633-430.vf9a_e567a_244f aws-java-sdk-cloudformation:1.12.633-430.vf9a_e567a_244f aws-java-sdk-codebuild:1.12.633-430.vf9a_e567a_244f aws-java-sdk-ec2:1.12.633-430.vf9a_e567a_244f aws-java-sdk-ecr:1.12.633-430.vf9a_e567a_244f aws-java-sdk-ecs:1.12.633-430.vf9a_e567a_244f aws-java-sdk-efs:1.12.633-430.vf9a_e567a_244f aws-java-sdk-elasticbeanstalk:1.12.633-430.vf9a_e567a_244f aws-java-sdk-iam:1.12.633-430.vf9a_e567a_244f aws-java-sdk-kinesis:1.12.633-430.vf9a_e567a_244f aws-java-sdk-logs:1.12.633-430.vf9a_e567a_244f aws-java-sdk-minimal:1.12.633-430.vf9a_e567a_244f aws-java-sdk-secretsmanager:1.12.633-430.vf9a_e567a_244f aws-java-sdk-sns:1.12.633-430.vf9a_e567a_244f aws-java-sdk-sqs:1.12.633-430.vf9a_e567a_244f aws-java-sdk-ssm:1.12.633-430.vf9a_e567a_244f aws-secrets-manager-credentials-provider:1.214.va_0a_d8268d068 aws-secrets-manager-secret-source:1.72.v61781b_35c542 aws-sqs-trigger:1.2 bitbucket-oauth:0.13 blueocean:1.27.11 blueocean-bitbucket-pipeline:1.27.11 blueocean-commons:1.27.11 blueocean-config:1.27.11 blueocean-core-js:1.27.11 blueocean-dashboard:1.27.11 blueocean-display-url:2.4.2 blueocean-events:1.27.11 blueocean-git-pipeline:1.27.11 blueocean-github-pipeline:1.27.11 blueocean-i18n:1.27.11 blueocean-jwt:1.27.11 blueocean-personalization:1.27.11 blueocean-pipeline-api-impl:1.27.11 blueocean-pipeline-editor:1.27.11 blueocean-pipeline-scm-api:1.27.11 blueocean-rest:1.27.11 blueocean-rest-impl:1.27.11 blueocean-web:1.27.11 bootstrap5-api:5.3.3-1 bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9 branch-api:2.1152.v6f101e97dd77 caffeine-api:3.1.8-133.v17b_1ff2e0599 checks-api:2.0.2 cloudbees-bitbucket-branch-source:877.vb_b_d5243f6794 cloudbees-folder:6.901.vb_4c7a_da_75da_3 command-launcher:107.v773860566e2e commons-lang3-api:3.13.0-62.v7d18e55f51e2 commons-text-api:1.11.0-95.v22a_d30ee5d36 configuration-as-code:1775.v810dc950b_514 credentials:1337.v60b_d7b_c7b_c9f credentials-binding:657.v2b_19db_7d6e6d display-url-api:2.200.vb_9327d658781 durable-task:550.v0930093c4b_a_6 echarts-api:5.5.0-1 envinject:2.908.v66a_774b_31d93 envinject-api:1.199.v3ce31253ed13 favorite:2.208.v91d65b_7792a_c folder-properties:1.2.1 font-awesome-api:6.5.1-3 git:5.2.1 git-client:4.6.0 github:1.38.0 github-api:1.318-461.v7a_c09c9fa_d63 github-branch-source:1772.va_69eda_d018d4 gson-api:2.10.1-15.v0d99f670e0a_7 handy-uri-templates-2-api:2.1.8-30.v7e777411b_148 htmlpublisher:1.33 instance-identity:185.v303dc7c645f9 ionicons-api:56.v1b_1c8c49374e jackson2-api:2.16.1-373.ve709c6871598 jakarta-activation-api:2.1.3-1 jakarta-mail-api:2.1.3-1 javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.9-1 jdk-tool:73.vddf737284550 jenkins-design-language:1.27.11 jjwt-api:0.11.5-77.v646c772fddb_0 job-dsl:1.87 joda-time-api:2.12.7-29.v5a_b_e3a_82269a_ jquery3-api:3.7.1-2 json-api:20240205-27.va_007549e895c json-path-api:2.8.0-5.v07cb_a_1ca_738c junit:1259.v65ffcef24a_88 mailer:470.vc91f60c5d8e2 matrix-auth:3.2.2 matrix-project:822.824.v14451b_c0fd42 mina-sshd-api-common:2.12.0-90.v9f7fb_9fa_3d3b_ mina-sshd-api-core:2.12.0-90.v9f7fb_9fa_3d3b_ okhttp-api:4.11.0-172.vda_da_1feeb_c6e pipeline-build-step:540.vb_e8849e1a_b_d8 pipeline-cloudwatch-logs:0.2 pipeline-graph-analysis:216.vfd8b_ece330ca_ pipeline-groovy-lib:704.vc58b_8890a_384 pipeline-input-step:491.vb_07d21da_1a_fb_ pipeline-milestone-step:111.v449306f708b_7 pipeline-model-api:2.2175.v76a_fff0a_2618 pipeline-model-definition:2.2175.v76a_fff0a_2618 pipeline-model-extensions:2.2175.v76a_fff0a_2618 pipeline-rest-api:2.34 pipeline-stage-step:305.ve96d0205c1c6 pipeline-stage-tags-metadata:2.2175.v76a_fff0a_2618 pipeline-stage-view:2.34 plain-credentials:179.vc5cb_98f6db_38 plugin-util-api:4.1.0 prism-api:1.29.0-13 pubsub-light:1.18 scm-api:683.vb_16722fb_b_80b_ scm-filter-branch-pr:148.v0b_5f06e8b_c84 scmskip:45.vfdec6fb_05267 script-security:1326.vdb_c154de8669 snakeyaml-api:2.2-111.vc6598e30cc65 sse-gateway:1.26 ssh-credentials:322.v124df57ed808 sshd:3.312.v1c601b_c83b_0e structs:337.v1b_04ea_4df7c8 token-macro:400.v35420b_922dcb_ trilead-api:2.141.v284120fd0c46 variant:60.v7290fc0eb_b_cd workflow-api:1291.v51fd2a_625da_7 workflow-basic-steps:1049.v257a_e6b_30fb_d workflow-cps:3883.vb_3ff2a_e3eea_f workflow-durable-task-step:1331.vc8c2fed35334 workflow-job:1400.v7fd111b_ec82f workflow-multibranch:773.vc4fe1378f1d5 workflow-scm-step:415.v434365564324 workflow-step-api:657.v03b_e8115821b_ workflow-support:865.v43e78cc44e0d
Whether you’re running Jenkins directly or in a container like Tomcat (which one, in which version?)
direct
Whether Jenkins is accessed through a reverse proxy (if so, how is it configured?)
no
How you installed Jenkins, and how you’re launching any involved agents
- The Jenkins controller and slaves are docker images, FROM jenkins/jenkins:lts-jdk11
- agents are launched as AWS ECS Tasks by the amazon-ecs plugin
- the docker containers log to STDOUT and the AWS Log drivers sends the logs to Cloudwatch
Your web browser (+ version). Make sure the issue occurs on supported web browsers.
Chrome (Linux) Version 121.0.6167.139 (Official Build) (64-bit)
- links to
[JENKINS-72924] 'logStreamNames' failed to satisfy constraint: Member must have length less than or equal to 100
Also be advised that this plugin is experimental (developed more as a reference implementation for an API than as an actual feature for end users) and I only occasionally even look at it.
Chandana
added a comment - Thanks markewaite and jglick . Apologies for the bad post. I will update jglick, thanks for that. I have been relying on it for some time now, and other than this it works well, and hope I don't have to remove the plugin
Chandana
added a comment - - edited jglick , thanks for that. I have been relying on it for some time now, and other than this it works well, and hope I don't have to remove the plugin Ah I had to scroll way over to even see the relevant message
'logStreamNames' failed to satisfy constraint: Member must have length less than or equal to 100
I suppose this is coming from https://github.com/jenkinsci/pipeline-cloudwatch-logs-plugin/blob/7a3f481bf3170174414214763ca257ce3fd1734e/src/main/java/io/jenkins/plugins/pipeline_cloudwatch_logs/CloudWatchRetriever.java#L241-L251 which I think would apply if you are running a fairly large number of agents concurrently. Is your pipeline using parallel with a bunch of branches? I would need to refresh my memory of how this filter even works (it has been many years now). Probably the filter could be broken up into chunks or use patterns or something, requiring https://github.com/jenkinsci/pipeline-cloudwatch-logs-plugin/blob/7a3f481bf3170174414214763ca257ce3fd1734e/src/main/java/io/jenkins/plugins/pipeline_cloudwatch_logs/CloudWatchRetriever.java#L241-L251 to be used to interleave result lines.
Chandana
added a comment - Hello jglick,
Thanks for the prompt response.
Yes, I have parallel stages with a number of branches (in a multibranch pipeline)
I wonder if there is a workaround ?.
My controller and slaves are all docker containers. Jenkins logs to STDOUT and I rely on AWS Logs driver to send the output to cloudwatch, and this plugin very nicely bring them back to the Jenkins console. Except for this issue.
What would happen if I removed the plugin ?. How would Jenkins display the logs ?
I would prefer not to remove this plugin, but this error is quite common, which is a problem
Chandana
added a comment - Hello jglick ,
Thanks for the prompt response.
Yes, I have parallel stages with a number of branches (in a multibranch pipeline)
I wonder if there is a workaround ?.
My controller and slaves are all docker containers. Jenkins logs to STDOUT and I rely on AWS Logs driver to send the output to cloudwatch, and this plugin very nicely bring them back to the Jenkins console. Except for this issue.
What would happen if I removed the plugin ?. How would Jenkins display the logs ?
I would prefer not to remove this plugin, but this error is quite common, which is a problem I wonder if there is a workaround?
I doubt it. https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_FilterLogEvents.html#API_FilterLogEvents_RequestSyntax does say it is limited to 100 names. The actual list passed was truncated in the error message, so I presume it was a lot.
I do see that this API accepts a logStreamNamePrefix parameter, which would be simpler to use and not hit the rate limit. I can try to prepare a patch.
What would happen if I removed the plugin? How would Jenkins display the logs?
New builds would work normally. Builds run while the plugin was installed would not have any logs visible in Jenkins.
chandana_desilva_id_au I have a fix which passes automated tests but is otherwise unverified. (I did not attempt to reproduce your error.) Would you mind trying a prerelease build to verify that it fixes your error and does not otherwise seem to harm anything? https://github.com/jenkinsci/pipeline-cloudwatch-logs-plugin/pull/95/checks?check_run_id=23212180341 has everything you need (you might for example copy the *.hpi link and upload at /pluginManager/advanced).
Hello jglick,
Thank You!
I can definitely install it and let it run for a week or two, so see if the error occurs.
I am a bit unclear on how to install it through. My Dockerfile installs plugins with :
jenkins-plugin-cli --plugin-file /usr/share/jenkins/ref/plugins.txt
My Jenkins insall/update is scripted, so I can't use the gui as it can get overwritten. So I need to script the installation of the .hpi
I will read up on how the install a dot hpi on the command line as it needs to go in the Dockerfile.
Also, I have updated Jenkins to v.2.451 and the JDK to jdk21. Hope that will not be an issue
Thanks very much once again
Chandana
added a comment - - edited Hello jglick ,
Thank You!
I can definitely install it and let it run for a week or two, so see if the error occurs.
I am a bit unclear on how to install it through. My Dockerfile installs plugins with :
jenkins-plugin-cli --plugin-file /usr/share/jenkins/ref/plugins.txt
My Jenkins insall/update is scripted, so I can't use the gui as it can get overwritten. So I need to script the installation of the .hpi
I will read up on how the install a dot hpi on the command line as it needs to go in the Dockerfile.
Also, I have updated Jenkins to v.2.451 and the JDK to jdk21. Hope that will not be an issue
Thanks very much once again Chandana
added a comment - Ok, it seems to have installed it cleanly, let's see how it goes over the next two weeks.
Thanks again
Chandana
added a comment - jglick
Ok, it seems to have installed it cleanly, let's see how it goes over the next two weeks.
Thanks again
If you change the plugin definition in your plugins.txt file to replace the cloudwatch line with:
pipeline-cloudwatch-logs:incrementals;io.jenkins.plugins;177.vb_8734c5b_0018
That will load the pull request build without requiring that you use the plugin manager page interactively from the Jenkins controller.
I've uploaded the sample plugins.txt
that I used to confirm in my environment. I believe the only line that will need to change in plugins.txt is the single line that refers to pipeline-cloudwatch-logs.
Yes the line markewaite quoted is the easiest way to install from plugins.txt. I think this is documented somewhere on the plugin installer tool site, but perhaps we should edit the Incrementals check to link to a page with instructions for installing in various modes (in case the check is on a plugin repo).
You can ignore the downgrade message, as that is always shown after upgrading a plugin. Jenkins 2.451 weekly and Java 21 are probably fine.
Chandana
added a comment - markewaite , jglick Thanks. I will certainly update my plugins.txt.
FTI, this is what I did yesterday, seems to work, but obviously your solution is far better:
$ git diff docker/controller/
diff --git a/docker/controller/Dockerfile b/docker/controller/Dockerfile
index ae2cbb5..b73fba8 100644
--- a/docker/controller/Dockerfile
+++ b/docker/controller/Dockerfile
@@ -34,4 +34,9 @@ COPY plugins.txt /usr/share/jenkins/ref/plugins.txt
RUN date -u > /tmp/timestamp && \
jenkins-plugin-cli --plugin-file /usr/share/jenkins/ref/plugins.txt
+# temporary : install the updated pipeline-cloudwatch-logs v 176.v322a_1b_1f0246 directly
+# See: https://issues.jenkins.io/browse/JENKINS-72924
+RUN curl -qs --create-dirs -o ${JENKINS_HOME}/plugins/pipeline-cloudwatch-logs-176.v322a_1b_1f0246.jpi \
+ https://repo.jenkins-ci.org/incrementals/io/jenkins/plugins/pipeline-cloudwatch-logs/176.v322a_1b_1f0246/pipeline-cloudwatch-logs-176.v322a_1b_1f0246.hpi
+
ENTRYPOINT /usr/local/bin/entrypoint.sh
diff --git a/docker/controller/plugins.txt b/docker/controller/plugins.txt
index 3b5bdb5..e3ad29d 100644
--- a/docker/controller/plugins.txt
+++ b/docker/controller/plugins.txt
@@ -1,7 +1,6 @@
amazon-ecs:latest
json-path-api:2.8.0-5.v07cb_a_1ca_738c
blueocean:latest
-pipeline-cloudwatch-logs:latest
pipeline-stage-view:latest
configuration-as-code:latest
cloudbees-bitbucket-branch-source:latest
Chandana
added a comment - markewaite , jglick Thanks. I will certainly update my plugins.txt.
FTI, this is what I did yesterday, seems to work, but obviously your solution is far better:
$ git diff docker/controller/
diff --git a/docker/controller/Dockerfile b/docker/controller/Dockerfile
index ae2cbb5..b73fba8 100644
--- a/docker/controller/Dockerfile
+++ b/docker/controller/Dockerfile
@@ -34,4 +34,9 @@ COPY plugins.txt /usr/share/jenkins/ref/plugins.txt
RUN date -u > /tmp/timestamp && \
jenkins-plugin-cli --plugin-file /usr/share/jenkins/ref/plugins.txt
+# temporary : install the updated pipeline-cloudwatch-logs v 176.v322a_1b_1f0246 directly
+# See: https://issues.jenkins.io/browse/JENKINS-72924
+RUN curl -qs --create-dirs -o ${JENKINS_HOME}/plugins/pipeline-cloudwatch-logs-176.v322a_1b_1f0246.jpi \
+ https://repo.jenkins-ci.org/incrementals/io/jenkins/plugins/pipeline-cloudwatch-logs/176.v322a_1b_1f0246/pipeline-cloudwatch-logs-176.v322a_1b_1f0246.hpi
+
ENTRYPOINT /usr/local/bin/entrypoint.sh
diff --git a/docker/controller/plugins.txt b/docker/controller/plugins.txt
index 3b5bdb5..e3ad29d 100644
--- a/docker/controller/plugins.txt
+++ b/docker/controller/plugins.txt
@@ -1,7 +1,6 @@
amazon-ecs:latest
json-path-api:2.8.0-5.v07cb_a_1ca_738c
blueocean:latest
-pipeline-cloudwatch-logs:latest
pipeline-stage-view:latest
configuration-as-code:latest
cloudbees-bitbucket-branch-source:latest
Chandana
added a comment - markewaite , the version has now moved to 177.vb_8734c5b_0018 from the previous 176.v322a_1b_1f0246. Is that correct ?
Chandana
added a comment - markewaite , the version has now moved to 177.vb_8734c5b_0018 from the previous 176.v322a_1b_1f0246. Is that correct ? chandana_desilva_id_au that is correct. You can find the most recent successful build of the pull request at https://ci.jenkins.io/job/Plugins/job/pipeline-cloudwatch-logs-plugin/job/PR-95/lastSuccessfulBuild/ . The artifacts listed on that page include the version number in their URL.
markewaite thanks for that.
I applied the plugin with
pipeline-cloudwatch-logs:incrementals;io.jenkins.plugins;177.vb_8734c5b_0018
However, my Jenkins is still showing 176.v322a_1b_1f0246.
I would have expected it to show 177.vb_8734c5b_0018
I am sending a copy of the /var/jenkins_home/plugins/pipeline-cloudwatch-logs directory:
Chandana
added a comment - - edited markewaite thanks for that.
I applied the plugin with
pipeline-cloudwatch-logs:incrementals;io.jenkins.plugins;177.vb_8734c5b_0018
However, my Jenkins is still showing 176.v322a_1b_1f0246.
I would have expected it to show 177.vb_8734c5b_0018
I am sending a copy of the /var/jenkins_home/plugins/pipeline-cloudwatch-logs directory:
pipeline-cloudwatch-logs.tgz If your controller is still showing the previous version, then there is likely a step that you missed in your deployment process. I don't know your deployment process, so can only suggest that you confirm that your deployment process did what you expect.
I took the following steps to confirm the plugins.txt syntax:
- Create the mwaite-plugins.txt file that lists the exact plugin versions to use
- Create the run-jenkins.sh shell script that downloads Jenkins 2.440.2 and uses mwaite-plugins.txt to download the plugins
- Run the run-jenkins.sh shell script and open a web browser to the running Jenkins
- Complete the setup wizard by creating a user account and choosing the advanced plugin installation path to install no additional plugins
- Use the Jenkins plugin manager "Installed" page to confirm that the 177.vb_8734c5b_0018 release of the pipeline-cloudwatch-logs was installed
Chandana
added a comment - Hello markewaite , thanks for the prompt reply. Yes, you are right, the problem was with the deployment.
Since you asked, my deployment process is a docker image running on ECS, deployed via CFN. The issue was a an EFS mount for jenkins home which had a directory from the previous plugin install, which did not get overwritten.
Chandana
added a comment - Hello markewaite , thanks for the prompt reply. Yes, you are right, the problem was with the deployment.
Since you asked, my deployment process is a docker image running on ECS, deployed via CFN. The issue was a an EFS mount for jenkins home which had a directory from the previous plugin install, which did not get overwritten. Chandana
added a comment - The plugin has been running for a few days, and I have not seen the issue. It is early days yet.
I am not seeing a lot of builds as I am having problems with my multi branch pipeline, which is using SCM Polling. I have raised JENKINS-72963 for that
Chandana
added a comment - jglick markewaite
The plugin has been running for a few days, and I have not seen the issue. It is early days yet.
I am not seeing a lot of builds as I am having problems with my multi branch pipeline, which is using SCM Polling. I have raised JENKINS-72963 for that JENKINS-72963 is a job-dsl misconfiguration on your part. After fixing that, let me know if this patch helps at scale.
jglick RE https://issues.jenkins.io/browse/JENKINS-72924?focusedId=446506&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-446506
The incrementals check has had instructions for awhile:
https://github.com/jenkinsci/pipeline-cloudwatch-logs-plugin/pull/95/checks?check_run_id=23214937615
Chandana
added a comment - Hello jglick and markewaite
Thanks very much for this. I can confirm that the logs are working properly now.
Chandana
added a comment - Hello jglick and markewaite
Thanks very much for this. I can confirm that the logs are working properly now. 
chandana_desilva_id_au others are much more likely to investigate issue reports when you provide enough details so that they can duplicate the issue on a fresh installation of Jenkins. "How to report an issue" provides a detailed list of the items to include in the report.