Problem

After upgrading from 483.vcb_db_3dcee68f => 515.vc2f8153c5dc6 the following problem occurred most (but not all) of the time:

• The s3Upload step hung for hours doing seemingly nothing, after logging a list of all the files it was going to upload:

  16:06:55  Publish artifacts to S3 Bucket Build is still running
  16:06:55  Publish artifacts to S3 Bucket Using S3 profile: my_metadata_bucket_profile
  16:06:56  Publish artifacts to S3 Bucket bucket=my_metadata_bucket_name/foo/1.2.3, file=first_file.json region=us-east-1, will be uploaded from slave=false managed=false , server encryption false
  ...
  <more files>
  ...
  16:06:57  Publish artifacts to S3 Bucket bucket=my_metadata_bucket_name/foo/1.2.3, file=last_file.json region=us-east-1, will be uploaded from slave=false managed=false , server encryption false
  

• After that final line there were no errors (or any further output at all) and the job would just spin
• Cancelling the job after giving up 18 hours later the following stack trace was logged to the job console, showing where it was stuck:

  10:29:04  java.lang.InterruptedException: sleep interrupted
  10:29:04  	at java.base/java.lang.Thread.sleep0(Native Method)
  10:29:04  	at java.base/java.lang.Thread.sleep(Thread.java:509)
  10:29:04  	at PluginClassLoader for s3//hudson.plugins.s3.S3Profile.repeat(S3Profile.java:287)
  10:29:04  	at PluginClassLoader for s3//hudson.plugins.s3.S3Profile.upload(S3Profile.java:175)
  10:29:04  	at PluginClassLoader for s3//hudson.plugins.s3.S3BucketPublisher.perform(S3BucketPublisher.java:330)
  10:29:04  	at jenkins.tasks.SimpleBuildStep.perform(SimpleBuildStep.java:123)
  10:29:04  	at PluginClassLoader for workflow-basic-steps//org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:101)
  10:29:04  	at PluginClassLoader for workflow-basic-steps//org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:71)
  10:29:04  	at PluginClassLoader for workflow-step-api//org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:49)
  10:29:04  	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
  10:29:04  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
  10:29:04  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
  10:29:04  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  10:29:04  	at java.base/java.lang.Thread.run(Thread.java:1583)
  10:29:04  org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: d7eee5e9-7b2d-428a-a2f8-4f5e4c82ef03
  

More details

The invocation of s3Upload looks like this in groovy:

s3Upload consoleLogLevel: 'INFO',
         dontSetBuildResultOnFailure: true,
         dontWaitForConcurrentBuildCompletion: false,
         entries: [[bucket                 : 'my_metadata_bucket_name/foo/1.2.3',
                    excludedFile           : '',
                    flatten                : false,
                    gzipFiles              : false,
                    keepForever            : false,
                    managedArtifacts       : false,
                    noUploadOnFailure      : false,
                    selectedRegion         : 'us-east-1',
                    showDirectlyInBrowser  : false,
                    sourceFile             : 'build_metadata_from_workspace/**/*',
                    storageClass           : 'STANDARD',
                    uploadFromSlave        : false,
                    useServerSideEncryption: false]],
         pluginFailureResultConstraint: 'FAILURE',
         profileName: 'my_metadata_bucket_profile',  // this refers to a orking S3 credentials profile configured on jenkins under /manage/configure
         userMetadata: []

Workaround

I checked that the configured AWS credentials were working (which they were) and also I observed occasional successes, so it wasn't always getting stuck here.

However it was happening far to often and blocking many developers, so the only workaround I could find for now was to downgrade back to 483.vcb_db_3dcee68f after which "S3 publisher" started working reliably again.