Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-40654

Make plugin compatible with storage backends compatible with Amazon S3 (OpenStack Swift...)

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Minor Minor
    • s3-plugin
    • None

      Storage systems that are compatible with Amazon S3 (OpenStack Swift...) have to change the Amazon S3 endpoints configuration.

      To do this with the AWS SDK, you have to provide a file classpath://com/amazonaws/partitions/override/endpoints.json using the JSON format defined in https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/resources/com/amazonaws/partitions/endpoints.json

          [JENKINS-40654] Make plugin compatible with storage backends compatible with Amazon S3 (OpenStack Swift...)

          Cyrille Le Clerc added a comment - See https://github.com/jenkinsci/s3-plugin/pull/100

          Code changed in jenkins
          User: Cyrille Le Clerc
          Path:
          src/main/java/hudson/plugins/s3/ClientHelper.java
          src/main/java/hudson/plugins/s3/Entry.java
          src/main/java/hudson/plugins/s3/S3BucketPublisher.java
          http://jenkins-ci.org/commit/s3-plugin/ff7d34914fbd52a9e44ae7029194d0351c1671c8
          Log:
          JENKINS-40654 Load the list of Amazon S3 regions from c.a.r.RegionMetadataProvider, stop relying on a static lists and constants.

          • Adapt the http proxy logic of the plugin to discover the S3 endpoint hostname with `com.amazonaws.regions.Region#getServiceEndpoint("s3")` instead of the hardcoded constant `s3.amazonaws.com`
          • Replace
          • `com.amazonaws.regions.Regions` by `com.amazonaws.regions.RegionUtils#getRegionsForService("s3")`
          • `com.amazonaws.services.s3.model.Region` by `com.amazonaws.regions.Region`
          • For the default region used in some places of the plugin, introduce the system property `hudson.plugins.s3.DEFAULT_AMAZON_S3_REGION` to override the default `us-east-1`. Note that it would be better to no longer rely on a default AWS Region and to ask the user to specify the desired AWS Region.

          The list of AWS regions can be overridden specifying a file `classpath://com/amazonaws/partitions/override/endpoints.json` matching the format defined in https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/resources/com/amazonaws/partitions/endpoints.json .

          A solution to add this file in the classpath of Jenkins is to use the `java` command line parameter `-Xbootclasspath/a:/path/to/boot/classpath/folder/` and to locate `com/amazonaws/partitions/override/endpoints.json` in `/path/to/boot/classpath/folder/`.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Cyrille Le Clerc Path: src/main/java/hudson/plugins/s3/ClientHelper.java src/main/java/hudson/plugins/s3/Entry.java src/main/java/hudson/plugins/s3/S3BucketPublisher.java http://jenkins-ci.org/commit/s3-plugin/ff7d34914fbd52a9e44ae7029194d0351c1671c8 Log: JENKINS-40654 Load the list of Amazon S3 regions from c.a.r.RegionMetadataProvider , stop relying on a static lists and constants. Adapt the http proxy logic of the plugin to discover the S3 endpoint hostname with `com.amazonaws.regions.Region#getServiceEndpoint("s3")` instead of the hardcoded constant `s3.amazonaws.com` Replace `com.amazonaws.regions.Regions` by `com.amazonaws.regions.RegionUtils#getRegionsForService("s3")` `com.amazonaws.services.s3.model.Region` by `com.amazonaws.regions.Region` For the default region used in some places of the plugin, introduce the system property `hudson.plugins.s3.DEFAULT_AMAZON_S3_REGION` to override the default `us-east-1`. Note that it would be better to no longer rely on a default AWS Region and to ask the user to specify the desired AWS Region. The list of AWS regions can be overridden specifying a file `classpath://com/amazonaws/partitions/override/endpoints.json` matching the format defined in https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/resources/com/amazonaws/partitions/endpoints.json . A solution to add this file in the classpath of Jenkins is to use the `java` command line parameter `-Xbootclasspath/a:/path/to/boot/classpath/folder/` and to locate `com/amazonaws/partitions/override/endpoints.json` in `/path/to/boot/classpath/folder/`.

          Code changed in jenkins
          User: Cyrille Le Clerc
          Path:
          README.md
          src/main/java/hudson/plugins/s3/ClientHelper.java
          src/main/java/hudson/plugins/s3/Entry.java
          http://jenkins-ci.org/commit/s3-plugin/2d1b795c62f8bc37ac02c69e4fb24bfc5b1092e6
          Log:
          JENKINS-40654 @Jimilian’s recommendations

          • Document `classpath://com/amazonaws/partitions/override/endpoints.json` and `hudson.plugins.s3.DEFAULT_AMAZON_S3_REGION` in README.md
          • Use `@nonnull` and `@nullable` in helper methods (not in the whole ClientHelper class to not change too many things at once)
          • Cleanup imports

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Cyrille Le Clerc Path: README.md src/main/java/hudson/plugins/s3/ClientHelper.java src/main/java/hudson/plugins/s3/Entry.java http://jenkins-ci.org/commit/s3-plugin/2d1b795c62f8bc37ac02c69e4fb24bfc5b1092e6 Log: JENKINS-40654 @Jimilian’s recommendations Document `classpath://com/amazonaws/partitions/override/endpoints.json` and `hudson.plugins.s3.DEFAULT_AMAZON_S3_REGION` in README.md Use `@nonnull` and `@nullable` in helper methods (not in the whole ClientHelper class to not change too many things at once) Cleanup imports

          Code changed in jenkins
          User: Akbashev Alexander
          Path:
          README.md
          src/main/java/hudson/plugins/s3/ClientHelper.java
          src/main/java/hudson/plugins/s3/Entry.java
          src/main/java/hudson/plugins/s3/S3BucketPublisher.java
          http://jenkins-ci.org/commit/s3-plugin/945f3315a246888482561d12406fa39b84f8a3f2
          Log:
          Merge pull request #100 from cyrille-leclerc/master

          JENKINS-40654 Make plugin compatible with storage backends compatible with Amazon S3 (OpenStack Swift...)

          Compare: https://github.com/jenkinsci/s3-plugin/compare/92247d2d099b...945f3315a246

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Akbashev Alexander Path: README.md src/main/java/hudson/plugins/s3/ClientHelper.java src/main/java/hudson/plugins/s3/Entry.java src/main/java/hudson/plugins/s3/S3BucketPublisher.java http://jenkins-ci.org/commit/s3-plugin/945f3315a246888482561d12406fa39b84f8a3f2 Log: Merge pull request #100 from cyrille-leclerc/master JENKINS-40654 Make plugin compatible with storage backends compatible with Amazon S3 (OpenStack Swift...) Compare: https://github.com/jenkinsci/s3-plugin/compare/92247d2d099b...945f3315a246

          It would be much easier if the S3 endpoint could be specified in the configuration page (just use setEndpoint()), rather than the current endpoints.json classpath solution, which requires changes to jenkins startup.

          Jamshid Afshar added a comment - It would be much easier if the S3 endpoint could be specified in the configuration page (just use setEndpoint()), rather than the current endpoints.json classpath solution, which requires changes to jenkins startup.

          Simon Richter added a comment -

          We use both Amazon S3 and CERN's OpenStack installation from the same Jenkins instance, so we cannot just override the endpoints globally.

          It would be nice if there was a per-transfer configuration setting, selecting one of several configured providers (which all have their own sets of credentials).

          Simon Richter added a comment - We use both Amazon S3 and CERN's OpenStack installation from the same Jenkins instance, so we cannot just override the endpoints globally. It would be nice if there was a per-transfer configuration setting, selecting one of several configured providers (which all have their own sets of credentials).

          Cyrille Le Clerc added a comment - - edited

          simonrichter can't you overwrite the endpoints.json file to add your S3 endpoint? You would have to introduce and "s3.endpoint" I think; you may also have to define an "s3-control.endpoint".

          https://github.com/aws/aws-sdk-java/blob/1.11.587/aws-java-sdk-core/src/main/resources/com/amazonaws/partitions/endpoints.json#L2367

          Cyrille Le Clerc added a comment - - edited simonrichter can't you overwrite the endpoints.json file to add your S3 endpoint? You would have to introduce and "s3.endpoint" I think; you may also have to define an "s3-control.endpoint". https://github.com/aws/aws-sdk-java/blob/1.11.587/aws-java-sdk-core/src/main/resources/com/amazonaws/partitions/endpoints.json#L2367

            jimilian Alexander A
            cleclerc Cyrille Le Clerc
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: