Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-40654

Make plugin compatible with storage backends compatible with Amazon S3 (OpenStack Swift...)

    XMLWordPrintable

Details

    • Improvement
    • Status: Open (View Workflow)
    • Minor
    • Resolution: Unresolved
    • s3-plugin
    • None

    Description

      Storage systems that are compatible with Amazon S3 (OpenStack Swift...) have to change the Amazon S3 endpoints configuration.

      To do this with the AWS SDK, you have to provide a file classpath://com/amazonaws/partitions/override/endpoints.json using the JSON format defined in https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/resources/com/amazonaws/partitions/endpoints.json

      Attachments

        Issue Links

          Activity

            cleclerc Cyrille Le Clerc added a comment - See https://github.com/jenkinsci/s3-plugin/pull/100

            Code changed in jenkins
            User: Cyrille Le Clerc
            Path:
            src/main/java/hudson/plugins/s3/ClientHelper.java
            src/main/java/hudson/plugins/s3/Entry.java
            src/main/java/hudson/plugins/s3/S3BucketPublisher.java
            http://jenkins-ci.org/commit/s3-plugin/ff7d34914fbd52a9e44ae7029194d0351c1671c8
            Log:
            JENKINS-40654 Load the list of Amazon S3 regions from c.a.r.RegionMetadataProvider, stop relying on a static lists and constants.

            • Adapt the http proxy logic of the plugin to discover the S3 endpoint hostname with `com.amazonaws.regions.Region#getServiceEndpoint("s3")` instead of the hardcoded constant `s3.amazonaws.com`
            • Replace
            • `com.amazonaws.regions.Regions` by `com.amazonaws.regions.RegionUtils#getRegionsForService("s3")`
            • `com.amazonaws.services.s3.model.Region` by `com.amazonaws.regions.Region`
            • For the default region used in some places of the plugin, introduce the system property `hudson.plugins.s3.DEFAULT_AMAZON_S3_REGION` to override the default `us-east-1`. Note that it would be better to no longer rely on a default AWS Region and to ask the user to specify the desired AWS Region.

            The list of AWS regions can be overridden specifying a file `classpath://com/amazonaws/partitions/override/endpoints.json` matching the format defined in https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/resources/com/amazonaws/partitions/endpoints.json .

            A solution to add this file in the classpath of Jenkins is to use the `java` command line parameter `-Xbootclasspath/a:/path/to/boot/classpath/folder/` and to locate `com/amazonaws/partitions/override/endpoints.json` in `/path/to/boot/classpath/folder/`.

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Cyrille Le Clerc Path: src/main/java/hudson/plugins/s3/ClientHelper.java src/main/java/hudson/plugins/s3/Entry.java src/main/java/hudson/plugins/s3/S3BucketPublisher.java http://jenkins-ci.org/commit/s3-plugin/ff7d34914fbd52a9e44ae7029194d0351c1671c8 Log: JENKINS-40654 Load the list of Amazon S3 regions from c.a.r.RegionMetadataProvider , stop relying on a static lists and constants. Adapt the http proxy logic of the plugin to discover the S3 endpoint hostname with `com.amazonaws.regions.Region#getServiceEndpoint("s3")` instead of the hardcoded constant `s3.amazonaws.com` Replace `com.amazonaws.regions.Regions` by `com.amazonaws.regions.RegionUtils#getRegionsForService("s3")` `com.amazonaws.services.s3.model.Region` by `com.amazonaws.regions.Region` For the default region used in some places of the plugin, introduce the system property `hudson.plugins.s3.DEFAULT_AMAZON_S3_REGION` to override the default `us-east-1`. Note that it would be better to no longer rely on a default AWS Region and to ask the user to specify the desired AWS Region. The list of AWS regions can be overridden specifying a file `classpath://com/amazonaws/partitions/override/endpoints.json` matching the format defined in https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/resources/com/amazonaws/partitions/endpoints.json . A solution to add this file in the classpath of Jenkins is to use the `java` command line parameter `-Xbootclasspath/a:/path/to/boot/classpath/folder/` and to locate `com/amazonaws/partitions/override/endpoints.json` in `/path/to/boot/classpath/folder/`.

            Code changed in jenkins
            User: Cyrille Le Clerc
            Path:
            README.md
            src/main/java/hudson/plugins/s3/ClientHelper.java
            src/main/java/hudson/plugins/s3/Entry.java
            http://jenkins-ci.org/commit/s3-plugin/2d1b795c62f8bc37ac02c69e4fb24bfc5b1092e6
            Log:
            JENKINS-40654 @Jimilian’s recommendations

            • Document `classpath://com/amazonaws/partitions/override/endpoints.json` and `hudson.plugins.s3.DEFAULT_AMAZON_S3_REGION` in README.md
            • Use `@nonnull` and `@nullable` in helper methods (not in the whole ClientHelper class to not change too many things at once)
            • Cleanup imports
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Cyrille Le Clerc Path: README.md src/main/java/hudson/plugins/s3/ClientHelper.java src/main/java/hudson/plugins/s3/Entry.java http://jenkins-ci.org/commit/s3-plugin/2d1b795c62f8bc37ac02c69e4fb24bfc5b1092e6 Log: JENKINS-40654 @Jimilian’s recommendations Document `classpath://com/amazonaws/partitions/override/endpoints.json` and `hudson.plugins.s3.DEFAULT_AMAZON_S3_REGION` in README.md Use `@nonnull` and `@nullable` in helper methods (not in the whole ClientHelper class to not change too many things at once) Cleanup imports

            Code changed in jenkins
            User: Akbashev Alexander
            Path:
            README.md
            src/main/java/hudson/plugins/s3/ClientHelper.java
            src/main/java/hudson/plugins/s3/Entry.java
            src/main/java/hudson/plugins/s3/S3BucketPublisher.java
            http://jenkins-ci.org/commit/s3-plugin/945f3315a246888482561d12406fa39b84f8a3f2
            Log:
            Merge pull request #100 from cyrille-leclerc/master

            JENKINS-40654 Make plugin compatible with storage backends compatible with Amazon S3 (OpenStack Swift...)

            Compare: https://github.com/jenkinsci/s3-plugin/compare/92247d2d099b...945f3315a246

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Akbashev Alexander Path: README.md src/main/java/hudson/plugins/s3/ClientHelper.java src/main/java/hudson/plugins/s3/Entry.java src/main/java/hudson/plugins/s3/S3BucketPublisher.java http://jenkins-ci.org/commit/s3-plugin/945f3315a246888482561d12406fa39b84f8a3f2 Log: Merge pull request #100 from cyrille-leclerc/master JENKINS-40654 Make plugin compatible with storage backends compatible with Amazon S3 (OpenStack Swift...) Compare: https://github.com/jenkinsci/s3-plugin/compare/92247d2d099b...945f3315a246

            It would be much easier if the S3 endpoint could be specified in the configuration page (just use setEndpoint()), rather than the current endpoints.json classpath solution, which requires changes to jenkins startup.

            jamshid Jamshid Afshar added a comment - It would be much easier if the S3 endpoint could be specified in the configuration page (just use setEndpoint()), rather than the current endpoints.json classpath solution, which requires changes to jenkins startup.
            simonrichter Simon Richter added a comment -

            We use both Amazon S3 and CERN's OpenStack installation from the same Jenkins instance, so we cannot just override the endpoints globally.

            It would be nice if there was a per-transfer configuration setting, selecting one of several configured providers (which all have their own sets of credentials).

            simonrichter Simon Richter added a comment - We use both Amazon S3 and CERN's OpenStack installation from the same Jenkins instance, so we cannot just override the endpoints globally. It would be nice if there was a per-transfer configuration setting, selecting one of several configured providers (which all have their own sets of credentials).
            cleclerc Cyrille Le Clerc added a comment - - edited

            simonrichter can't you overwrite the endpoints.json file to add your S3 endpoint? You would have to introduce and "s3.endpoint" I think; you may also have to define an "s3-control.endpoint".

            https://github.com/aws/aws-sdk-java/blob/1.11.587/aws-java-sdk-core/src/main/resources/com/amazonaws/partitions/endpoints.json#L2367

            cleclerc Cyrille Le Clerc added a comment - - edited simonrichter can't you overwrite the endpoints.json file to add your S3 endpoint? You would have to introduce and "s3.endpoint" I think; you may also have to define an "s3-control.endpoint". https://github.com/aws/aws-sdk-java/blob/1.11.587/aws-java-sdk-core/src/main/resources/com/amazonaws/partitions/endpoints.json#L2367

            People

              jimilian Alexander A
              cleclerc Cyrille Le Clerc
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated: