Uploaded image for project: 'Infrastructure'
  1. Infrastructure
  2. INFRA-1223

Upgrade puppet enterprise to 2017.2.1

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Resolved (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: puppet
    • Labels:
      None
    • Similar Issues:

      Description

      Puppet agent runs are failing because of a weird bug which has been fixed in LTS versions of Puppet Enterprise, which we're apparently not on.

      We're currently at 2016.5.1.

      18:21 <@binford2k> that problem is resolved in PE2016.4.4 (LTS) and PE2017.2.0+, or in puppet-agent 1.10.0 (puppet 4.10)+
      18:22 < rtyler> HOW DOES THIS VERSIONING SCHEME WORK
      18:22 < rtyler> how is 2016.5.1 not newer than 2016.5.1?
      18:22 < _rc> numbers get bigger over time
      18:22 < rtyler> er, 2016.4.4
      18:22 <@binford2k> haha, 2016.4 is the LTS, it will get backported fixes
      18:22 < rtyler> binford2k: was there a newsletter update that I missed when this notion of "LTS" was introduced
      18:23 < rtyler> I thought everything PE was ... kind of like an LTS
      18:23 <@binford2k> so 2016.4 can have a limited number of fixes that don't exist in 2016.5
      

        Attachments

          Activity

          Hide
          rtyler R. Tyler Croy added a comment -

          Upgrade was pretty painless, running Puppet on lettuce results in this error now:

          2017-06-13 19:11:46,527 INFO  [qtp918037316-135] [puppetserver] Puppet Computing checksum on string
          2017-06-13 19:11:46,529 ERROR [qtp918037316-135] [puppetserver] Puppet Unable to verify existing FileBucket backup at '/opt/puppetlabs/server/data/puppetserver/bucket/1/3/7/3/1/d/2/7/13731d27f1276cdd3314f7fa30aa5eb1/contents'.
          2017-06-13 19:11:46,530 ERROR [qtp918037316-135] [puppetserver] Puppet Server Error: Existing backup and new file have different content but same checksum, {md5}13731d27f1276cdd3314f7fa30aa5eb1. Verify existing backup and remove if incorrect.
          /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/file_bucket_file/file.rb:170:in `save_to_disk'
          /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/file_system/file_impl.rb:57:in `exclusive_open'
          org/jruby/RubyIO.java:1201:in `open'
          /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/file_system/file_impl.rb:55:in `exclusive_open'
          /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/file_system.rb:123:in `exclusive_open'
          /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/file_bucket_file/file.rb:157:in `save_to_disk'
          /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:143:in `withumask'
          /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/file_bucket_file/file.rb:148:in `save_to_disk'
          /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/file_bucket_file/file.rb:102:in `save'
          /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:285:in `save'
          /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:176:in `do_save'
          /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:48:in `call'
          /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
          /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:294:in `override'
          /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:47:in `call'
          /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:82:in `process'
          org/jruby/RubyArray.java:1613:in `each'
          /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:81:in `process'
          /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:87:in `process'
          /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:87:in `process'
          /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:60:in `process'
          /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
          /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
          /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:58:in `process'
          file:/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/puppetserver-lib/puppet/server/master.rb:42:in `handleRequest'
          Puppet$$Server$$Master_679071754.gen:13:in `handleRequest'
          request_handler_core.clj:273:in `invoke'
          jruby_request.clj:46:in `invoke'
          jruby_request.clj:31:in `invoke'
          request_handler_service.clj:34:in `handle_request'
          request_handler.clj:3:in `invoke'
          request_handler.clj:3:in `invoke'
          core.clj:2515:in `invoke'
          ring_middleware.clj:284:in `invoke'
          core.clj:168:in `invoke'
          core.clj:211:in `invoke'
          core.clj:45:in `invoke'
          core.clj:357:in `invoke'
          core.clj:51:in `invoke'
          ringutils.clj:83:in `invoke'
          master_core.clj:421:in `invoke'
          ring.clj:21:in `invoke'
          ring.clj:12:in `invoke'
          comidi.clj:249:in `invoke'
          http.clj:152:in `invoke'
          http.clj:152:in `invoke'
          http.clj:148:in `invoke'
          comidi.clj:336:in `invoke'
          jetty9_core.clj:423:in `invoke'
          normalized_uri_helpers.clj:81:in `invoke'
          
          
          Show
          rtyler R. Tyler Croy added a comment - Upgrade was pretty painless, running Puppet on lettuce results in this error now: 2017-06-13 19:11:46,527 INFO [qtp918037316-135] [puppetserver] Puppet Computing checksum on string 2017-06-13 19:11:46,529 ERROR [qtp918037316-135] [puppetserver] Puppet Unable to verify existing FileBucket backup at '/opt/puppetlabs/server/data/puppetserver/bucket/1/3/7/3/1/d/2/7/13731d27f1276cdd3314f7fa30aa5eb1/contents' . 2017-06-13 19:11:46,530 ERROR [qtp918037316-135] [puppetserver] Puppet Server Error: Existing backup and new file have different content but same checksum, {md5}13731d27f1276cdd3314f7fa30aa5eb1. Verify existing backup and remove if incorrect. /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/file_bucket_file/file.rb:170:in `save_to_disk' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/file_system/file_impl.rb:57:in `exclusive_open' org/jruby/RubyIO.java:1201:in `open' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/file_system/file_impl.rb:55:in `exclusive_open' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/file_system.rb:123:in `exclusive_open' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/file_bucket_file/file.rb:157:in `save_to_disk' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:143:in `withumask' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/file_bucket_file/file.rb:148:in `save_to_disk' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/file_bucket_file/file.rb:102:in `save' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:285:in `save' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:176:in `do_save' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:48:in `call' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:294:in `override' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:47:in `call' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:82:in `process' org/jruby/RubyArray.java:1613:in `each' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:81:in `process' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:87:in `process' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:87:in `process' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:60:in `process' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:58:in `process' file:/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/puppetserver-lib/puppet/server/master.rb:42:in `handleRequest' Puppet$$Server$$Master_679071754.gen:13:in `handleRequest' request_handler_core.clj:273:in `invoke' jruby_request.clj:46:in `invoke' jruby_request.clj:31:in `invoke' request_handler_service.clj:34:in `handle_request' request_handler.clj:3:in `invoke' request_handler.clj:3:in `invoke' core.clj:2515:in `invoke' ring_middleware.clj:284:in `invoke' core.clj:168:in `invoke' core.clj:211:in `invoke' core.clj:45:in `invoke' core.clj:357:in `invoke' core.clj:51:in `invoke' ringutils.clj:83:in `invoke' master_core.clj:421:in `invoke' ring.clj:21:in `invoke' ring.clj:12:in `invoke' comidi.clj:249:in `invoke' http.clj:152:in `invoke' http.clj:152:in `invoke' http.clj:148:in `invoke' comidi.clj:336:in `invoke' jetty9_core.clj:423:in `invoke' normalized_uri_helpers.clj:81:in `invoke'
          Hide
          rtyler R. Tyler Croy added a comment -

          I removed the file bucketed file on the puppet master (referenced in the above error) and it looks like puppet runs are completing properly.

          Did we srsly have an md5 collision :-!

          Show
          rtyler R. Tyler Croy added a comment - I removed the file bucketed file on the puppet master (referenced in the above error) and it looks like puppet runs are completing properly. Did we srsly have an md5 collision :-!

            People

            Assignee:
            rtyler R. Tyler Croy
            Reporter:
            rtyler R. Tyler Croy
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: