Error sending reports on EL9 with DEFAULT crypto-policy and SHA1 in root CA certificate

[gvde]

This has been mentioned here and here and @evgeni has some additional logs.

If you are running on a EL9 system with crypto-policy DEFAULT

# update-crypto-policies 
Setting system policy to DEFAULT
Note: System-wide crypto policies are applied on application start-up.
It is recommended to restart the system for the change of policies
to fully take place.

and you are using custom certificates which use a CA chain with a root CA certificate signed using SHA1 (e.g. AAA Certificate Services used by Sectigo) the reports module cannot send reports to the foreman server.

The puppetserver log file /var/log/puppetlabs/puppetserver/puppetserver.log contains the error

2025-02-05T07:02:18.624+01:00 ERROR [qtp1986596714-67] [puppetserver] Puppet Report processor failed: Could not send report to Foreman at https://foreman8.example.com/api/config_reports: Certificates do not conform to algorithm constraints
["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1106:in `connect'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:995:in `do_start'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:984:in `start'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1564:in `request'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/reports/foreman.rb:69:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:39:in `block in process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:57:in `block in processors'", "org/jruby/RubyArray.java:1981:in `each'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:54:in `processors'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:32:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:16:in `save'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:324:in `save'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:177:in `do_save'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:54:in `block in call'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:292:in `override'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:53:in `call'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/server/v3.rb:18:in `block in wrap'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:85:in `block in process'", "org/jruby/RubyArray.java:1981:in `each'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:84:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:91:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:91:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:88:in `block in process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:71:in `block in with_request_profiling'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:59:in `profile'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:53:in `profile'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:67:in `with_request_profiling'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:87:in `block in process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:94:in `respond_to_errors'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:86:in `process'", "uri:classloader:/puppetserver-lib/puppet/server/master.rb:69:in `block in handleRequest'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:292:in `override'", "uri:classloader:/puppetserver-lib/puppet/server/master.rb:68:in `handleRequest'"]
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/reports/foreman.rb:75:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:39:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:57:in `block in processors'
org/jruby/RubyArray.java:1981:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:54:in `processors'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:32:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:16:in `save'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:324:in `save'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:177:in `do_save'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:54:in `block in call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:292:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:53:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/server/v3.rb:18:in `block in wrap'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:85:in `block in process'
org/jruby/RubyArray.java:1981:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:84:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:91:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:91:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:88:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:71:in `block in with_request_profiling'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:59:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:53:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:67:in `with_request_profiling'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:87:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:94:in `respond_to_errors'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:86:in `process'
uri:classloader:/puppetserver-lib/puppet/server/master.rb:69:in `block in handleRequest'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:292:in `override'
uri:classloader:/puppetserver-lib/puppet/server/master.rb:68:in `handleRequest'

Setting crypto-policy to DEFAULT:SHA1 or removing the root CA from the chain file mitigates the problem, but as far as I understand, when using custom certificates the ca bundle should include the root CA to allow for certificate pinning.

The same ca bundle is used in other places like httpd without any issue, thus it must be something particular about how ruby handles the ssl ca file and how it evaluates the certs included.

Generally, the signature of self-signed root ca certificates is irrelevant. It doesn't matter whether it's sha1 or sha256 signed. The only relevent information in the certificate is the public key.

Currently running

foreman-3.12.1-1.el9.noarch
katello-4.14.1-1.el9.noarch

on AlmaLinux 9.5, all updates applied.

[ekohl]

I think it's because of the openssl vendored into puppet-agent that's newer and started to reject sha1 in root certs

[gvde]

Are you sure? I thought puppetserver is a java application and it uses (like anything else) the system's openssl library which is why changing the crypto-policy has an effect. Otherwise, it shouldn't.

Generally, the signature in self-signed root ca certificates are irrelevant. There is no purpose checking the signature. And other applications using openssl do it just fine.

[ekohl]

It also uses JRuby and the code is Ruby. IIRC @evgeni or @ehelms found that

[evgeni]

Now that you mention it… Puppetserver is Clojure, so should not have any (J)Ruby in it, so we might have looked wrongly at the situation?

[ekohl]

The report processor itself is written in Ruby but I think it does use HTTP routines from the JVM. It's a complex stack.

[evgeni]

Ah, right, and then it uses the (j)ruby from the agent, as the server depends on that? Complex indeed