Error 1049 (42000): Unknown database 'boulder_sa'

[vsc55]

Hi,

First of all I wanted to thank you for this great project.

The problem has arisen when updating from version 25.12 to 26.03.
Now when trying to access sections other than the LOG, the following error appears on the website: Some unexpected error occurred!
This is logged in the console:

gui-1             | 2026/03/11 20:41:11 GET /
gui-1             | 2026/03/11 20:41:11 errorHandler: err=Error 1049 (42000): Unknown database 'boulder_sa'
gui-1             | main.CollectDashboardData({0x16b87f8, 0xc00015c780}, 0xc000d58500)
gui-1             |     /go/src/labca/dashboard.go:605 +0x9aa
gui-1             | main.rootHandler({0x16b87f8, 0xc00015c780}, 0xc000d58500)
gui-1             |     /go/src/labca/main.go:392 +0x6c
gui-1             | net/http.HandlerFunc.ServeHTTP(0xfbf740?, {0x16b87f8?, 0xc00015c780?}, 0xc00015c780?)
gui-1             |     /usr/local/go/src/net/http/server.go:2322 +0x29
gui-1             | main.authorized.func1({0x16b87f8, 0xc00015c780}, 0xc000d58500)
gui-1             |     /go/src/labca/main.go:3154 +0x32e
gui-1             | net/http.HandlerFunc.ServeHTTP(0xc000d583c0?, {0x16b87f8?, 0xc00015c780?}, 0x3052b87b7d?)
gui-1             |     /usr/local/go/src/net/http/server.go:2322 +0x29
gui-1             | github.com/gorilla/mux.(*Router).ServeHTTP(0xc000d54000, {0x16b87f8, 0xc00015c780}, 0xc000d58280)
gui-1             |     /root/go/pkg/mod/github.com/gorilla/mux@v1.8.1/mux.go:212 +0x1e2
gui-1             | net/http.serverHandler.ServeHTTP({0xc000bf2680?}, {0x16b87f8?, 0xc00015c780?}, 0x6?)
gui-1             |     /usr/local/go/src/net/http/server.go:3340 +0x8e
gui-1             | net/http.(*conn).serve(0xc0000e7cb0, {0x16bad80, 0xc000c002d0})
gui-1             |     /usr/local/go/src/net/http/server.go:2109 +0x665
gui-1             | created by net/http.(*Server).Serve in goroutine 1
gui-1             |     /usr/local/go/src/net/http/server.go:3493 +0x485
nginx-1           | ::ffff:192.xxxxx - - [11/Mar/2026:20:41:11 +0000] "GET /admin/ HTTP/1.1" 500 9077 "https://labca.dominio.local/admin/about" "Mozill

The mysql update process seems to be fine, everything is ok:

bmysql-upgrade-1  | Phase 1/8: Checking and upgrading mysql database
bmysql-upgrade-1  | Processing databases
bmysql-upgrade-1  | mysql
bmysql-upgrade-1  | mysql.column_stats                                 OK
bmysql-upgrade-1  | mysql.columns_priv                                 OK
bmysql-upgrade-1  | mysql.db                                           OK
bmysql-upgrade-1  | mysql.event                                        OK
bmysql-upgrade-1  | mysql.func                                         OK
bmysql-upgrade-1  | mysql.global_priv                                  OK
bmysql-upgrade-1  | mysql.gtid_slave_pos                               OK
bmysql-upgrade-1  | mysql.help_category                                OK
bmysql-upgrade-1  | mysql.help_keyword                                 OK
bmysql-upgrade-1  | mysql.help_relation                                OK
bmysql-upgrade-1  | mysql.help_topic                                   OK
bmysql-upgrade-1  | mysql.index_stats                                  OK
bmysql-upgrade-1  | mysql.innodb_index_stats                           OK
bmysql-upgrade-1  | mysql.innodb_table_stats                           OK
bmysql-upgrade-1  | mysql.plugin                                       OK
bmysql-upgrade-1  | mysql.proc                                         OK
bmysql-upgrade-1  | mysql.procs_priv                                   OK
bmysql-upgrade-1  | mysql.proxies_priv                                 OK
bmysql-upgrade-1  | mysql.roles_mapping                                OK
bmysql-upgrade-1  | mysql.servers                                      OK
bmysql-upgrade-1  | mysql.table_stats                                  OK
bmysql-upgrade-1  | mysql.tables_priv                                  OK
bmysql-upgrade-1  | mysql.time_zone                                    OK
bmysql-upgrade-1  | mysql.time_zone_leap_second                        OK
bmysql-upgrade-1  | mysql.time_zone_name                               OK
bmysql-upgrade-1  | mysql.time_zone_transition                         OK
bmysql-upgrade-1  | mysql.time_zone_transition_type                    OK
bmysql-upgrade-1  | mysql.transaction_registry                         OK
bmysql-upgrade-1  | Phase 2/8: Installing used storage engines... Skipped
bmysql-upgrade-1  | Phase 3/8: Running 'mysql_fix_privilege_tables'
boulder-1         | labca/entrypoint.sh: line 26: USE_VITESS: unbound variable
boulder-1 exited with code 1 (restarting)
boulder-1         | labca/entrypoint.sh: line 26: USE_VITESS: unbound variable
boulder-1 exited with code 1 (restarting)
boulder-1         | labca/entrypoint.sh: line 26: USE_VITESS: unbound variable
boulder-1 exited with code 1 (restarting)
boulder-1 exited with code 1 (restarting)
boulder-1         | labca/entrypoint.sh: line 26: USE_VITESS: unbound variable
bmysql-upgrade-1  | Phase 4/8: Fixing views
bmysql-upgrade-1  | mysql.user                                         OK
bmysql-upgrade-1  | sys.host_summary                                   OK
bmysql-upgrade-1  | sys.host_summary_by_file_io                        OK
bmysql-upgrade-1  | sys.host_summary_by_file_io_type                   OK
bmysql-upgrade-1  | sys.host_summary_by_stages                         OK
bmysql-upgrade-1  | sys.host_summary_by_statement_latency              OK
bmysql-upgrade-1  | sys.host_summary_by_statement_type                 OK
bmysql-upgrade-1  | sys.innodb_buffer_stats_by_schema                  OK
bmysql-upgrade-1  | sys.innodb_buffer_stats_by_table                   OK
bmysql-upgrade-1  | sys.innodb_lock_waits                              OK
bmysql-upgrade-1  | sys.io_by_thread_by_latency                        OK
bmysql-upgrade-1  | sys.io_global_by_file_by_bytes                     OK
bmysql-upgrade-1  | sys.io_global_by_file_by_latency                   OK
bmysql-upgrade-1  | sys.io_global_by_wait_by_bytes                     OK
bmysql-upgrade-1  | sys.io_global_by_wait_by_latency                   OK
bmysql-upgrade-1  | sys.latest_file_io                                 OK
bmysql-upgrade-1  | sys.memory_by_host_by_current_bytes                OK
bmysql-upgrade-1  | sys.memory_by_thread_by_current_bytes              OK
bmysql-upgrade-1  | sys.memory_by_user_by_current_bytes                OK
bmysql-upgrade-1  | sys.memory_global_by_current_bytes                 OK
bmysql-upgrade-1  | sys.memory_global_total                            OK
bmysql-upgrade-1  | sys.metrics                                        OK
bmysql-upgrade-1  | sys.processlist                                    OK
bmysql-upgrade-1  | sys.ps_check_lost_instrumentation                  OK
bmysql-upgrade-1  | sys.schema_auto_increment_columns                  OK
bmysql-upgrade-1  | sys.schema_index_statistics                        OK
bmysql-upgrade-1  | sys.schema_object_overview                         OK
bmysql-upgrade-1  | sys.schema_redundant_indexes                       OK
bmysql-upgrade-1  | sys.schema_table_lock_waits                        OK
bmysql-upgrade-1  | sys.schema_table_statistics                        OK
bmysql-upgrade-1  | sys.schema_table_statistics_with_buffer            OK
bmysql-upgrade-1  | sys.schema_tables_with_full_table_scans            OK
bmysql-upgrade-1  | sys.schema_unused_indexes                          OK
bmysql-upgrade-1  | sys.session                                        OK
bmysql-upgrade-1  | sys.session_ssl_status                             OK
bmysql-upgrade-1  | sys.statement_analysis                             OK
bmysql-upgrade-1  | sys.statements_with_errors_or_warnings             OK
bmysql-upgrade-1  | sys.statements_with_full_table_scans               OK
bmysql-upgrade-1  | sys.statements_with_runtimes_in_95th_percentile    OK
bmysql-upgrade-1  | sys.statements_with_sorting                        OK
bmysql-upgrade-1  | sys.statements_with_temp_tables                    OK
bmysql-upgrade-1  | sys.user_summary                                   OK
bmysql-upgrade-1  | sys.user_summary_by_file_io                        OK
bmysql-upgrade-1  | sys.user_summary_by_file_io_type                   OK
bmysql-upgrade-1  | sys.user_summary_by_stages                         OK
bmysql-upgrade-1  | sys.user_summary_by_statement_latency              OK
bmysql-upgrade-1  | sys.user_summary_by_statement_type                 OK
bmysql-upgrade-1  | sys.version                                        OK
bmysql-upgrade-1  | sys.wait_classes_global_by_avg_latency             OK
bmysql-upgrade-1  | sys.wait_classes_global_by_latency                 OK
bmysql-upgrade-1  | sys.waits_by_host_by_latency                       OK
bmysql-upgrade-1  | sys.waits_by_user_by_latency                       OK
bmysql-upgrade-1  | sys.waits_global_by_latency                        OK
bmysql-upgrade-1  | sys.x$host_summary                                 OK
bmysql-upgrade-1  | sys.x$host_summary_by_file_io                      OK
bmysql-upgrade-1  | sys.x$host_summary_by_file_io_type                 OK
bmysql-upgrade-1  | sys.x$host_summary_by_stages                       OK
bmysql-upgrade-1  | sys.x$host_summary_by_statement_latency            OK
bmysql-upgrade-1  | sys.x$host_summary_by_statement_type               OK
bmysql-upgrade-1  | sys.x$innodb_buffer_stats_by_schema                OK
bmysql-upgrade-1  | sys.x$innodb_buffer_stats_by_table                 OK
bmysql-upgrade-1  | sys.x$innodb_lock_waits                            OK
bmysql-upgrade-1  | sys.x$io_by_thread_by_latency                      OK
bmysql-upgrade-1  | sys.x$io_global_by_file_by_bytes                   OK
bmysql-upgrade-1  | sys.x$io_global_by_file_by_latency                 OK
bmysql-upgrade-1  | sys.x$io_global_by_wait_by_bytes                   OK
bmysql-upgrade-1  | sys.x$io_global_by_wait_by_latency                 OK
bmysql-upgrade-1  | sys.x$latest_file_io                               OK
bmysql-upgrade-1  | sys.x$memory_by_host_by_current_bytes              OK
bmysql-upgrade-1  | sys.x$memory_by_thread_by_current_bytes            OK
bmysql-upgrade-1  | sys.x$memory_by_user_by_current_bytes              OK
bmysql-upgrade-1  | sys.x$memory_global_by_current_bytes               OK
bmysql-upgrade-1  | sys.x$memory_global_total                          OK
bmysql-upgrade-1  | sys.x$processlist                                  OK
bmysql-upgrade-1  | sys.x$ps_digest_95th_percentile_by_avg_us          OK
bmysql-upgrade-1  | sys.x$ps_digest_avg_latency_distribution           OK
bmysql-upgrade-1  | sys.x$ps_schema_table_statistics_io                OK
bmysql-upgrade-1  | sys.x$schema_flattened_keys                        OK
bmysql-upgrade-1  | sys.x$schema_index_statistics                      OK
bmysql-upgrade-1  | sys.x$schema_table_lock_waits                      OK
bmysql-upgrade-1  | sys.x$schema_table_statistics                      OK
bmysql-upgrade-1  | sys.x$schema_table_statistics_with_buffer          OK
bmysql-upgrade-1  | sys.x$schema_tables_with_full_table_scans          OK
bmysql-upgrade-1  | sys.x$session                                      OK
bmysql-upgrade-1  | sys.x$statement_analysis                           OK
bmysql-upgrade-1  | sys.x$statements_with_errors_or_warnings           OK
bmysql-upgrade-1  | sys.x$statements_with_full_table_scans             OK
bmysql-upgrade-1  | sys.x$statements_with_runtimes_in_95th_percentile  OK
bmysql-upgrade-1  | sys.x$statements_with_sorting                      OK
bmysql-upgrade-1  | sys.x$statements_with_temp_tables                  OK
bmysql-upgrade-1  | sys.x$user_summary                                 OK
bmysql-upgrade-1  | sys.x$user_summary_by_file_io                      OK
bmysql-upgrade-1  | sys.x$user_summary_by_file_io_type                 OK
bmysql-upgrade-1  | sys.x$user_summary_by_stages                       OK
bmysql-upgrade-1  | sys.x$user_summary_by_statement_latency            OK
bmysql-upgrade-1  | sys.x$user_summary_by_statement_type               OK
bmysql-upgrade-1  | sys.x$wait_classes_global_by_avg_latency           OK
bmysql-upgrade-1  | sys.x$wait_classes_global_by_latency               OK
bmysql-upgrade-1  | sys.x$waits_by_host_by_latency                     OK
bmysql-upgrade-1  | sys.x$waits_by_user_by_latency                     OK
bmysql-upgrade-1  | sys.x$waits_global_by_latency                      OK
bmysql-upgrade-1  | Phase 5/8: Fixing table and database names
bmysql-upgrade-1  | Phase 6/8: Checking and upgrading tables
bmysql-upgrade-1  | Processing databases
bmysql-upgrade-1  | boulder_sa_integration
bmysql-upgrade-1  | boulder_sa_integration.authz2                      OK
bmysql-upgrade-1  | boulder_sa_integration.blockedKeys                 OK
bmysql-upgrade-1  | boulder_sa_integration.certificateStatus           OK
bmysql-upgrade-1  | boulder_sa_integration.certificates                OK
bmysql-upgrade-1  | boulder_sa_integration.certificatesPerName         OK
bmysql-upgrade-1  | boulder_sa_integration.challenges                  OK
bmysql-upgrade-1  | boulder_sa_integration.crlShards                   OK
bmysql-upgrade-1  | boulder_sa_integration.fqdnSets                    OK
bmysql-upgrade-1  | boulder_sa_integration.gorp_migrations             OK
bmysql-upgrade-1  | boulder_sa_integration.incidents                   OK
bmysql-upgrade-1  | boulder_sa_integration.issuedNames                 OK
bmysql-upgrade-1  | boulder_sa_integration.keyHashToSerial             OK
bmysql-upgrade-1  | boulder_sa_integration.newOrdersRL                 OK
bmysql-upgrade-1  | boulder_sa_integration.orderFqdnSets               OK
bmysql-upgrade-1  | boulder_sa_integration.orderToAuthz2               OK
bmysql-upgrade-1  | boulder_sa_integration.orders                      OK
bmysql-upgrade-1  | boulder_sa_integration.paused                      OK
bmysql-upgrade-1  | boulder_sa_integration.precertificates             OK
bmysql-upgrade-1  | boulder_sa_integration.registrations               OK
bmysql-upgrade-1  | boulder_sa_integration.replacementOrders           OK
bmysql-upgrade-1  | boulder_sa_integration.requestedNames              OK
bmysql-upgrade-1  | boulder_sa_integration.revokedCertificates         OK
bmysql-upgrade-1  | boulder_sa_integration.serials                     OK
bmysql-upgrade-1  | boulder_sa_test
bmysql-upgrade-1  | boulder_sa_test.authz2                             OK
bmysql-upgrade-1  | boulder_sa_test.blockedKeys                        OK
bmysql-upgrade-1  | boulder_sa_test.certificateStatus                  OK
bmysql-upgrade-1  | boulder_sa_test.certificates                       OK
bmysql-upgrade-1  | boulder_sa_test.certificatesPerName                OK
bmysql-upgrade-1  | boulder_sa_test.challenges                         OK
bmysql-upgrade-1  | boulder_sa_test.crlShards                          OK
bmysql-upgrade-1  | boulder_sa_test.fqdnSets                           OK
bmysql-upgrade-1  | boulder_sa_test.gorp_migrations                    OK
bmysql-upgrade-1  | boulder_sa_test.incidents                          OK
bmysql-upgrade-1  | boulder_sa_test.issuedNames                        OK
bmysql-upgrade-1  | boulder_sa_test.keyHashToSerial                    OK
bmysql-upgrade-1  | boulder_sa_test.newOrdersRL                        OK
bmysql-upgrade-1  | boulder_sa_test.orderFqdnSets                      OK
bmysql-upgrade-1  | boulder_sa_test.orderToAuthz2                      OK
bmysql-upgrade-1  | boulder_sa_test.orders                             OK
bmysql-upgrade-1  | boulder_sa_test.paused                             OK
bmysql-upgrade-1  | boulder_sa_test.precertificates                    OK
bmysql-upgrade-1  | boulder_sa_test.registrations                      OK
bmysql-upgrade-1  | boulder_sa_test.replacementOrders                  OK
bmysql-upgrade-1  | boulder_sa_test.requestedNames                     OK
bmysql-upgrade-1  | boulder_sa_test.revokedCertificates                OK
bmysql-upgrade-1  | boulder_sa_test.serials                            OK
bmysql-upgrade-1  | incidents_sa_integration
bmysql-upgrade-1  | incidents_sa_integration.gorp_migrations           OK
bmysql-upgrade-1  | incidents_sa_integration.incident_bar              OK
bmysql-upgrade-1  | incidents_sa_integration.incident_foo              OK
bmysql-upgrade-1  | incidents_sa_test
bmysql-upgrade-1  | incidents_sa_test.gorp_migrations                  OK
bmysql-upgrade-1  | incidents_sa_test.incident_bar                     OK
bmysql-upgrade-1  | incidents_sa_test.incident_foo                     OK
bmysql-upgrade-1  | information_schema
bmysql-upgrade-1  | performance_schema
bmysql-upgrade-1  | sys
bmysql-upgrade-1  | sys.sys_config                                     OK
bmysql-upgrade-1  | Phase 7/8: uninstalling plugins
bmysql-upgrade-1  | Phase 8/8: Running 'FLUSH PRIVILEGES'
bmysql-upgrade-1  | OK

Best Regards.

[vsc55]

Sorry, I forgot to do git pull before doing composer up. 🤦‍♂️

I went back to a copy before doing anything and now it has updated, and loads everything except the "Orders" and "Challenges" sections which still generate an error:

Orders:

gui-1             | 2026/03/11 21:35:13 GET /orders
gui-1             | 2026/03/11 21:35:13 errorHandler: err=Error 1146 (42S02): Table 'boulder_sa.requestedNames' doesn't exist
gui-1             | main.GetOrders({0x16b87f8, 0xc0010363c0}, 0xc000d15540, {0x0, 0x0})
gui-1             |     /go/src/labca/acme.go:404 +0x528
gui-1             | main.ordersHandler({0x16b87f8, 0xc0010363c0}, 0xc000d15540)
gui-1             |     /go/src/labca/main.go:2771 +0x95
gui-1             | net/http.HandlerFunc.ServeHTTP(0xfbf740?, {0x16b87f8?, 0xc0010363c0?}, 0xc0010363c0?)
gui-1             |     /usr/local/go/src/net/http/server.go:2322 +0x29
gui-1             | main.authorized.func1({0x16b87f8, 0xc0010363c0}, 0xc000d15540)
gui-1             |     /go/src/labca/main.go:3154 +0x32e
gui-1             | net/http.HandlerFunc.ServeHTTP(0xc000d15180?, {0x16b87f8?, 0xc0010363c0?}, 0x5f1fd10859?)
gui-1             |     /usr/local/go/src/net/http/server.go:2322 +0x29
gui-1             | github.com/gorilla/mux.(*Router).ServeHTTP(0xc000148480, {0x16b87f8, 0xc0010363c0}, 0xc000d15040)
gui-1             |     /root/go/pkg/mod/github.com/gorilla/mux@v1.8.1/mux.go:212 +0x1e2
gui-1             | net/http.serverHandler.ServeHTTP({0xc000911540?}, {0x16b87f8?, 0xc0010363c0?}, 0x6?)
gui-1             |     /usr/local/go/src/net/http/server.go:3340 +0x8e
gui-1             | net/http.(*conn).serve(0xc000d4b950, {0x16bad80, 0xc000bbe6f0})
gui-1             |     /usr/local/go/src/net/http/server.go:2109 +0x665
gui-1             | created by net/http.(*Server).Serve in goroutine 1
gui-1             |     /usr/local/go/src/net/http/server.go:3493 +0x485

Challenges:

gui-1             | 2026/03/11 21:35:56 GET /challenges
gui-1             | 2026/03/11 21:35:56 errorHandler: err=Error 1146 (42S02): Table 'boulder_sa.challenges' doesn't exist
gui-1             | main.GetChallenges({0x16b87f8, 0xc001036000}, 0xc000d14280, {0x0, 0x0}, {0x1f63c60, 0x0, 0x0})
gui-1             |     /go/src/labca/acme.go:925 +0x5ad
gui-1             | main.challengesHandler({0x16b87f8, 0xc001036000}, 0xc000d14280)
gui-1             |     /go/src/labca/main.go:2825 +0xa5
gui-1             | net/http.HandlerFunc.ServeHTTP(0xfbf740?, {0x16b87f8?, 0xc001036000?}, 0xc001036000?)
gui-1             |     /usr/local/go/src/net/http/server.go:2322 +0x29
gui-1             | main.authorized.func1({0x16b87f8, 0xc001036000}, 0xc000d14280)
gui-1             |     /go/src/labca/main.go:3154 +0x32e
gui-1             | net/http.HandlerFunc.ServeHTTP(0xc000d14140?, {0x16b87f8?, 0xc001036000?}, 0x693760b04c?)
gui-1             |     /usr/local/go/src/net/http/server.go:2322 +0x29
gui-1             | github.com/gorilla/mux.(*Router).ServeHTTP(0xc000148480, {0x16b87f8, 0xc001036000}, 0xc000d14000)
gui-1             |     /root/go/pkg/mod/github.com/gorilla/mux@v1.8.1/mux.go:212 +0x1e2
gui-1             | net/http.serverHandler.ServeHTTP({0xc00093e080?}, {0x16b87f8?, 0xc001036000?}, 0x6?)
gui-1             |     /usr/local/go/src/net/http/server.go:3340 +0x8e
gui-1             | net/http.(*conn).serve(0xc0000da090, {0x16bad80, 0xc000bbe6f0})
gui-1             |     /usr/local/go/src/net/http/server.go:2109 +0x665
gui-1             | created by net/http.(*Server).Serve in goroutine 1
gui-1             |     /usr/local/go/src/net/http/server.go:3493 +0x485

Process Upgrade:

bmysql-1     | 2026-03-11 21:21:51+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.13+maria~ubu2204 started.
bmysql-1     | 2026-03-11 21:21:52+00:00 [Warn] [Entrypoint]: /sys/fs/cgroup///memory.pressure not writable, functionality unavailable to MariaDB
bmysql-1     | 2026-03-11 21:21:52+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
bmysql-1     | 2026-03-11 21:21:52+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.13+maria~ubu2204 started.
bmysql-1     | 2026-03-11 21:21:52+00:00 [Note] [Entrypoint]: MariaDB upgrade not required
bmysql-1     | 2026-03-11 21:21:52 0 [Note] Starting MariaDB 10.11.13-MariaDB-ubu2204 source revision 8fb09426b98583916ccfd4f8c49741adc115bac3 server_uid QTxwsD1WsfVyzXd4WrfjDQW2Mio= as process 1
bmysql-1     | 2026-03-11 21:21:53 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
bmysql-1     | 2026-03-11 21:21:53 0 [Note] InnoDB: Number of transaction pools: 1
bmysql-1     | 2026-03-11 21:21:53 0 [Note] InnoDB: Using SSE4.2 crc32 instructions
bmysql-1     | 2026-03-11 21:21:53 0 [Warning] mariadbd: io_uring_queue_init() failed with EPERM: sysctl kernel.io_uring_disabled has the value 2, or 1 and the user of the process is not a member of sysctl kernel.io_uring_group. (see man 2 io_uring_setup).
bmysql-1     | 2026-03-11 21:21:53 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF
bmysql-1     | 2026-03-11 21:21:53 0 [Note] InnoDB: innodb_buffer_pool_size_max=128m, innodb_buffer_pool_size=128m
bmysql-1     | 2026-03-11 21:21:53 0 [Note] InnoDB: Completed initialization of buffer pool
bmysql-1     | 2026-03-11 21:21:53 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
bmysql-1     | 2026-03-11 21:21:53 0 [Note] InnoDB: End of log at LSN=2958154
bmysql-1     | 2026-03-11 21:21:53 0 [Note] InnoDB: 128 rollback segments are active.
bmysql-1     | 2026-03-11 21:21:53 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
bmysql-1     | 2026-03-11 21:21:53 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
bmysql-1     | 2026-03-11 21:21:53 0 [Note] InnoDB: log sequence number 2958154; transaction id 3088
bmysql-1     | 2026-03-11 21:21:53 0 [Note] Plugin 'FEEDBACK' is disabled.
bmysql-1     | 2026-03-11 21:21:53 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
bmysql-1     | 2026-03-11 21:21:53 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
bmysql-1     | 2026-03-11 21:21:53 0 [Note] Server socket created on IP: '0.0.0.0'.
bmysql-1     | 2026-03-11 21:21:53 0 [Note] Server socket created on IP: '::'.
bmysql-1     | 2026-03-11 21:21:53 0 [Note] InnoDB: Buffer pool(s) load completed at 260311 21:21:53
bmysql-1     | 2026-03-11 21:21:54 0 [Note] mariadbd: ready for connections.
bmysql-1     | Version: '10.11.13-MariaDB-ubu2204'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
bmysql-1     | 2026-03-11 21:21:56 3 [Warning] Aborted connection 3 to db: 'unconnected' user: 'unauthenticated' host: '10.77.77.77' (This connection closed normally without authentication)
bmysql-1     | 2026-03-11 21:21:59 9 [Warning] Aborted connection 9 to db: 'unconnected' user: 'unauthenticated' host: '10.77.77.77' (This connection closed normally without authentication)
bmysql-1     | 2026-03-11 21:22:02 15 [Warning] Aborted connection 15 to db: 'unconnected' user: 'unauthenticated' host: '10.77.77.77' (This connection closed normally without authentication)
bconsul-1    |      Gossip Encryption: false
bconsul-1    |       Auto-Encrypt-TLS: false
bconsul-1    |            ACL Enabled: false
bconsul-1    |      Reporting Enabled: false
bconsul-1    |     ACL Default Policy: allow
bconsul-1    |              HTTPS TLS: Verify Incoming: false, Verify Outgoing: false, Min Version: TLSv1_2
bconsul-1    |               gRPC TLS: Verify Incoming: false, Min Version: TLSv1_2
bconsul-1    |       Internal RPC TLS: Verify Incoming: false, Verify Outgoing: false (Verify Hostname: false), Min Version: TLSv1_2
bconsul-1    |
bconsul-1    | ==> Log data will now stream in as it occurs:
bconsul-1    |
bmysql-1     | 2026-03-11 21:22:05 21 [Warning] Aborted connection 21 to db: 'unconnected' user: 'unauthenticated' host: '10.77.77.77' (This connection closed normally without authentication)
boulder-1    | Connected to boulder-mysql:3306
boulder-1    | Connected to bpkimetal:8080
boulder-1    |
boulder-1    | boulder_sa
boulder-1    | Already exists - skipping create
boulder-1    | sql-migrate: Migration failed: Error 1049: Unknown database 'boulder_sa'
boulder-1    | Try `docker compose down --volumes` and running again
boulder-1    | Connected to boulder-mysql:3306
boulder-1    | Connected to bpkimetal:8080
boulder-1    |
boulder-1    | boulder_sa
boulder-1    | Already exists - skipping create
boulder-1    | sql-migrate: Migration failed: Error 1049: Unknown database 'boulder_sa'
boulder-1    | Try `docker compose down --volumes` and running again
boulder-1    | Connected to boulder-mysql:3306
boulder-1    | Connected to bpkimetal:8080
boulder-1    |
boulder-1    | boulder_sa
boulder-1    | Already exists - skipping create
boulder-1    | sql-migrate: Migration failed: Error 1049: Unknown database 'boulder_sa'
boulder-1    | Try `docker compose down --volumes` and running again
boulder-1    | Connected to boulder-mysql:3306
boulder-1    | Connected to bpkimetal:8080
boulder-1    |
boulder-1    | boulder_sa
boulder-1    | Already exists - skipping create
boulder-1 exited with code 1 (restarting)
boulder-1    | Connected to boulder-mysql:3306
boulder-1    | Connected to bpkimetal:8080
boulder-1    |
boulder-1    | boulder_sa
boulder-1    | Already exists - skipping create
boulder-1    | sql-migrate: Migration failed: Error 1049: Unknown database 'boulder_sa'
boulder-1    | Try `docker compose down --volumes` and running again
boulder-1    | sql-migrate: Migration failed: Error 1049: Unknown database 'boulder_sa'
boulder-1    | Try `docker compose down --volumes` and running again
bmysql-1     | 2026-03-11 21:22:09 27 [Warning] Aborted connection 27 to db: 'unconnected' user: 'unauthenticated' host: '10.77.77.77' (This connection closed normally without authentication)
boulder-1 exited with code 1 (restarting)
boulder-1    | Connected to boulder-mysql:3306
boulder-1    | Connected to bpkimetal:8080
boulder-1    |
boulder-1    | boulder_sa
boulder-1    | Already exists - skipping create
boulder-1    | sql-migrate: Migration failed: Error 1049: Unknown database 'boulder_sa'
boulder-1    | Try `docker compose down --volumes` and running again
boulder-1 exited with code 1 (restarting)
boulder-1    | Connected to boulder-mysql:3306
bmysql-1     | 2026-03-11 21:22:12 33 [Warning] Aborted connection 33 to db: 'unconnected' user: 'unauthenticated' host: '10.77.77.77' (This connection closed normally without authentication)
boulder-1    | Connected to bpkimetal:8080
boulder-1    |
boulder-1    | boulder_sa
boulder-1    | Already exists - skipping create
boulder-1    | sql-migrate: Migration failed: Error 1049: Unknown database 'boulder_sa'
boulder-1    | Try `docker compose down --volumes` and running again
bmysql-upgrade-1  | Phase 1/8: Checking and upgrading mysql database
bmysql-upgrade-1  | Processing databases
bmysql-upgrade-1  | mysql
bmysql-upgrade-1  | mysql.column_stats                                 OK
bmysql-upgrade-1  | mysql.columns_priv                                 OK
bmysql-upgrade-1  | mysql.db                                           OK
bmysql-upgrade-1  | mysql.event                                        OK
bmysql-upgrade-1  | mysql.func                                         OK
bmysql-upgrade-1  | mysql.global_priv                                  OK
bmysql-upgrade-1  | mysql.gtid_slave_pos                               OK
bmysql-upgrade-1  | mysql.help_category                                OK
bmysql-upgrade-1  | mysql.help_keyword                                 OK
bmysql-upgrade-1  | mysql.help_relation                                OK
bmysql-upgrade-1  | mysql.help_topic                                   OK
bmysql-upgrade-1  | mysql.index_stats                                  OK
bmysql-upgrade-1  | mysql.innodb_index_stats                           OK
bmysql-upgrade-1  | mysql.innodb_table_stats                           OK
bmysql-upgrade-1  | mysql.plugin                                       OK
bmysql-upgrade-1  | mysql.proc                                         OK
bmysql-upgrade-1  | mysql.procs_priv                                   OK
bmysql-upgrade-1  | mysql.proxies_priv                                 OK
bmysql-upgrade-1  | mysql.roles_mapping                                OK
bmysql-upgrade-1  | mysql.servers                                      OK
bmysql-upgrade-1  | mysql.table_stats                                  OK
bmysql-upgrade-1  | mysql.tables_priv                                  OK
bmysql-upgrade-1  | mysql.time_zone                                    OK
bmysql-upgrade-1  | mysql.time_zone_leap_second                        OK
bmysql-upgrade-1  | mysql.time_zone_name                               OK
bmysql-upgrade-1  | mysql.time_zone_transition                         OK
bmysql-upgrade-1  | mysql.time_zone_transition_type                    OK
bmysql-upgrade-1  | mysql.transaction_registry                         OK
bmysql-upgrade-1  | Phase 2/8: Installing used storage engines... Skipped
bmysql-upgrade-1  | Phase 3/8: Running 'mysql_fix_privilege_tables'
boulder-1 exited with code 1 (restarting)
bmysql-1          | 2026-03-11 21:22:18 46 [Warning] Aborted connection 46 to db: 'unconnected' user: 'unauthenticated' host: '10.77.77.77' (This connection closed normally without authentication)
boulder-1         | Connected to boulder-mysql:3306
boulder-1         | Connected to bpkimetal:8080
boulder-1         |
boulder-1         | boulder_sa
boulder-1         | Already exists - skipping create
boulder-1         | sql-migrate: Migration failed: Error 1049: Unknown database 'boulder_sa'
boulder-1         | Try `docker compose down --volumes` and running again
boulder-1 exited with code 1 (restarting)
bmysql-1          | 2026-03-11 21:22:26 52 [Warning] Aborted connection 52 to db: 'unconnected' user: 'unauthenticated' host: '10.77.77.77' (This connection closed normally without authentication)
boulder-1         | Connected to boulder-mysql:3306
boulder-1         | Connected to bpkimetal:8080
boulder-1         |
boulder-1         | boulder_sa
boulder-1         | Already exists - skipping create
boulder-1         | sql-migrate: Migration failed: Error 1049: Unknown database 'boulder_sa'
boulder-1         | Try `docker compose down --volumes` and running again
boulder-1 exited with code 1 (restarting)
bmysql-1          | 2026-03-11 21:22:41 58 [Warning] Aborted connection 58 to db: 'unconnected' user: 'unauthenticated' host: '10.77.77.77' (This connection closed normally without authentication)
boulder-1         | Connected to boulder-mysql:3306
boulder-1         | Connected to bpkimetal:8080
boulder-1         |
boulder-1         | boulder_sa
boulder-1         | Already exists - skipping create
boulder-1         | sql-migrate: Migration failed: Error 1049: Unknown database 'boulder_sa'
boulder-1         | Try `docker compose down --volumes` and running again
boulder-1 exited with code 1 (restarting)
bmysql-1          | 2026-03-11 21:23:09 64 [Warning] Aborted connection 64 to db: 'unconnected' user: 'unauthenticated' host: '10.77.77.77' (This connection closed normally without authentication)
boulder-1         | Connected to boulder-mysql:3306
boulder-1         | Connected to bpkimetal:8080
boulder-1         |
boulder-1         | boulder_sa
boulder-1         | Already exists - skipping create
boulder-1         | sql-migrate: Migration failed: Error 1049: Unknown database 'boulder_sa'
boulder-1         | Try `docker compose down --volumes` and running again
boulder-1 exited with code 1 (restarting)
bmysql-upgrade-1  | Phase 4/8: Fixing views
bmysql-upgrade-1  | mysql.user                                         OK
bmysql-upgrade-1  | sys.host_summary                                   OK
bmysql-upgrade-1  | sys.host_summary_by_file_io                        OK
bmysql-upgrade-1  | sys.host_summary_by_file_io_type                   OK
bmysql-upgrade-1  | sys.host_summary_by_stages                         OK
bmysql-upgrade-1  | sys.host_summary_by_statement_latency              OK
bmysql-upgrade-1  | sys.host_summary_by_statement_type                 OK
bmysql-upgrade-1  | sys.innodb_buffer_stats_by_schema                  OK
bmysql-upgrade-1  | sys.innodb_buffer_stats_by_table                   OK
bmysql-upgrade-1  | sys.innodb_lock_waits                              OK
bmysql-upgrade-1  | sys.io_by_thread_by_latency                        OK
bmysql-upgrade-1  | sys.io_global_by_file_by_bytes                     OK
bmysql-upgrade-1  | sys.io_global_by_file_by_latency                   OK
bmysql-upgrade-1  | sys.io_global_by_wait_by_bytes                     OK
bmysql-upgrade-1  | sys.io_global_by_wait_by_latency                   OK
bmysql-upgrade-1  | sys.latest_file_io                                 OK
bmysql-upgrade-1  | sys.memory_by_host_by_current_bytes                OK
bmysql-upgrade-1  | sys.memory_by_thread_by_current_bytes              OK
bmysql-upgrade-1  | sys.memory_by_user_by_current_bytes                OK
bmysql-upgrade-1  | sys.memory_global_by_current_bytes                 OK
bmysql-upgrade-1  | sys.memory_global_total                            OK
bmysql-upgrade-1  | sys.metrics                                        OK
bmysql-upgrade-1  | sys.processlist                                    OK
bmysql-upgrade-1  | sys.ps_check_lost_instrumentation                  OK
bmysql-upgrade-1  | sys.schema_auto_increment_columns                  OK
bmysql-upgrade-1  | sys.schema_index_statistics                        OK
bmysql-upgrade-1  | sys.schema_object_overview                         OK
bmysql-upgrade-1  | sys.schema_redundant_indexes                       OK
bmysql-upgrade-1  | sys.schema_table_lock_waits                        OK
bmysql-upgrade-1  | sys.schema_table_statistics                        OK
bmysql-upgrade-1  | sys.schema_table_statistics_with_buffer            OK
bmysql-upgrade-1  | sys.schema_tables_with_full_table_scans            OK
bmysql-upgrade-1  | sys.schema_unused_indexes                          OK
bmysql-upgrade-1  | sys.session                                        OK
bmysql-upgrade-1  | sys.session_ssl_status                             OK
bmysql-upgrade-1  | sys.statement_analysis                             OK
bmysql-upgrade-1  | sys.statements_with_errors_or_warnings             OK
bmysql-upgrade-1  | sys.statements_with_full_table_scans               OK
bmysql-upgrade-1  | sys.statements_with_runtimes_in_95th_percentile    OK
bmysql-upgrade-1  | sys.statements_with_sorting                        OK
bmysql-upgrade-1  | sys.statements_with_temp_tables                    OK
bmysql-upgrade-1  | sys.user_summary                                   OK
bmysql-upgrade-1  | sys.user_summary_by_file_io                        OK
bmysql-upgrade-1  | sys.user_summary_by_file_io_type                   OK
bmysql-upgrade-1  | sys.user_summary_by_stages                         OK
bmysql-upgrade-1  | sys.user_summary_by_statement_latency              OK
bmysql-upgrade-1  | sys.user_summary_by_statement_type                 OK
bmysql-upgrade-1  | sys.version                                        OK
bmysql-upgrade-1  | sys.wait_classes_global_by_avg_latency             OK
bmysql-upgrade-1  | sys.wait_classes_global_by_latency                 OK
bmysql-upgrade-1  | sys.waits_by_host_by_latency                       OK
bmysql-upgrade-1  | sys.waits_by_user_by_latency                       OK
bmysql-upgrade-1  | sys.waits_global_by_latency                        OK
bmysql-upgrade-1  | sys.x$host_summary                                 OK
bmysql-upgrade-1  | sys.x$host_summary_by_file_io                      OK
bmysql-upgrade-1  | sys.x$host_summary_by_file_io_type                 OK
bmysql-upgrade-1  | sys.x$host_summary_by_stages                       OK
bmysql-upgrade-1  | sys.x$host_summary_by_statement_latency            OK
bmysql-upgrade-1  | sys.x$host_summary_by_statement_type               OK
bmysql-upgrade-1  | sys.x$innodb_buffer_stats_by_schema                OK
bmysql-upgrade-1  | sys.x$innodb_buffer_stats_by_table                 OK
bmysql-upgrade-1  | sys.x$innodb_lock_waits                            OK
bmysql-upgrade-1  | sys.x$io_by_thread_by_latency                      OK
bmysql-upgrade-1  | sys.x$io_global_by_file_by_bytes                   OK
bmysql-upgrade-1  | sys.x$io_global_by_file_by_latency                 OK
bmysql-upgrade-1  | sys.x$io_global_by_wait_by_bytes                   OK
bmysql-upgrade-1  | sys.x$io_global_by_wait_by_latency                 OK
bmysql-upgrade-1  | sys.x$latest_file_io                               OK
bmysql-upgrade-1  | sys.x$memory_by_host_by_current_bytes              OK
bmysql-upgrade-1  | sys.x$memory_by_thread_by_current_bytes            OK
bmysql-upgrade-1  | sys.x$memory_by_user_by_current_bytes              OK
bmysql-upgrade-1  | sys.x$memory_global_by_current_bytes               OK
bmysql-upgrade-1  | sys.x$memory_global_total                          OK
bmysql-upgrade-1  | sys.x$processlist                                  OK
bmysql-upgrade-1  | sys.x$ps_digest_95th_percentile_by_avg_us          OK
bmysql-upgrade-1  | sys.x$ps_digest_avg_latency_distribution           OK
bmysql-upgrade-1  | sys.x$ps_schema_table_statistics_io                OK
bmysql-upgrade-1  | sys.x$schema_flattened_keys                        OK
bmysql-upgrade-1  | sys.x$schema_index_statistics                      OK
bmysql-upgrade-1  | sys.x$schema_table_lock_waits                      OK
bmysql-upgrade-1  | sys.x$schema_table_statistics                      OK
bmysql-upgrade-1  | sys.x$schema_table_statistics_with_buffer          OK
bmysql-upgrade-1  | sys.x$schema_tables_with_full_table_scans          OK
bmysql-upgrade-1  | sys.x$session                                      OK
bmysql-upgrade-1  | sys.x$statement_analysis                           OK
bmysql-upgrade-1  | sys.x$statements_with_errors_or_warnings           OK
bmysql-upgrade-1  | sys.x$statements_with_full_table_scans             OK
bmysql-upgrade-1  | sys.x$statements_with_runtimes_in_95th_percentile  OK
bmysql-upgrade-1  | sys.x$statements_with_sorting                      OK
bmysql-upgrade-1  | sys.x$statements_with_temp_tables                  OK
bmysql-upgrade-1  | sys.x$user_summary                                 OK
bmysql-upgrade-1  | sys.x$user_summary_by_file_io                      OK
bmysql-upgrade-1  | sys.x$user_summary_by_file_io_type                 OK
bmysql-upgrade-1  | sys.x$user_summary_by_stages                       OK
bmysql-upgrade-1  | sys.x$user_summary_by_statement_latency            OK
bmysql-upgrade-1  | sys.x$user_summary_by_statement_type               OK
bmysql-upgrade-1  | sys.x$wait_classes_global_by_avg_latency           OK
bmysql-upgrade-1  | sys.x$wait_classes_global_by_latency               OK
bmysql-upgrade-1  | sys.x$waits_by_host_by_latency                     OK
bmysql-upgrade-1  | sys.x$waits_by_user_by_latency                     OK
bmysql-upgrade-1  | sys.x$waits_global_by_latency                      OK
bmysql-upgrade-1  | Phase 5/8: Fixing table and database names
bmysql-upgrade-1  | Phase 6/8: Checking and upgrading tables
bmysql-upgrade-1  | Processing databases
bmysql-upgrade-1  | boulder_sa_integration
bmysql-upgrade-1  | boulder_sa_integration.authz2                      OK
bmysql-upgrade-1  | boulder_sa_integration.blockedKeys                 OK
bmysql-upgrade-1  | boulder_sa_integration.certificateStatus           OK
bmysql-upgrade-1  | boulder_sa_integration.certificates                OK
bmysql-upgrade-1  | boulder_sa_integration.certificatesPerName         OK
bmysql-upgrade-1  | boulder_sa_integration.challenges                  OK
bmysql-upgrade-1  | boulder_sa_integration.crlShards                   OK
bmysql-upgrade-1  | boulder_sa_integration.fqdnSets                    OK
bmysql-upgrade-1  | boulder_sa_integration.gorp_migrations             OK
bmysql-upgrade-1  | boulder_sa_integration.incidents                   OK
bmysql-upgrade-1  | boulder_sa_integration.issuedNames                 OK
bmysql-upgrade-1  | boulder_sa_integration.keyHashToSerial             OK
bmysql-upgrade-1  | boulder_sa_integration.newOrdersRL                 OK
bmysql-upgrade-1  | boulder_sa_integration.orderFqdnSets               OK
bmysql-upgrade-1  | boulder_sa_integration.orderToAuthz2               OK
bmysql-upgrade-1  | boulder_sa_integration.orders                      OK
bmysql-upgrade-1  | boulder_sa_integration.paused                      OK
bmysql-upgrade-1  | boulder_sa_integration.precertificates             OK
bmysql-upgrade-1  | boulder_sa_integration.registrations               OK
bmysql-upgrade-1  | boulder_sa_integration.replacementOrders           OK
bmysql-upgrade-1  | boulder_sa_integration.requestedNames              OK
bmysql-upgrade-1  | boulder_sa_integration.revokedCertificates         OK
bmysql-upgrade-1  | boulder_sa_integration.serials                     OK
bmysql-upgrade-1  | boulder_sa_test
bmysql-upgrade-1  | boulder_sa_test.authz2                             OK
bmysql-upgrade-1  | boulder_sa_test.blockedKeys                        OK
bmysql-upgrade-1  | boulder_sa_test.certificateStatus                  OK
bmysql-upgrade-1  | boulder_sa_test.certificates                       OK
bmysql-upgrade-1  | boulder_sa_test.certificatesPerName                OK
bmysql-upgrade-1  | boulder_sa_test.challenges                         OK
bmysql-upgrade-1  | boulder_sa_test.crlShards                          OK
bmysql-upgrade-1  | boulder_sa_test.fqdnSets                           OK
bmysql-upgrade-1  | boulder_sa_test.gorp_migrations                    OK
bmysql-upgrade-1  | boulder_sa_test.incidents                          OK
bmysql-upgrade-1  | boulder_sa_test.issuedNames                        OK
bmysql-upgrade-1  | boulder_sa_test.keyHashToSerial                    OK
bmysql-upgrade-1  | boulder_sa_test.newOrdersRL                        OK
bmysql-upgrade-1  | boulder_sa_test.orderFqdnSets                      OK
bmysql-upgrade-1  | boulder_sa_test.orderToAuthz2                      OK
bmysql-upgrade-1  | boulder_sa_test.orders                             OK
bmysql-upgrade-1  | boulder_sa_test.paused                             OK
bmysql-upgrade-1  | boulder_sa_test.precertificates                    OK
bmysql-upgrade-1  | boulder_sa_test.registrations                      OK
bmysql-upgrade-1  | boulder_sa_test.replacementOrders                  OK
bmysql-upgrade-1  | boulder_sa_test.requestedNames                     OK
bmysql-upgrade-1  | boulder_sa_test.revokedCertificates                OK
bmysql-upgrade-1  | boulder_sa_test.serials                            OK
bmysql-upgrade-1  | incidents_sa_integration
bmysql-upgrade-1  | incidents_sa_integration.gorp_migrations           OK
bmysql-upgrade-1  | incidents_sa_integration.incident_bar              OK
bmysql-upgrade-1  | incidents_sa_integration.incident_foo              OK
bmysql-upgrade-1  | incidents_sa_test
bmysql-upgrade-1  | incidents_sa_test.gorp_migrations                  OK
bmysql-upgrade-1  | incidents_sa_test.incident_bar                     OK
bmysql-upgrade-1  | incidents_sa_test.incident_foo                     OK
bmysql-upgrade-1  | information_schema
bmysql-upgrade-1  | performance_schema
bmysql-upgrade-1  | sys
bmysql-upgrade-1  | sys.sys_config                                     OK
bmysql-upgrade-1  | Phase 7/8: uninstalling plugins
bmysql-upgrade-1  | Phase 8/8: Running 'FLUSH PRIVILEGES'
bmysql-upgrade-1  | OK
bmysql-upgrade-1  | Database "boulder_sa_integration" dropped
bmysql-upgrade-1  | Database "incidents_sa_integration" dropped
bmysql-upgrade-1 exited with code 0
bmysql-1          | 2026-03-11 21:24:02 88 [Warning] Aborted connection 88 to db: 'unconnected' user: 'unauthenticated' host: '10.77.77.77' (This connection closed normally without authentication)
boulder-1         | Connected to boulder-mysql:3306
boulder-1         | Connected to bpkimetal:8080
boulder-1         |
boulder-1         | boulder_sa
boulder-1         | Already exists - skipping create
boulder-1         | sql-migrate: Applied 1 migration
boulder-1         | Added users from ../db-users/boulder_sa.sql
boulder-1         |
boulder-1         | incidents_sa
boulder-1         | Already exists - skipping create
boulder-1         | sql-migrate: Applied 1 migration
boulder-1         | Added users from ../db-users/incidents_sa.sql
boulder-1         |
boulder-1         | database setup complete
boulder-1         | chall-test-srv - 2026/03/11 21:24:08 Creating HTTP-01 challenge server on 64.112.117.122:80
boulder-1         | chall-test-srv - 2026/03/11 21:24:08 Creating HTTPS HTTP-01 challenge server on 64.112.117.122:443
boulder-1         | chall-test-srv - 2026/03/11 21:24:08 Creating TCP and UDP DNS server on :8053
boulder-1         | chall-test-srv - 2026/03/11 21:24:08 Creating TCP and UDP DNS server on :8054
boulder-1         | chall-test-srv - 2026/03/11 21:24:08 Creating DoH server on :8343
boulder-1         | chall-test-srv - 2026/03/11 21:24:08 Creating DoH server on :8443
boulder-1         | chall-test-srv - 2026/03/11 21:24:08 Creating TLS-ALPN-01 challenge server on 64.112.117.134:443
boulder-1         | chall-test-srv - 2026/03/11 21:24:08 Answering A queries with 64.112.117.122 by default
boulder-1         | chall-test-srv - 2026/03/11 21:24:08 Starting challenge servers
boulder-1         | chall-test-srv - 2026/03/11 21:24:08 Starting management server on :8055
boulder-1         | 2026-03-11T21:24:09.061820+00:00Z nonce-service[73]: 6 nonce-service SHwOMg Debug server listening on :8111
boulder-1         | 2026-03-11T21:24:09.061932+00:00Z nonce-service[73]: 6 nonce-service DcD3Ew [AUDIT] Process starting JSON={"Command":"nonce-service","BuildID":"v0.20260309.0 +5c3a8315","BuildTime":"Wed Mar 11 19:07:18 UTC 2026","GoVersion":"go1.25.8","BuildHost":"labca-v26.03"}
boulder-1         | 2026-03-11T21:24:09.094942+00:00Z nonce-service[73]: 6 nonce-service XMNWLw grpc listening on 10.77.77.77:9301
boulder-1         | 2026-03-11T21:24:09.099011+00:00Z nonce-service[73]: 6 nonce-service 9e42wQ Nonce server listening on 10.77.77.77:9301 with prefix "GKXSL_3j"
boulder-1         | 2026-03-11T21:24:09.892806+00:00Z remoteva[86]: 6 remoteva xUfFqg Debug server listening on :8012
boulder-1         | 2026-03-11T21:24:09.892856+00:00Z remoteva[86]: 6 remoteva R2cMcg [AUDIT] Process starting JSON={"Command":"remoteva","BuildID":"v0.20260309.0 +5c3a8315","BuildTime":"Wed Mar 11 19:07:18 UTC 2026","GoVersion":"go1.25.8","BuildHost":"labca-v26.03"}
boulder-1         | 2026-03-11T21:24:09.912479+00:00Z remoteva[86]: 6 remoteva dl4Llw grpc listening on :9498
boulder-1         | 2026-03-11T21:24:10.960236+00:00Z boulder-sa[99]: 6 boulder-sa JSwbxQ Debug server listening on :8103
boulder-1         | 2026-03-11T21:24:10.960296+00:00Z boulder-sa[99]: 6 boulder-sa xqr57Q [AUDIT] Process starting JSON={"Command":"boulder-sa","BuildID":"v0.20260309.0 +5c3a8315","BuildTime":"Wed Mar 11 19:07:18 UTC 2026","GoVersion":"go1.25.8","BuildHost":"labca-v26.03"}
boulder-1         | 2026-03-11T21:24:10.979483+00:00Z boulder-sa[99]: 6 boulder-sa yyK66Q grpc listening on :9495
boulder-1         | 2026-03-11T21:24:10.980880+00:00Z boulder-sa[99]: 6 boulder-sa TxT0-Q transitioning overall health from "NOT_SERVING" to "SERVING"
boulder-1         | 2026-03-11T21:24:10.981311+00:00Z boulder-sa[99]: 6 boulder-sa YDS2_Q transitioning health of "sa.StorageAuthorityReadOnly" from "NOT_SERVING" to "SERVING"
boulder-1         | 2026-03-11T21:24:10.982051+00:00Z boulder-sa[99]: 6 boulder-sa TxT0-Q transitioning overall health from "NOT_SERVING" to "SERVING"
boulder-1         | 2026-03-11T21:24:10.982075+00:00Z boulder-sa[99]: 6 boulder-sa woWwCg transitioning health of "sa.StorageAuthority" from "NOT_SERVING" to "SERVING"
boulder-1         | 2026-03-11T21:24:12.352366+00:00Z boulder-sa[117]: 6 boulder-sa EkbZxA Debug server listening on :8003
boulder-1         | 2026-03-11T21:24:12.352598+00:00Z boulder-sa[117]: 6 boulder-sa xqr57Q [AUDIT] Process starting JSON={"Command":"boulder-sa","BuildID":"v0.20260309.0 +5c3a8315","BuildTime":"Wed Mar 11 19:07:18 UTC 2026","GoVersion":"go1.25.8","BuildHost":"labca-v26.03"}
boulder-1         | 2026-03-11T21:24:12.362406+00:00Z boulder-sa[117]: 6 boulder-sa TjT17A grpc listening on :9395
boulder-1         | 2026-03-11T21:24:12.363935+00:00Z boulder-sa[117]: 6 boulder-sa TxT0-Q transitioning overall health from "NOT_SERVING" to "SERVING"
boulder-1         | 2026-03-11T21:24:12.364342+00:00Z boulder-sa[117]: 6 boulder-sa YDS2_Q transitioning health of "sa.StorageAuthorityReadOnly" from "NOT_SERVING" to "SERVING"
boulder-1         | 2026-03-11T21:24:12.365462+00:00Z boulder-sa[117]: 6 boulder-sa TxT0-Q transitioning overall health from "NOT_SERVING" to "SERVING"
boulder-1         | 2026-03-11T21:24:12.365624+00:00Z boulder-sa[117]: 6 boulder-sa woWwCg transitioning health of "sa.StorageAuthority" from "NOT_SERVING" to "SERVING"
boulder-1         | 2026/03/11 21:24:13 zendesk-test-srv listening at :9701
boulder-1         | 2026-03-11T21:24:13.803032+00:00Z remoteva[137]: 6 remoteva fxbMMw Debug server listening on :8011
boulder-1         | 2026-03-11T21:24:13.803073+00:00Z remoteva[137]: 6 remoteva R2cMcg [AUDIT] Process starting JSON={"Command":"remoteva","BuildID":"v0.20260309.0 +5c3a8315","BuildTime":"Wed Mar 11 19:07:18 UTC 2026","GoVersion":"go1.25.8","BuildHost":"labca-v26.03"}
boulder-1         | 2026-03-11T21:24:13.806084+00:00Z remoteva[137]: 6 remoteva YlX7Ag grpc listening on :9397
boulder-1         | 2026-03-11T21:24:14.851164+00:00Z log-validator[150]: 6 log-validator 3IOorQ Debug server listening on :8016
boulder-1         | 2026-03-11T21:24:14.851229+00:00Z log-validator[150]: 6 log-validator xaEfTA [AUDIT] Process starting JSON={"Command":"log-validator","BuildID":"v0.20260309.0 +5c3a8315","BuildTime":"Wed Mar 11 19:07:18 UTC 2026","GoVersion":"go1.25.8","BuildHost":"labca-v26.03"}
boulder-1         | 2026-03-11T21:24:14.977902+00:00Z nonce-service[157]: 6 nonce-service ZB0A3A Debug server listening on :8113
boulder-1         | 2026-03-11T21:24:14.978433+00:00Z nonce-service[157]: 6 nonce-service DcD3Ew [AUDIT] Process starting JSON={"Command":"nonce-service","BuildID":"v0.20260309.0 +5c3a8315","BuildTime":"Wed Mar 11 19:07:18 UTC 2026","GoVersion":"go1.25.8","BuildHost":"labca-v26.03"}
boulder-1         | 2026-03-11T21:24:14.989226+00:00Z nonce-service[157]: 6 nonce-service 7r_bKw grpc listening on 10.77.77.77:9501
boulder-1         | 2026-03-11T21:24:14.989620+00:00Z nonce-service[157]: 6 nonce-service ZfDxSA Nonce server listening on 10.77.77.77:9501 with prefix "Zy_ps6C1"
boulder-1         | 2026-03-11T21:24:16.465790+00:00Z nonce-service[177]: 6 nonce-service 8i0Hqw Debug server listening on :8112
boulder-1         | 2026-03-11T21:24:16.465839+00:00Z nonce-service[177]: 6 nonce-service DcD3Ew [AUDIT] Process starting JSON={"Command":"nonce-service","BuildID":"v0.20260309.0 +5c3a8315","BuildTime":"Wed Mar 11 19:07:18 UTC 2026","GoVersion":"go1.25.8","BuildHost":"labca-v26.03"}
boulder-1         | 2026-03-11T21:24:16.472868+00:00Z nonce-service[177]: 6 nonce-service 2dUZKg grpc listening on 10.77.77.77:9401
boulder-1         | 2026-03-11T21:24:16.473251+00:00Z nonce-service[177]: 6 nonce-service NzfMng Nonce server listening on 10.77.77.77:9401 with prefix "kyO0YGSo"
boulder-1         | 2026-03-11T21:24:17.499681+00:00Z remoteva[190]: 6 remoteva kCTv9g Debug server listening on :8023
boulder-1         | 2026-03-11T21:24:17.499731+00:00Z remoteva[190]: 6 remoteva R2cMcg [AUDIT] Process starting JSON={"Command":"remoteva","BuildID":"v0.20260309.0 +5c3a8315","BuildTime":"Wed Mar 11 19:07:18 UTC 2026","GoVersion":"go1.25.8","BuildHost":"labca-v26.03"}
boulder-1         | 2026-03-11T21:24:17.502751+00:00Z remoteva[190]: 6 remoteva 4G4M4A grpc listening on :9499
boulder-1         | 2026-03-11T21:24:18.532088+00:00Z boulder-va[203]: 6 boulder-va sdO9Wg Debug server listening on :8004
boulder-1         | 2026-03-11T21:24:18.532410+00:00Z boulder-va[203]: 6 boulder-va oAsahw [AUDIT] Process starting JSON={"Command":"boulder-va","BuildID":"v0.20260309.0 +5c3a8315","BuildTime":"Wed Mar 11 19:07:18 UTC 2026","GoVersion":"go1.25.8","BuildHost":"labca-v26.03"}
boulder-1         | 2026-03-11T21:24:18.538857+00:00Z boulder-va[203]: 6 boulder-va 7aGRcg grpc listening on :9392
boulder-1         | 2026-03-11T21:24:19.569101+00:00Z boulder-publisher[215]: 6 boulder-publisher O8XOJQ Debug server listening on :8109
boulder-1         | 2026-03-11T21:24:19.569144+00:00Z boulder-publisher[215]: 6 boulder-publisher LFw2AA [AUDIT] Process starting JSON={"Command":"boulder-publisher","BuildID":"v0.20260309.0 +5c3a8315","BuildTime":"Wed Mar 11 19:07:18 UTC 2026","GoVersion":"go1.25.8","BuildHost":"labca-v26.03"}
boulder-1         | 2026-03-11T21:24:19.577162+00:00Z boulder-publisher[215]: 6 boulder-publisher 0ubX7g grpc listening on :9491
boulder-1         | 2026-03-11T21:24:20.626210+00:00Z boulder-publisher[227]: 6 boulder-publisher DK8MJA Debug server listening on :8009
boulder-1         | 2026-03-11T21:24:20.626760+00:00Z boulder-publisher[227]: 6 boulder-publisher LFw2AA [AUDIT] Process starting JSON={"Command":"boulder-publisher","BuildID":"v0.20260309.0 +5c3a8315","BuildTime":"Wed Mar 11 19:07:18 UTC 2026","GoVersion":"go1.25.8","BuildHost":"labca-v26.03"}
boulder-1         | 2026-03-11T21:24:20.630993+00:00Z boulder-publisher[227]: 6 boulder-publisher V_CY6w grpc listening on :9391
boulder-1         | 2026-03-11T21:24:22.005083+00:00Z boulder-ra[246]: 6 boulder-ra 7MTSSw Debug server listening on :8118
boulder-1         | 2026-03-11T21:24:22.005683+00:00Z boulder-ra[246]: 6 boulder-ra 2N8z8A [AUDIT] Process starting JSON={"Command":"boulder-ra","BuildID":"v0.20260309.0 +5c3a8315","BuildTime":"Wed Mar 11 19:07:18 UTC 2026","GoVersion":"go1.25.8","BuildHost":"labca-v26.03"}
boulder-1         | 2026-03-11T21:24:22.006406+00:00Z boulder-ra[246]: 6 boulder-ra Hnmo6w loading identifier policy, sha256: a1225cd7c1d0f2c0bca2cdeab6b28a8c0914995a7253dcdd19ae94acb86f2722
boulder-1         | 2026-03-11T21:24:22.048817+00:00Z boulder-ra[246]: 6 boulder-ra anh_nw grpc listening on :9594
boulder-1         | 2026-03-11T21:24:22.049397+00:00Z boulder-ra[246]: 6 boulder-ra JcOwJw health check of gRPC service "ra.RegistrationAuthority" failed: waiting for overrides
boulder-1         | 2026-03-11T21:24:22.049634+00:00Z boulder-ra[246]: 6 boulder-ra VX65Bw health check of gRPC service "ra.SCTProvider" failed: waiting for overrides
boulder-1         | health checking ra.boulder (localhost:9594): rpc error: code = Unavailable desc = pickfirst: health check failure: connection active but health check failed. status=NOT_SERVING
boulder-1         | health checking ra.boulder (localhost:9594): rpc error: code = Unavailable desc = pickfirst: health check failure: connection active but health check failed. status=NOT_SERVING
boulder-1         | health checking ra.boulder (localhost:9594): rpc error: code = Unavailable desc = pickfirst: health check failure: connection active but health check failed. status=NOT_SERVING
boulder-1         | health checking ra.boulder (localhost:9594): rpc error: code = Unavailable desc = pickfirst: health check failure: connection active but health check failed. status=NOT_SERVING
boulder-1         | health checking ra.boulder (localhost:9594): rpc error: code = Unavailable desc = pickfirst: health check failure: connection active but health check failed. status=NOT_SERVING
boulder-1         | 2026-03-11T21:24:27.050395+00:00Z boulder-ra[246]: 6 boulder-ra TxT0-Q transitioning overall health from "NOT_SERVING" to "SERVING"
boulder-1         | 2026-03-11T21:24:27.050462+00:00Z boulder-ra[246]: 6 boulder-ra TxT0-Q transitioning overall health from "NOT_SERVING" to "SERVING"
boulder-1         | 2026-03-11T21:24:27.050485+00:00Z boulder-ra[246]: 6 boulder-ra evkj9A transitioning health of "ra.SCTProvider" from "NOT_SERVING" to "SERVING"
boulder-1         | 2026-03-11T21:24:27.050797+00:00Z boulder-ra[246]: 6 boulder-ra Hvsl7g transitioning health of "ra.RegistrationAuthority" from "NOT_SERVING" to "SERVING"
boulder-1         | 2026-03-11T21:24:28.069815+00:00Z crl-storer[259]: 6 crl-storer Wj7fKQ Debug server listening on :9667
boulder-1         | 2026-03-11T21:24:28.070190+00:00Z crl-storer[259]: 6 crl-storer MaWc7w [AUDIT] Process starting JSON={"Command":"crl-storer","BuildID":"v0.20260309.0 +5c3a8315","BuildTime":"Wed Mar 11 19:07:18 UTC 2026","GoVersion":"go1.25.8","BuildHost":"labca-v26.03"}
boulder-1         | 2026-03-11T21:24:28.084420+00:00Z crl-storer[259]: 6 crl-storer LMOBNA grpc listening on :9309
boulder-1         | 2026-03-11T21:24:28.201577+00:00Z boulder-va[266]: 6 boulder-va hrl_Ww Debug server listening on :8104
boulder-1         | 2026-03-11T21:24:28.202091+00:00Z boulder-va[266]: 6 boulder-va oAsahw [AUDIT] Process starting JSON={"Command":"boulder-va","BuildID":"v0.20260309.0 +5c3a8315","BuildTime":"Wed Mar 11 19:07:18 UTC 2026","GoVersion":"go1.25.8","BuildHost":"labca-v26.03"}
boulder-1         | 2026-03-11T21:24:28.205557+00:00Z boulder-va[266]: 6 boulder-va aLfedw grpc listening on :9492
boulder-1         | 2026/03/11 21:24:29 salesforce-test-srv Salesforce REST API and OAuth server listening at :9601
boulder-1         | 2026/03/11 21:24:29 salesforce-test-srv Salesforce Pardot API server listening at :9602
boulder-1         | 2026-03-11T21:24:29.574477+00:00Z boulder-ra[286]: 6 boulder-ra evTVPA Debug server listening on :8119
boulder-1         | 2026-03-11T21:24:29.574596+00:00Z boulder-ra[286]: 6 boulder-ra 2N8z8A [AUDIT] Process starting JSON={"Command":"boulder-ra","BuildID":"v0.20260309.0 +5c3a8315","BuildTime":"Wed Mar 11 19:07:18 UTC 2026","GoVersion":"go1.25.8","BuildHost":"labca-v26.03"}
boulder-1         | 2026-03-11T21:24:29.575078+00:00Z boulder-ra[286]: 6 boulder-ra Hnmo6w loading identifier policy, sha256: a1225cd7c1d0f2c0bca2cdeab6b28a8c0914995a7253dcdd19ae94acb86f2722
boulder-1         | 2026-03-11T21:24:29.585597+00:00Z boulder-ra[286]: 6 boulder-ra M8Y5nQ grpc listening on :9694
boulder-1         | 2026-03-11T21:24:29.586743+00:00Z boulder-ra[286]: 6 boulder-ra JcOwJw health check of gRPC service "ra.RegistrationAuthority" failed: waiting for overrides
boulder-1         | 2026-03-11T21:24:29.586815+00:00Z boulder-ra[286]: 6 boulder-ra VX65Bw health check of gRPC service "ra.SCTProvider" failed: waiting for overrides
boulder-1         | health checking ra.boulder (localhost:9694): rpc error: code = Unavailable desc = pickfirst: health check failure: connection active but health check failed. status=NOT_SERVING
boulder-1         | health checking ra.boulder (localhost:9694): rpc error: code = Unavailable desc = pickfirst: health check failure: connection active but health check failed. status=NOT_SERVING
boulder-1         | health checking ra.boulder (localhost:9694): rpc error: code = Unavailable desc = pickfirst: health check failure: connection active but health check failed. status=NOT_SERVING
boulder-1         | health checking ra.boulder (localhost:9694): rpc error: code = Unavailable desc = pickfirst: health check failure: connection active but health check failed. status=NOT_SERVING
boulder-1         | health checking ra.boulder (localhost:9694): rpc error: code = Unavailable desc = pickfirst: health check failure: connection active but health check failed. status=NOT_SERVING
boulder-1         | 2026-03-11T21:24:34.586885+00:00Z boulder-ra[286]: 6 boulder-ra TxT0-Q transitioning overall health from "NOT_SERVING" to "SERVING"
boulder-1         | 2026-03-11T21:24:34.587095+00:00Z boulder-ra[286]: 6 boulder-ra evkj9A transitioning health of "ra.SCTProvider" from "NOT_SERVING" to "SERVING"
boulder-1         | 2026-03-11T21:24:34.587216+00:00Z boulder-ra[286]: 6 boulder-ra TxT0-Q transitioning overall health from "NOT_SERVING" to "SERVING"
boulder-1         | 2026-03-11T21:24:34.587259+00:00Z boulder-ra[286]: 6 boulder-ra Hvsl7g transitioning health of "ra.RegistrationAuthority" from "NOT_SERVING" to "SERVING"
boulder-1         | 2026-03-11T21:24:35.638840+00:00Z boulder-ca[299]: 6 boulder-ca CU0VKw Debug server listening on :8101
boulder-1         | 2026-03-11T21:24:35.638951+00:00Z boulder-ca[299]: 6 boulder-ca Z_0v6g [AUDIT] Process starting JSON={"Command":"boulder-ca","BuildID":"v0.20260309.0 +5c3a8315","BuildTime":"Wed Mar 11 19:07:18 UTC 2026","GoVersion":"go1.25.8","BuildHost":"labca-v26.03"}
boulder-1         | 2026-03-11T21:24:35.639518+00:00Z boulder-ca[299]: 6 boulder-ca Hnmo6w loading identifier policy, sha256: a1225cd7c1d0f2c0bca2cdeab6b28a8c0914995a7253dcdd19ae94acb86f2722
boulder-1         | health checking ca.boulder (localhost:9493): rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:9493: connect: connection refused"
boulder-1         | 2026-03-11T21:24:36.798327+00:00Z boulder-ca[299]: 6 boulder-ca _ofZAA grpc listening on :9493
boulder-1         | 2026-03-11T21:24:37.706549+00:00Z email-exporter[311]: 6 email-exporter x4hkQg Debug server listening on :8114
boulder-1         | 2026-03-11T21:24:37.706651+00:00Z email-exporter[311]: 6 email-exporter EbjYSw [AUDIT] Process starting JSON={"Command":"email-exporter","BuildID":"v0.20260309.0 +5c3a8315","BuildTime":"Wed Mar 11 19:07:18 UTC 2026","GoVersion":"go1.25.8","BuildHost":"labca-v26.03"}
boulder-1         | 2026-03-11T21:24:37.713673+00:00Z email-exporter[311]: 6 email-exporter 2eif0g grpc listening on :9603
boulder-1         | 2026-03-11T21:24:37.819874+00:00Z boulder-ca[318]: 6 boulder-ca PifXKg Debug server listening on :8001
boulder-1         | 2026-03-11T21:24:37.819920+00:00Z boulder-ca[318]: 6 boulder-ca Z_0v6g [AUDIT] Process starting JSON={"Command":"boulder-ca","BuildID":"v0.20260309.0 +5c3a8315","BuildTime":"Wed Mar 11 19:07:18 UTC 2026","GoVersion":"go1.25.8","BuildHost":"labca-v26.03"}
boulder-1         | 2026-03-11T21:24:37.820046+00:00Z boulder-ca[318]: 6 boulder-ca Hnmo6w loading identifier policy, sha256: a1225cd7c1d0f2c0bca2cdeab6b28a8c0914995a7253dcdd19ae94acb86f2722
boulder-1         | 2026-03-11T21:24:38.434614+00:00Z boulder-ca[318]: 6 boulder-ca e5GWBQ grpc listening on :9393
boulder-1         | 2026-03-11T21:24:38.898781+00:00Z boulder-ra[332]: 6 boulder-ra sxwcsg Debug server listening on :8102
boulder-1         | 2026-03-11T21:24:38.899594+00:00Z boulder-ra[332]: 6 boulder-ra 2N8z8A [AUDIT] Process starting JSON={"Command":"boulder-ra","BuildID":"v0.20260309.0 +5c3a8315","BuildTime":"Wed Mar 11 19:07:18 UTC 2026","GoVersion":"go1.25.8","BuildHost":"labca-v26.03"}
boulder-1         | 2026-03-11T21:24:38.899722+00:00Z boulder-ra[332]: 6 boulder-ra Hnmo6w loading identifier policy, sha256: a1225cd7c1d0f2c0bca2cdeab6b28a8c0914995a7253dcdd19ae94acb86f2722
boulder-1         | 2026-03-11T21:24:38.909896+00:00Z boulder-ra[332]: 6 boulder-ra XRK9ng grpc listening on :9494
boulder-1         | 2026-03-11T21:24:38.910332+00:00Z boulder-ra[332]: 6 boulder-ra TxT0-Q transitioning overall health from "NOT_SERVING" to "SERVING"
boulder-1         | 2026-03-11T21:24:38.910351+00:00Z boulder-ra[332]: 6 boulder-ra Hvsl7g transitioning health of "ra.RegistrationAuthority" from "NOT_SERVING" to "SERVING"
boulder-1         | 2026-03-11T21:24:38.910380+00:00Z boulder-ra[332]: 6 boulder-ra TxT0-Q transitioning overall health from "NOT_SERVING" to "SERVING"
boulder-1         | 2026-03-11T21:24:38.910392+00:00Z boulder-ra[332]: 6 boulder-ra evkj9A transitioning health of "ra.SCTProvider" from "NOT_SERVING" to "SERVING"
boulder-1         | 2026-03-11T21:24:39.937913+00:00Z boulder-ra[346]: 6 boulder-ra hHbesw Debug server listening on :8002
boulder-1         | 2026-03-11T21:24:39.937963+00:00Z boulder-ra[346]: 6 boulder-ra 2N8z8A [AUDIT] Process starting JSON={"Command":"boulder-ra","BuildID":"v0.20260309.0 +5c3a8315","BuildTime":"Wed Mar 11 19:07:18 UTC 2026","GoVersion":"go1.25.8","BuildHost":"labca-v26.03"}
boulder-1         | 2026-03-11T21:24:39.938023+00:00Z boulder-ra[346]: 6 boulder-ra Hnmo6w loading identifier policy, sha256: a1225cd7c1d0f2c0bca2cdeab6b28a8c0914995a7253dcdd19ae94acb86f2722
boulder-1         | 2026-03-11T21:24:39.949851+00:00Z boulder-ra[346]: 6 boulder-ra 2ATymw grpc listening on :9394
boulder-1         | 2026-03-11T21:24:39.950577+00:00Z boulder-ra[346]: 6 boulder-ra JcOwJw health check of gRPC service "ra.RegistrationAuthority" failed: waiting for overrides
boulder-1         | 2026-03-11T21:24:39.950611+00:00Z boulder-ra[346]: 6 boulder-ra VX65Bw health check of gRPC service "ra.SCTProvider" failed: waiting for overrides
boulder-1         | health checking ra.boulder (localhost:9394): rpc error: code = Unavailable desc = pickfirst: health check failure: connection active but health check failed. status=NOT_SERVING
boulder-1         | health checking ra.boulder (localhost:9394): rpc error: code = Unavailable desc = pickfirst: health check failure: connection active but health check failed. status=NOT_SERVING
boulder-1         | health checking ra.boulder (localhost:9394): rpc error: code = Unavailable desc = pickfirst: health check failure: connection active but health check failed. status=NOT_SERVING
boulder-1         | health checking ra.boulder (localhost:9394): rpc error: code = Unavailable desc = pickfirst: health check failure: connection active but health check failed. status=NOT_SERVING
boulder-1         | health checking ra.boulder (localhost:9394): rpc error: code = Unavailable desc = pickfirst: health check failure: connection active but health check failed. status=NOT_SERVING
boulder-1         | 2026-03-11T21:24:44.951554+00:00Z boulder-ra[346]: 6 boulder-ra TxT0-Q transitioning overall health from "NOT_SERVING" to "SERVING"
boulder-1         | 2026-03-11T21:24:44.951596+00:00Z boulder-ra[346]: 6 boulder-ra TxT0-Q transitioning overall health from "NOT_SERVING" to "SERVING"
boulder-1         | 2026-03-11T21:24:44.951625+00:00Z boulder-ra[346]: 6 boulder-ra Hvsl7g transitioning health of "ra.RegistrationAuthority" from "NOT_SERVING" to "SERVING"
boulder-1         | 2026-03-11T21:24:44.951678+00:00Z boulder-ra[346]: 6 boulder-ra evkj9A transitioning health of "ra.SCTProvider" from "NOT_SERVING" to "SERVING"
boulder-1         | 2026-03-11T21:24:46.001636+00:00Z boulder-wfe2[360]: 6 boulder-wfe2 U3fC3Q Debug server listening on :8013
boulder-1         | 2026-03-11T21:24:46.002128+00:00Z boulder-wfe2[360]: 6 boulder-wfe2 8Apf9g [AUDIT] Process starting JSON={"Command":"boulder-wfe2","BuildID":"v0.20260309.0 +5c3a8315","BuildTime":"Wed Mar 11 19:07:18 UTC 2026","GoVersion":"go1.25.8","BuildHost":"labca-v26.03"}
boulder-1         | 2026-03-11T21:24:46.009543+00:00Z boulder-wfe2[360]: 6 boulder-wfe2 Hnmo6w loading identifier policy, sha256: a1225cd7c1d0f2c0bca2cdeab6b28a8c0914995a7253dcdd19ae94acb86f2722
boulder-1         | 2026-03-11T21:24:46.013260+00:00Z boulder-wfe2[360]: 6 boulder-wfe2 7351yQ Server running, listening on :4001....
boulder-1         | 2026-03-11T21:24:46.015326+00:00Z boulder-wfe2[360]: 6 boulder-wfe2 BMBqTQ TLS server listening on :4431
boulder-1         | 2026-03-11T21:24:46.332048+00:00Z sfe[380]: 6 sfe ZtKhNA Debug server listening on :8015
boulder-1         | 2026-03-11T21:24:46.332149+00:00Z sfe[380]: 6 sfe A9GvMQ [AUDIT] Process starting JSON={"Command":"sfe","BuildID":"v0.20260309.0 +5c3a8315","BuildTime":"Wed Mar 11 19:07:18 UTC 2026","GoVersion":"go1.25.8","BuildHost":"labca-v26.03"}
boulder-1         | 2026-03-11T21:24:46.338217+00:00Z sfe[380]: 6 sfe Gk0Gqw Server running, listening on 0.0.0.0:4003....

[MassiPi]

i also got the Unknown database 'boulder_sa' error trying to update 25.12
i have a dedicated lxc running labca so nothing strange.
thanks

[ChrisThePCGeek]

Same Issue here. reverted back to a backup prior to watchtower updating stuff and ran git pull then docker compose pull and docker compose up -d --force-recreate and still same issue with boulder crash looping with the unknown database error.

[hakwerk]

I think this is now sorted in the latest release v26.03.1

[MassiPi]

Thanks a lot, i'll check asap

[ChrisThePCGeek]

It worked for me this morning. (after I added more disk space to my VM sine the mysql upgrade said out of space and crashed lol) the web gui still says update available though to 26.03.1 and shows version of 26.03 but I did git pull, compose pull and re-created the stack

[vsc55]

Confirmed, with the new version everything now works correctly. 🎉

[MassiPi]

Thanks a lot, i'll check asap

mine still doesn't work, i also tried increasing the disk size, but i still get the same error after the update.
just to ckeck i'm not missing something: the update is as easy as a

docker compose pull
docker compose up -d --remove-orphans
docker image prune

inside /labca/labca/build folder, right?

The install didn't return any error and i can't find dedicated logs for the update update, but these are the logs of the containers that show errors:

mysql

2026-03-21 17:20:44 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2026-03-21 17:20:44 0 [Note] InnoDB: Using Linux native AIO
2026-03-21 17:20:44 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2026-03-21 17:20:44 0 [Note] InnoDB: Completed initialization of buffer pool
2026-03-21 17:20:44 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1653687,1653687
2026-03-21 17:20:45 0 [Note] InnoDB: Starting final batch to recover 47 pages from redo log.
2026-03-21 17:20:46 0 [Note] InnoDB: 128 rollback segments are active.
2026-03-21 17:20:46 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2026-03-21 17:20:46 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2026-03-21 17:20:46 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2026-03-21 17:20:46 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2026-03-21 17:20:46 0 [Note] InnoDB: 10.5.29 started; log sequence number 1754390; transaction id 3843
2026-03-21 17:20:46 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2026-03-21 17:20:46 0 [Note] Plugin 'FEEDBACK' is disabled.
2026-03-21 17:20:47 0 [Note] Server socket created on IP: '0.0.0.0'.
2026-03-21 17:20:49 0 [Note] Reading of all Master_info entries succeeded
2026-03-21 17:20:49 0 [Note] Added new Master_info '' to hash table
2026-03-21 17:20:49 0 [Note] mysqld: ready for connections.
Version: '10.5.29-MariaDB-ubu2004'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
2026-03-21 17:20:49 3 [Warning] Aborted connection 3 to db: 'unconnected' user: 'unauthenticated' host: '10.77.77.77' (This connection closed normally without authentication)
2026-03-21 17:20:49 0 [Note] InnoDB: Buffer pool(s) load completed at 260321 17:20:49
2026-03-22  7:48:30 0 [Note] mysqld (initiated by: unknown): Normal shutdown
2026-03-22  7:48:30 0 [Note] InnoDB: FTS optimize thread exiting.
2026-03-22  7:48:30 0 [Note] InnoDB: Starting shutdown...
2026-03-22  7:48:30 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2026-03-22  7:48:30 0 [Note] InnoDB: Buffer pool(s) dump completed at 260322  7:48:30
2026-03-22  7:48:30 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2026-03-22  7:48:30 0 [Note] InnoDB: Shutdown completed; log sequence number 1756199; transaction id 3855
2026-03-22  7:48:30 0 [Note] Event Scheduler: Purging the queue. 0 events
2026-03-22  7:48:30 0 [Note] mysqld: Shutdown complete
2026-03-22 07:48:48+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.5.29+maria~ubu2004 started.
2026-03-22 07:48:50+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2026-03-22 07:48:50+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.5.29+maria~ubu2004 started.
2026-03-22 07:48:51+00:00 [Note] [Entrypoint]: MariaDB upgrade not required
2026-03-22  7:48:51 0 [Note] Starting MariaDB 10.5.29-MariaDB-ubu2004 source revision c461188ca6ad6ec3a54201eb87ebd75797d296df server_uid CpYkXR2i5BX6cTN/AcE62Kft1us= as process 1
2026-03-22  7:48:51 0 [Note] InnoDB: Uses event mutexes
2026-03-22  7:48:51 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2026-03-22  7:48:51 0 [Note] InnoDB: Number of pools: 1
2026-03-22  7:48:51 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2026-03-22  7:48:51 0 [Note] InnoDB: Using Linux native AIO
2026-03-22  7:48:51 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2026-03-22  7:48:51 0 [Note] InnoDB: Completed initialization of buffer pool
2026-03-22  7:48:51 0 [Note] InnoDB: 128 rollback segments are active.
2026-03-22  7:48:51 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2026-03-22  7:48:51 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2026-03-22  7:48:51 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2026-03-22  7:48:51 0 [Note] InnoDB: 10.5.29 started; log sequence number 1756199; transaction id 3856
2026-03-22  7:48:51 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2026-03-22  7:48:51 0 [Note] Plugin 'FEEDBACK' is disabled.
2026-03-22  7:48:51 0 [Note] Server socket created on IP: '0.0.0.0'.
2026-03-22  7:48:51 0 [Note] Reading of all Master_info entries succeeded
2026-03-22  7:48:51 0 [Note] Added new Master_info '' to hash table
2026-03-22  7:48:51 0 [Note] mysqld: ready for connections.
Version: '10.5.29-MariaDB-ubu2004'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
2026-03-22  7:48:51 0 [Note] InnoDB: Buffer pool(s) load completed at 260322  7:48:51
2026-03-22  7:48:54 3 [Warning] Aborted connection 3 to db: 'unconnected' user: 'unauthenticated' host: '10.77.77.77' (This connection closed normally without authentication)
2026-03-22  7:57:04 0 [Note] mysqld (initiated by: unknown): Normal shutdown
2026-03-22  7:57:04 0 [Note] InnoDB: FTS optimize thread exiting.
2026-03-22  7:57:04 0 [Note] InnoDB: Starting shutdown...
2026-03-22  7:57:04 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2026-03-22  7:57:04 0 [Note] InnoDB: Buffer pool(s) dump completed at 260322  7:57:04
2026-03-22  7:57:04 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2026-03-22  7:57:04 0 [Note] InnoDB: Shutdown completed; log sequence number 1756211; transaction id 3866
2026-03-22  7:57:04 0 [Note] Event Scheduler: Purging the queue. 0 events
2026-03-22  7:57:04 0 [Note] mysqld: Shutdown complete
2026-03-22 07:57:15+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.5.29+maria~ubu2004 started.
2026-03-22 07:57:16+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2026-03-22 07:57:16+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.5.29+maria~ubu2004 started.
2026-03-22 07:57:17+00:00 [Note] [Entrypoint]: MariaDB upgrade not required
2026-03-22  7:57:17 0 [Note] Starting MariaDB 10.5.29-MariaDB-ubu2004 source revision c461188ca6ad6ec3a54201eb87ebd75797d296df server_uid ZCyLw9iUvOV1kdLq1/ZtVPlhX04= as process 1
2026-03-22  7:57:17 0 [Note] InnoDB: Uses event mutexes
2026-03-22  7:57:17 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2026-03-22  7:57:17 0 [Note] InnoDB: Number of pools: 1
2026-03-22  7:57:17 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2026-03-22  7:57:17 0 [Note] InnoDB: Using Linux native AIO
2026-03-22  7:57:17 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2026-03-22  7:57:17 0 [Note] InnoDB: Completed initialization of buffer pool
2026-03-22  7:57:17 0 [Note] InnoDB: 128 rollback segments are active.
2026-03-22  7:57:17 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2026-03-22  7:57:17 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2026-03-22  7:57:17 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2026-03-22  7:57:17 0 [Note] InnoDB: 10.5.29 started; log sequence number 1756211; transaction id 3856
2026-03-22  7:57:17 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2026-03-22  7:57:17 0 [Note] Plugin 'FEEDBACK' is disabled.
2026-03-22  7:57:17 0 [Note] Server socket created on IP: '0.0.0.0'.
2026-03-22  7:57:17 0 [Note] Reading of all Master_info entries succeeded
2026-03-22  7:57:17 0 [Note] Added new Master_info '' to hash table
2026-03-22  7:57:17 0 [Note] mysqld: ready for connections.
Version: '10.5.29-MariaDB-ubu2004'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
2026-03-22  7:57:17 0 [Note] InnoDB: Buffer pool(s) load completed at 260322  7:57:17
2026-03-22  7:57:21 3 [Warning] Aborted connection 3 to db: 'unconnected' user: 'unauthenticated' host: '10.77.77.77' (This connection closed normally without authentication)
2026-03-22  7:57:52 27 [Warning] Aborted connection 27 to db: 'boulder_sa_integration' user: 'badkeyrevoker' host: '10.77.77.77' (Got an error reading communication packets)
2026-03-22  7:57:52 25 [Warning] Aborted connection 25 to db: 'boulder_sa_integration' user: 'sa_ro' host: '10.77.77.77' (Got an error reading communication packets)
2026-03-22  7:57:52 24 [Warning] Aborted connection 24 to db: 'incidents_sa_integration' user: 'incidents_sa' host: '10.77.77.77' (Got an error reading communication packets)
2026-03-22  7:57:52 23 [Warning] Aborted connection 23 to db: 'boulder_sa_integration' user: 'sa_ro' host: '10.77.77.77' (Got an error reading communication packets)
2026-03-22  7:57:52 22 [Warning] Aborted connection 22 to db: 'boulder_sa_integration' user: 'sa' host: '10.77.77.77' (Got an error reading communication packets)
2026-03-22  7:57:53 26 [Warning] Aborted connection 26 to db: 'boulder_sa_integration' user: 'sa_ro' host: '10.77.77.77' (Got an error reading communication packets)
2026-03-22  7:57:53 19 [Warning] Aborted connection 19 to db: 'boulder_sa_integration' user: 'sa' host: '10.77.77.77' (Got an error reading communication packets)
2026-03-22  7:57:53 21 [Warning] Aborted connection 21 to db: 'incidents_sa_integration' user: 'incidents_sa' host: '10.77.77.77' (Got an error reading communication packets)
2026-03-22  7:57:53 20 [Warning] Aborted connection 20 to db: 'boulder_sa_integration' user: 'sa_ro' host: '10.77.77.77' (Got an error reading communication packets)

gui

2026/03/22 07:58:03 Listening on 0.0.0.0:3000...
2026/03/22 07:58:14 GET /certificates?active=1
2026/03/22 07:58:14 errorHandler: err=Error 1049 (42000): Unknown database 'boulder_sa'
main.GetCertificates({0x16b88d8, 0xc0004581e0}, 0xc0003a3180, {0x0, 0x0})
	/go/src/labca/acme.go:1240 +0x71e
main.certificatesHandler({0x16b88d8, 0xc0004581e0}, 0xc0003a3180)
	/go/src/labca/main.go:2852 +0x95
net/http.HandlerFunc.ServeHTTP(0xfbf6e0?, {0x16b88d8?, 0xc0004581e0?}, 0xc0004581e0?)
	/usr/local/go/src/net/http/server.go:2322 +0x29
main.authorized.func1({0x16b88d8, 0xc0004581e0}, 0xc0003a3180)
	/go/src/labca/main.go:3157 +0x32e
net/http.HandlerFunc.ServeHTTP(0xc0003a3040?, {0x16b88d8?, 0xc0004581e0?}, 0x61aba97c0?)
	/usr/local/go/src/net/http/server.go:2322 +0x29
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000d70000, {0x16b88d8, 0xc0004581e0}, 0xc0003a2f00)
	/root/go/pkg/mod/github.com/gorilla/mux@v1.8.1/mux.go:212 +0x1e2
net/http.serverHandler.ServeHTTP({0xc00018aec0?}, {0x16b88d8?, 0xc0004581e0?}, 0x6?)
	/usr/local/go/src/net/http/server.go:3340 +0x8e
net/http.(*conn).serve(0xc000d6e510, {0x16bae60, 0xc000d81350})
	/usr/local/go/src/net/http/server.go:2109 +0x665
created by net/http.(*Server).Serve in goroutine 1
	/usr/local/go/src/net/http/server.go:3493 +0x485
2026/03/22 07:58:18 GET /logs/boulder
2026/03/22 07:58:18 GET /ws?logType=boulder
2026/03/22 07:58:24 GET /logs/audit
2026/03/22 07:58:24 GET /ws?logType=audit
2026/03/22 07:58:26 GET /logs/cron
2026/03/22 07:58:26 GET /ws?logType=cron
2026/03/22 07:58:31 GET /logs/labca
2026/03/22 07:58:31 GET /ws?logType=labca
2026/03/22 07:58:50 GET /logs/cert
2026/03/22 07:58:59 GET /logs/commander
2026/03/22 07:59:04 GET /logs/web
2026/03/22 07:59:04 GET /ws?logType=web

bconsul

==> Log data will now stream in as it occurs:
2026-03-22T07:57:04.075Z [ERROR] agent.server: error performing anti-entropy sync of federation state: error="context canceled"
==> Starting Consul agent...
              Version: '1.15.4'
           Build Date: '2023-06-23 23:14:17 +0000 UTC'
              Node ID: 'f716c996-6932-bea1-e566-06dbe032ed01'
            Node name: 'cca072f1faad'
           Datacenter: 'dc1' (Segment: '<all>')
               Server: true (Bootstrap: false)
          Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: 8502, gRPC-TLS: 8503, DNS: 53)
         Cluster Addr: 10.77.77.10 (LAN: 8301, WAN: 8302)
    Gossip Encryption: false
     Auto-Encrypt-TLS: false
     Reporting Enabled: false
            HTTPS TLS: Verify Incoming: false, Verify Outgoing: false, Min Version: TLSv1_2
             gRPC TLS: Verify Incoming: false, Min Version: TLSv1_2
     Internal RPC TLS: Verify Incoming: false, Verify Outgoing: false (Verify Hostname: false), Min Version: TLSv1_2
==> Log data will now stream in as it occurs:
2026-03-22T07:57:16.497Z [ERROR] agent.server.cert-manager: failed to handle cache update event: error="leaf cert watch returned an error: CA is uninitialized and unable to sign certificates yet: no root certificate"
2026-03-22T07:57:16.497Z [ERROR] agent.server.cert-manager: failed to handle cache update event: error="leaf cert watch returned an error: CA is uninitialized and unable to sign certificates yet: no root certificate"
2026-03-22T07:57:16.498Z [ERROR] agent.server.cert-manager: failed to handle cache update event: error="leaf cert watch returned an error: CA is uninitialized and unable to sign certificates yet: no root certificate"
2026-03-22T07:57:16.498Z [ERROR] agent.server.cert-manager: failed to handle cache update event: error="leaf cert watch returned an error: CA is uninitialized and unable to sign certificates yet: no root certificate"

boulder

labca/entrypoint.sh: line 26: USE_VITESS: unbound variable
labca/entrypoint.sh: line 26: USE_VITESS: unbound variable
labca/entrypoint.sh: line 26: USE_VITESS: unbound variable
labca/entrypoint.sh: line 26: USE_VITESS: unbound variable
labca/entrypoint.sh: line 26: USE_VITESS: unbound variable
labca/entrypoint.sh: line 26: USE_VITESS: unbound variable
labca/entrypoint.sh: line 26: USE_VITESS: unbound variable
labca/entrypoint.sh: line 26: USE_VITESS: unbound variable
labca/entrypoint.sh: line 26: USE_VITESS: unbound variable
labca/entrypoint.sh: line 26: USE_VITESS: unbound variable
labca/entrypoint.sh: line 26: USE_VITESS: unbound variable
labca/entrypoint.sh: line 26: USE_VITESS: unbound variable
labca/entrypoint.sh: line 26: USE_VITESS: unbound variable

anything else i can check?
thanks

[vsc55]

Hi @MassiPi,

You mentioned that you have executed:

docker compose pull
docker compose up -d --remove-orphans
docker image prune

but before docker up this, have you executed git pull?

[MassiPi]

Hi @MassiPi,

You mentioned that you have executed:

docker compose pull
docker compose up -d --remove-orphans
docker image prune
but before docker up this, have you executed git pull?

i can't see a git pull here

https://github.com/hakwerk/labca?tab=readme-ov-file#update

and i did not in all my previous updates, but for sure i can try :)
thanks