TLS Handshake Fails with Error When Enabling ssl_asynch on; in QAT_SW Mode

[zelot1982]

When using the asynch-mode-nginx with the ssl_asynch on; directive enabled in the nginx.conf, TLS handshake fails and logs the following error:

peer closed connection in SSL handshake while SSL handshaking, client: 127.0.0.1, server: 0.0.0.0:443
Commenting out or removing ssl_asynch on; allows the handshake to proceed normally.

Environment:

Nginx version: 1.26.2
QAT Engine version: 1.6.2 (QAT_SW mode)
OpenSSL version: 3.0.16

Nginx Build Configuration:
--prefix=/usr/local/nginx
--with-http_ssl_module
--with-http_stub_status_module
--with-http_v2_module
--with-stream
--with-stream_ssl_module
--with-openssl=/usr/local/src/openssl-3.0.16
--with-openssl-opt='enable-async enable-legacy enable-engine'
--add-module=/usr/local/src/zstd-nginx-module-0.1.1
--add-dynamic-module=modules/nginx_qat_module/
--with-cc-opt=-I/usr/local/openssl/include
--with-ld-opt=-L/usr/local/openssl/lib64

QAT Engine Build Configuration:
--enable-qat_sw
--disable-qat_hw
--with-openssl_install_dir=/usr/local/openssl
--with-openssl_dir=/usr/local/src/openssl-3.0.16
--with-qat_sw_crypto_mb_install_dir=/usr/local
--with-qat_sw_ipsec_mb_install_dir=/usr/local
OpenSSL Build Configuration:

OpenSSL Build Configuration:
./Configure enable-legacy enable-async linux-x86_64 --prefix=/usr/local/openssl
Engine Status Output:

Run Command “openssl engine -c -t qatengine”
Echo:
(qatengine) Reference implementation of QAT crypto engine (qat_sw) v1.6.2
[RSA, id-aes128-GCM, id-aes192-GCM, id-aes256-GCM, X25519, SM2]
[ available ]

Nginx.conf:

worker_processes 48;
load_module modules/ngx_ssl_engine_qat_module.so;
events {
worker_connections 10240;
use epoll;
}

ssl_engine {
use_engine qatengine;
qat_engine {
qat_offload_mode async;
qat_notify_mode poll;
qat_poll_mode heuristic;
qat_heuristic_poll_asym_threshold 24;
qat_heuristic_poll_sym_threshold 12;
}
}

http {
include mime.types;
log_format main '[$time_local] $http_host "$request" $status $request_time [$upstream_addr $upstream_status $upstream_response_time] [$remote_addr] $body_bytes_sent "$http_user_agent"';
error_log logs/error.log debug;
access_log logs/access.log main;
zstd on;
zstd_comp_level 3;
zstd_min_length 256;
zstd_types text/plain application/json application/xml;
gzip on;
gzip_vary on;
gzip_types text/css application/javascript;
sendfile on;
tcp_nopush on;
keepalive_timeout 65;

server {
    listen 443 ssl;
    server_name localhost;
    ssl_certificate      ssl/zt-expresscom.pem;
    ssl_certificate_key  ssl/zt-expresscom.key;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_asynch on;
    ssl_ciphers AES128-GCM-SHA256;
    ssl_prefer_server_ciphers on;

nginx start qat-debug log：

OPENSSL_CONF=/usr/local/openssl/ssl/openssl_qat.cnf /usr/local/nginx/sbin/nginx

[WARN][1039828.472924] PID [8570] Thread [7f8ea1337740][e_qat.c:1122:bind_qat()] QAT Warnings enabled.[DEBUG][1039828.472941] PID [8570] Thread [7f8ea1337740][e_qat.c:1123:bind_qat()] QAT Debug enabled.[WARN][1039828.472946] PID [8570] Thread [7f8ea1337740][e_qat.c:1124:bind_qat()] (null) - Reference implementation of QAT crypto engine(qat_sw) v1.6.2
[DEBUG][1039828.472989] PID [8570] Thread [7f8ea1337740][e_qat.c:561:qat_sw_cpu_support()] QAT_SW - Processor supported: AVX512F = 1, VAES = 1, VPCLMULQDQ = 1
[DEBUG][1039828.473353] PID [8570] Thread [7f8ea1337740][qat_evp.c:1822:qat_get_RSA_methods()] QAT SW RSA Registration succeeded
[DEBUG][1039828.473371] PID [8570] Thread [7f8ea1337740][qat_evp.c:1643:qat_get_EC_methods()] QAT SW ECDSA registration succeeded
[DEBUG][1039828.473382] PID [8570] Thread [7f8ea1337740][qat_evp.c:1702:qat_get_EC_methods()] QAT SW ECDH registration succeeded
[DEBUG][1039828.473387] PID [8570] Thread [7f8ea1337740][qat_evp.c:857:qat_create_gcm_cipher_meth()] QAT SW AES_GCM_128 registration succeeded
[DEBUG][1039828.473391] PID [8570] Thread [7f8ea1337740][qat_evp.c:857:qat_create_gcm_cipher_meth()] QAT SW AES_GCM_192 registration succeeded
[DEBUG][1039828.473395] PID [8570] Thread [7f8ea1337740][qat_evp.c:857:qat_create_gcm_cipher_meth()] QAT SW AES_GCM_256 registration succeeded
[DEBUG][1039828.473403] PID [8570] Thread [7f8ea1337740][e_qat.c:603:qat_engine_init()] QAT Engine initialization:
[DEBUG][1039828.473406] PID [8570] Thread [7f8ea1337740][qat_sw_init.c:417:qat_sw_init()] QAT_SW initialization
[DEBUG][1039828.473409] PID [8570] Thread [7f8ea1337740][qat_sw_init.c:418:qat_sw_init()] - External polling: OFF
[DEBUG][1039828.473414] PID [8570] Thread [7f8ea1337740][qat_sw_init.c:419:qat_sw_init()] - Heuristic polling: OFF
[DEBUG][1039828.473423] PID [8570] Thread [7f8ea1337740][e_qat.c:651:qat_engine_finish_int()] ---- QAT Engine Finishing...

[DEBUG][1039828.473426] PID [8570] Thread [7f8ea1337740][e_qat.c:652:qat_engine_finish_int()] RSA Priv retries: 0, HW requests: 0, SW requests: 0
[DEBUG][1039828.473430] PID [8570] Thread [7f8ea1337740][e_qat.c:654:qat_engine_finish_int()] RSA Pub retries: 0, HW requests: 0, SW requests: 0
[DEBUG][1039828.473432] PID [8570] Thread [7f8ea1337740][e_qat.c:656:qat_engine_finish_int()] ECDH keygen retries: 0, HW requests: 0, SW requests: 0
[DEBUG][1039828.473435] PID [8570] Thread [7f8ea1337740][e_qat.c:659:qat_engine_finish_int()] ECDH derive retries: 0, HW requests: 0, SW requests: 0
[DEBUG][1039828.473437] PID [8570] Thread [7f8ea1337740][e_qat.c:662:qat_engine_finish_int()] ECX keygen retries: 0, HW requests: 0, SW requests: 0
[DEBUG][1039828.473440] PID [8570] Thread [7f8ea1337740][e_qat.c:664:qat_engine_finish_int()] ECX derive retries: 0, HW requests: 0, SW requests: 0
[DEBUG][1039828.473442] PID [8570] Thread [7f8ea1337740][e_qat.c:666:qat_engine_finish_int()] ECDSA sign retries: 0, HW requests: 0, SW requests: 0
[DEBUG][1039828.473444] PID [8570] Thread [7f8ea1337740][e_qat.c:668:qat_engine_finish_int()] SM4-CBC retries: 0, HW requests: 0, SW requests: 0
[DEBUG][1039828.473447] PID [8570] Thread [7f8ea1337740][qat_sw_init.c:447:qat_sw_finish_int()] ---- QAT_SW Finishing...

[DEBUG][1039828.473657] PID [8570] Thread [7f8ea1337740][e_qat.c:777:qat_engine_ctrl()] Enabled external polling
[DEBUG][1039828.473661] PID [8570] Thread [7f8ea1337740][e_qat.c:922:qat_engine_ctrl()] Enabled heuristic polling
[DEBUG][1039828.473667] PID [8570] Thread [7f8ea1337740][e_qat.c:603:qat_engine_init()] QAT Engine initialization:
[DEBUG][1039828.476797] PID [8570] Thread [7f8ea1337740][e_qat.c:651:qat_engine_finish_int()] ---- QAT Engine Finishing...

[DEBUG][1039828.476806] PID [8570] Thread [7f8ea1337740][e_qat.c:652:qat_engine_finish_int()] RSA Priv retries: 0, HW requests: 0, SW requests: 0
[DEBUG][1039828.476810] PID [8570] Thread [7f8ea1337740][e_qat.c:654:qat_engine_finish_int()] RSA Pub retries: 0, HW requests: 0, SW requests: 0
[DEBUG][1039828.476814] PID [8570] Thread [7f8ea1337740][e_qat.c:656:qat_engine_finish_int()] ECDH keygen retries: 0, HW requests: 0, SW requests: 0
[DEBUG][1039828.476818] PID [8570] Thread [7f8ea1337740][e_qat.c:659:qat_engine_finish_int()] ECDH derive retries: 0, HW requests: 0, SW requests: 0
[DEBUG][1039828.476822] PID [8570] Thread [7f8ea1337740][e_qat.c:662:qat_engine_finish_int()] ECX keygen retries: 0, HW requests: 0, SW requests: 0
[DEBUG][1039828.476826] PID [8570] Thread [7f8ea1337740][e_qat.c:664:qat_engine_finish_int()] ECX derive retries: 0, HW requests: 0, SW requests: 0
[DEBUG][1039828.476830] PID [8570] Thread [7f8ea1337740][e_qat.c:666:qat_engine_finish_int()] ECDSA sign retries: 0, HW requests: 0, SW requests: 0
[DEBUG][1039828.476834] PID [8570] Thread [7f8ea1337740][e_qat.c:668:qat_engine_finish_int()] SM4-CBC retries: 0, HW requests: 0, SW requests: 0
[DEBUG][1039828.476981] PID [8571] Thread [7f8ea1337740][e_qat.c:603:qat_engine_init()] QAT Engine initialization:

[hardikpatel9]

Hi @zelot1982 ,

To ensure QAT is functioning correctly, first verify its availability by running lsmod | grep qat. If QAT is not listed, install the QAT driver properly and ensure the configuration is updated, such as changing [SSL] to [SHIM] in the relevant configuration file (e.g., /etc/...).

Additionally, confirm that all QAT software-dependent libraries are installed. To rule out potential issues and cross-verify, individually build and test both QAT hardware and software. If you still see error then you can enable debug logs in Nginx by setting error_log in the configuration file, restarting Nginx, and share the detailed logs of error.log

[zelot1982]

Hi @zelot1982 ,

To ensure QAT is functioning correctly, first verify its availability by running lsmod | grep qat. If QAT is not listed, install the QAT driver properly and ensure the configuration is updated, such as changing [SSL] to [SHIM] in the relevant configuration file (e.g., /etc/...).

Additionally, confirm that all QAT software-dependent libraries are installed. To rule out potential issues and cross-verify, individually build and test both QAT hardware and software. If you still see error then you can enable debug logs in Nginx by setting error_log in the configuration file, restarting Nginx, and share the detailed logs of error.log

If my server doesn't have QAT devices installed, can't I use qat_sw for asynchronous acceleration

[hardikpatel9]

Hi @zelot1982 ,

Certainly, we can use QAT_SW without QAT hardware. My apologies for the earlier assumption that you were using QAT hardware as well. For QAT software, it should work fine. I have tested QAT software with a similar setup, and it worked successfully for me.

In my case, I did not use OpenSSL legacy mode in either Nginx or OpenSSL. During the OpenSSL compilation, I used the following configuration:./config --prefix=/usr/local/ssl -Wl,-rpath,/usr/local/ssl/lib
For testing, I used the OpenSSL client command: openssl s_client , If you are using a different client command, please share it so I can better understand your setup and assist you further.

Additionally, you may find it helpful to refer to the documentation provided in (docs/troubleshooting.md) for resolving common issues and docs/config_options.md for configuration options. These resources can provide valuable insights and guidance.

Please let me know if you need further clarification or assistance!