nginx cpu 100%

[weeweetan]

asynch_mode_nginx/src/event/ngx_event_openssl.c

Line 3662 in 4023ec5

//Work around: Readd write event on shutdown;

, What scenario is this solving? In the stream tls scenario, after nginx calls ssl_shutdown, the client does not respond, which will cause nginx cpu 100%

[mcdullbloom]

@weeweetan I meet the problem too. ngx_ssl_shutdown lead to single cpu core high usage.Is it the same?

[weeweetan]

@mcdullbloom yes, same problem, so I need the developer to confirm why these lines are added

[mcdullbloom]

how to reproduce?

[weeweetan]

@mcdullbloom after receiving the response, client calls the sleep function to block

[mcdullbloom]

I don't get it.What does the client should do after receive the response? Can you give more details?

[weeweetan]

ssl = SSL_new(ctx);
SSL_set_fd(ssl, sock);

if (SSL_connect(ssl) <= 0) {
    ERR_print_errors_fp(stderr);
} else {
    printf("Connected with %s encryption\n", SSL_get_cipher(ssl));
    SSL_write(ssl, "GET /HTTP/1.1\r\nHost: " HOSTNAME "\r\nConnection: close\r\n\r\n", strlen("GET / HTTP/1.1\r\nHost: " HOSTNAME "\r\nConnection: close\r\n\r\n"));

    char buffer[BUFFER_SIZE];
    int bytes;
    while ((bytes = SSL_read(ssl, buffer, sizeof(buffer))) > 0) {
        buffer[bytes] = 0;
        printf("%s", buffer);
    }
}
sleep(10000);
SSL_free(ssl);
close(sock);

implement a simple client like this

[hardikpatel9]

Hi @weeweetan ,
Tried the above client code with the latest version of the relevant required repo/Lib, I have tried both QAT- SW and QAT-HW with changes in the Nginx.conf with parameter changes but I am not able to reproduce the issue, Can you please give us more detail like about the client code by where we can run it and while executing it with async Nginx can get CPU 100% and in which scenario for reproduce it, with other configuration details.

[weeweetan]

Did you configure port 443 in the stream block? @hardikpatel9

stream {
    upstream backend {
        server xx.xx.xx.xx;
    }
    server{
        listen 443 ssl;
        ssl_certificate xx.crt;
        ssl_certificate_key xx.key;
        proxy_pass backend;
    }
}

[mcdullbloom]

@hardikpatel9 @weeweetan not only stream but also http.

Condition 1: asynch ssl

server {
    listen       443 ssl asynch;

Condition2: ssl_shutdown return -1

    for ( ;; ) {

        /*
         * For bidirectional shutdown, SSL_shutdown() needs to be called
         * twice: first call sends the "close notify" alert and returns 0,
         * second call waits for the peer's "close notify" alert.
         */

        n = SSL_shutdown(c->ssl->connection);
        n = -1;  // force n = -1  for mocking error
      ……

Condition 3: ssl_get_error return SSL_ERROR_WANT_READ

        /* before 0.9.8m SSL_shutdown() returned 0 instead of -1 on errors */

        sslerr = SSL_get_error(c->ssl->connection, n);
        sslerr = SSL_ERROR_WANT_READ;  // force  return  SSL_ERROR_WANT_READ
        ……

Condition 4: ssl client blocked and did not call ssl_shutdown

ssl = SSL_new(ctx);
SSL_set_fd(ssl, sock);

if (SSL_connect(ssl) <= 0) {
    ERR_print_errors_fp(stderr);
} else {
    printf("Connected with %s encryption\n", SSL_get_cipher(ssl));
    SSL_write(ssl, "GET /HTTP/1.1\r\nHost: " HOSTNAME "\r\nConnection: close\r\n\r\n", strlen("GET / HTTP/1.1\r\nHost: " HOSTNAME "\r\nConnection: close\r\n\r\n"));

    char buffer[BUFFER_SIZE];
    int bytes;
    while ((bytes = SSL_read(ssl, buffer, sizeof(buffer))) > 0) {
        buffer[bytes] = 0;
        printf("%s", buffer);
    }
}
sleep(10000);   // should  be blocked 
SSL_free(ssl);
close(sock);

[hardikpatel9]

We have fixed this issue. We will include it in the next release, which will be very soon in the next Async Nginx release

[hardikpatel9]

Hi @weeweetan ,
We have released the new version of the async Nginx: - Async Nginx v0.5.2