Fix Armeria HTTP event loop sizing and document gRPC/HTTP thread models

Code change: fix shared event loop formula from max(5, cores/4) to
min(5, cores). The previous formula scaled poorly — cores/4 gives 1
on 2-core and 6 on 24-core, defeating the intent to cap at 5.
min(5, cores) gives 2→2, 4→4, 8+→5, matching the design goal.

Documentation: add comprehensive class-level javadoc to both
GRPCServer and HTTPServer explaining their thread models, why we keep
framework default executor pools on JDK <25 (extensions/handlers may
block on long I/O), and how virtual threads replace them on JDK 25+.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Author: wu-sheng

docs/en/changes/changes.md

@@ -56,7 +56,7 @@
   | L2 Persistence (OAL + MAL)            | 3 (DataCarrier)    | 4 (BatchQueue) | Unified OAL + MAL                           |
   | TopN Persistence                      | 4 (DataCarrier)    | 1 (BatchQueue) |                                             |
   | gRPC Remote Client                    | 1 (DataCarrier)    | 1 (BatchQueue) | Per peer                                    |
-  | Armeria HTTP event loop               | 20                 | 5 | `max(5, cores/4)` shared group              |
+  | Armeria HTTP event loop               | 20                 | 5 | `min(5, cores)` shared group                |
   | Armeria HTTP handler                  | on-demand platform(increasing with payload) | - | Virtual threads on JDK 25+                  |
   | gRPC event loop                       | 10                 | 10 | Unchanged                                   |
   | gRPC handler                          | on-demand platform(increasing with payload)| - | Virtual threads on JDK 25+                  |

oap-server/server-library/library-server/src/main/java/org/apache/skywalking/oap/server/library/server/grpc/GRPCServer.java

@@ -40,6 +40,82 @@
 import org.apache.skywalking.oap.server.library.server.pool.CustomThreadFactory;
 import org.apache.skywalking.oap.server.library.util.VirtualThreads;
 
+/**
+ * gRPC server backed by Netty. Used by up to 4 OAP server endpoints (core-grpc,
+ * receiver-grpc, ebpf-grpc, als-grpc). gRPC is the primary telemetry ingestion path.
+ *
+ * <h3>Thread model</h3>
+ * gRPC-netty uses a three-tier thread model:
+ * <ol>
+ *   <li><b>Boss event loop</b> — 1 thread. Accepts TCP connections, creates Netty channels,
+ *       then hands them off to worker event loop. Shared across all gRPC servers.</li>
+ *   <li><b>Worker event loop</b> — non-blocking I/O multiplexing (HTTP/2 framing, read/write,
+ *       TLS). gRPC defaults to {@code cores} threads (halves Netty's {@code cores * 2}),
+ *       shared across all servers via {@code SharedResourcePool}. Must never block —
+ *       a few threads can serve thousands of connections.</li>
+ *   <li><b>Application executor</b> — where gRPC service methods actually run
+ *       ({@code onMessage}, {@code onHalfClose}, {@code onComplete}). gRPC dispatches
+ *       callbacks from the event loop to this executor via
+ *       {@code JumpToApplicationThreadServerStreamListener}. For streaming RPCs, the
+ *       thread is held only during each individual callback, not for the entire stream —
+ *       between messages the thread returns to the pool.</li>
+ * </ol>
+ *
+ * <h3>Application executor</h3>
+ * gRPC's default application executor is an <b>unbounded {@code CachedThreadPool}</b>
+ * ({@code Executors.newCachedThreadPool()}, named {@code grpc-default-executor}).
+ * gRPC chose this for safety — application code may block (JDBC, file I/O, synchronized),
+ * and blocking the event loop would freeze all connections. The {@code CachedThreadPool}
+ * never rejects work but grows unboundedly: each burst creates new threads (expensive),
+ * idle threads die after 60s, then the next burst creates them again.
+ *
+ * <p>While benchmarks show {@code CachedThreadPool} is <b>2x slower</b> than a fixed pool
+ * (see <a href="https://github.com/grpc/grpc-java/issues/7381">grpc-java#7381</a>),
+ * we keep the default on JDK &lt;25 because SkyWalking extensions may register gRPC handlers
+ * that perform long-blocking I/O (on-demand queries, external calls). A bounded pool would
+ * risk starving other gRPC services. On JDK 25+, virtual threads replace this pool —
+ * each callback gets its own virtual thread, combining unbounded concurrency with
+ * minimal resource overhead.
+ *
+ * <p>Using {@code directExecutor()} is unsafe for SkyWalking because some handlers call
+ * {@code BatchQueue.produce()} with {@code BLOCKING} strategy which can block the thread
+ * — that would freeze the event loop and stall all connections.
+ *
+ * <h3>Thread policies</h3>
+ * <pre>
+ *                     gRPC default                 SkyWalking
+ *   Boss EL:          1, shared                    (unchanged)
+ *   Worker EL:        cores, shared                (unchanged)
+ *   App executor:     CachedThreadPool (unbounded) JDK 25+: virtual threads
+ *                                                   JDK &lt;25: gRPC default (unchanged)
+ * </pre>
+ *
+ * <h4>Worker event loop: {@code cores}, shared by gRPC (default, unchanged)</h4>
+ * <pre>
+ *   cores:    2    4    8   10   24
+ *   threads:  2    4    8   10   24
+ * </pre>
+ * Non-blocking I/O multiplexing — a few threads handle thousands of connections.
+ * gRPC's internal {@code SharedResourcePool} already shares one event loop group across
+ * all {@code NettyServerBuilder} instances that use the default. No custom configuration
+ * needed.
+ *
+ * <h3>Comparison with HTTP (Armeria)</h3>
+ * <pre>
+ *                     gRPC                                HTTP (Armeria)
+ *   Event loop:       cores, shared (gRPC default)        min(5, cores), shared
+ *   Handler/blocking: JDK 25+: virtual threads            JDK 25+: virtual threads
+ *                     JDK &lt;25: CachedThreadPool (default) JDK &lt;25: Armeria default cached pool
+ * </pre>
+ * Both gRPC and HTTP keep their framework's default unbounded pool on JDK &lt;25 because
+ * handlers may block on long I/O (storage queries, extension callbacks). On JDK 25+,
+ * virtual threads replace both pools.
+ *
+ * <h3>User-configured thread pool</h3>
+ * When {@code threadPoolSize > 0} is set via config, it overrides the default with a
+ * per-server fixed pool of that size. On JDK 25+ it is ignored — virtual threads
+ * are always used.
+ */
 @Slf4j
 public class GRPCServer implements Server {
 
@@ -91,6 +167,16 @@ public GRPCServer(String host, int port, String certChainFile, String privateKey
         this.trustedCAsFile = trustedCAsFile;
     }
 
+    /**
+     * Build the gRPC server with optional TLS and handler executor.
+     *
+     * <p>Handler executor assignment:
+     * <ul>
+     *   <li>JDK 25+: virtual-thread-per-task executor (ignores threadPoolSize)</li>
+     *   <li>JDK &lt;25, threadPoolSize &gt; 0: per-server fixed pool (legacy config)</li>
+     *   <li>JDK &lt;25, threadPoolSize == 0: gRPC default CachedThreadPool (unbounded)</li>
+     * </ul>
+     */
     @Override
     public void initialize() {
         InetSocketAddress address = new InetSocketAddress(host, port);
@@ -102,6 +188,9 @@ public void initialize() {
         if (maxMessageSize > 0) {
             nettyServerBuilder.maxInboundMessageSize(maxMessageSize);
         }
+        // JDK 25+: virtual threads for all servers (threadPoolSize ignored)
+        // JDK <25, threadPoolSize > 0: per-server fixed pool (legacy config override)
+        // JDK <25, threadPoolSize == 0: gRPC default CachedThreadPool (safe for extensions)
         final ExecutorService executor = VirtualThreads.createExecutor(
             threadPoolName,
             () -> {

oap-server/server-library/library-server/src/main/java/org/apache/skywalking/oap/server/library/server/http/HTTPServer.java

@@ -46,18 +46,77 @@
 
 import static java.util.Objects.requireNonNull;
 
+/**
+ * Armeria-based HTTP server shared by all OAP HTTP endpoints (core-http, receiver-http,
+ * promql-http, logql-http, zipkin-query-http, zipkin-http, firehose-http — up to 7 servers).
+ *
+ * <h3>Thread model</h3>
+ * Armeria uses a two-tier thread model:
+ * <ul>
+ *   <li><b>Event loop threads</b> — non-blocking I/O multiplexers (epoll/kqueue). Handle
+ *       connection accept, read/write, and protocol parsing. A few threads can serve
+ *       thousands of connections because they never block.</li>
+ *   <li><b>Blocking task executor threads</b> — where request handlers actually run when
+ *       annotated with {@code @Blocking}. These threads block on storage queries,
+ *       downstream calls, and computation. Each concurrent blocking request occupies
+ *       one thread for its full duration.</li>
+ * </ul>
+ *
+ * The blocking executor needs more threads than the event loop because it's where
+ * requests spend most of their time (waiting on I/O), while event loop threads just
+ * shuttle bytes and are immediately available for the next connection.
+ *
+ * <h3>Thread policies</h3>
+ * <pre>
+ *                     Armeria default        SkyWalking
+ *   Event loop:       cores * 2 per server   min(5, cores) shared across all servers
+ *   Blocking exec:    cached, up to 200      JDK 25+: virtual threads
+ *                                             JDK &lt;25: Armeria default (unchanged)
+ * </pre>
+ *
+ * <h4>Event loop: {@code min(5, cores)}, shared</h4>
+ * <pre>
+ *   cores:    2    4    8   10   24
+ *   threads:  2    4    5    5    5
+ * </pre>
+ * Armeria's default creates cores*2 event loop threads <em>per server</em>, which for 7
+ * HTTP servers means 7 * cores * 2 = 140 threads on 10-core — far more than needed for
+ * HTTP traffic. All servers share one {@link EventLoopGroup} with min(5, cores) threads.
+ *
+ * <h4>Blocking executor: Armeria default on JDK &lt;25, virtual threads on JDK 25+</h4>
+ * On JDK &lt;25, Armeria's default cached pool (up to 200 on-demand threads) is kept
+ * unchanged. HTTP handlers block on storage/DB queries (BanyanDB, Elasticsearch) which
+ * can take 10ms–seconds. A bounded pool would cause request queuing and UI timeouts
+ * when many concurrent queries block simultaneously. The cached pool handles this
+ * correctly — threads are created on demand and released after idle timeout.
+ * On JDK 25+, virtual threads replace this pool entirely — each blocking request
+ * gets its own virtual thread backed by ~cores shared carrier threads.
+ *
+ * <h3>Comparison with gRPC</h3>
+ * gRPC is the primary telemetry ingestion path. HTTP is secondary (UI queries, PromQL,
+ * LogQL, and optionally telemetry), so it uses fewer event loop threads.
+ * <pre>
+ *                     gRPC                                HTTP (Armeria)
+ *   Event loop:       cores, shared (gRPC default)        min(5, cores), shared
+ *   Handler/blocking: JDK 25+: virtual threads            JDK 25+: virtual threads
+ *                     JDK &lt;25: CachedThreadPool (default) JDK &lt;25: Armeria default cached pool
+ * </pre>
+ * Both gRPC and HTTP keep their framework's default unbounded pool on JDK &lt;25 because
+ * handlers may block on long I/O (storage queries, extension callbacks). On JDK 25+,
+ * virtual threads replace both pools.
+ */
 @Slf4j
 public class HTTPServer implements Server {
     /**
-     * Shared event loop group for all HTTP servers. HTTP traffic (UI queries,
-     * PromQL, LogQL) is much lighter than gRPC, so we use a smaller pool
-     * instead of Armeria's default (availableProcessors * 2).
+     * Shared event loop group for all HTTP servers.
+     * Non-blocking I/O multiplexing — min(5, cores) threads can handle thousands
+     * of connections. Replaces Armeria's default of cores*2 per server.
      */
     private static final EventLoopGroup SHARED_WORKER_GROUP;
 
     static {
-        final int threads = Math.max(5, Runtime.getRuntime().availableProcessors() / 4);
-        SHARED_WORKER_GROUP = EventLoopGroups.newEventLoopGroup(threads);
+        final int cores = Runtime.getRuntime().availableProcessors();
+        SHARED_WORKER_GROUP = EventLoopGroups.newEventLoopGroup(Math.min(5, cores));
     }
 
     private final HTTPServerConfig config;
@@ -74,6 +133,16 @@ public void setBlockingTaskName(final String blockingTaskName) {
         this.blockingTaskName = blockingTaskName;
     }
 
+    /**
+     * Build the Armeria server with shared event loop, TLS, and blocking executor.
+     *
+     * <p>Thread pool assignment:
+     * <ul>
+     *   <li>{@code workerGroup} — shared event loop for I/O (min(5, cores) threads)</li>
+     *   <li>{@code blockingTaskExecutor} — JDK 25+: virtual threads per request;
+     *       JDK &lt;25: Armeria's default cached pool (handlers block on storage queries)</li>
+     * </ul>
+     */
     @Override
     public void initialize() {
         sb = com.linecorp.armeria.server.Server
@@ -115,6 +184,9 @@ public void initialize() {
             sb.absoluteUriTransformer(this::transformAbsoluteURI);
         }
 
+        // JDK 25+: virtual-thread-per-task executor (unbounded, ~cores carrier threads)
+        // JDK <25: Armeria's default cached pool (up to 200 threads) — kept unchanged
+        //          because HTTP handlers block on long storage queries (10ms-seconds)
         if (VirtualThreads.isSupported()) {
             final ScheduledExecutorService blockingExecutor = VirtualThreads.createScheduledExecutor(
                 blockingTaskName, () -> null);