test: panic after TestPublisherStopShutdownInactive

[carsonip]

=== RUN   TestPublisherStop
    logger.go:146: 2026-07-07T10:12:48.194Z	DEBUG	beat.publish	start pipeline event consumer
    logger.go:146: 2026-07-07T10:12:48.194Z	DEBUG	beat.publish	pipeline event consumer queue reader: start
    logger.go:146: 2026-07-07T10:12:49.297Z	DEBUG	beat.publish.memqueue	ackloop: return ack to broker loop:3
    logger.go:146: 2026-07-07T10:12:49.297Z	DEBUG	beat.publish.memqueue	ackloop:  done send ack
    logger.go:146: 2026-07-07T10:12:49.297Z	DEBUG	beat.publish.memqueue	ackloop: return ack to broker loop:16
    logger.go:146: 2026-07-07T10:12:49.297Z	DEBUG	beat.publish.memqueue	ackloop:  done send ack
    logger.go:146: 2026-07-07T10:12:49.297Z	DEBUG	beat.publish.memqueue	ackloop: return ack to broker loop:16
    logger.go:146: 2026-07-07T10:12:49.297Z	DEBUG	beat.publish.memqueue	ackloop:  done send ack
    logger.go:146: 2026-07-07T10:12:49.297Z	DEBUG	beat.publish.memqueue	ackloop: return ack to broker loop:1
    logger.go:146: 2026-07-07T10:12:49.297Z	DEBUG	beat.publish.memqueue	ackloop:  done send ack
    logger.go:146: 2026-07-07T10:12:49.298Z	DEBUG	beat.publish.memqueue	ackloop: return ack to broker loop:2
    logger.go:146: 2026-07-07T10:12:49.298Z	DEBUG	beat.publish.memqueue	ackloop:  done send ack
    logger.go:146: 2026-07-07T10:12:49.298Z	DEBUG	beat.publish.memqueue	ackloop: return ack to broker loop:1
    logger.go:146: 2026-07-07T10:12:49.298Z	DEBUG	beat.publish.memqueue	ackloop:  done send ack
    logger.go:146: 2026-07-07T10:12:49.298Z	DEBUG	beat.publish	client: close queue producer
    logger.go:146: 2026-07-07T10:12:49.298Z	DEBUG	beat.publish	client: done producer close
    logger.go:146: 2026-07-07T10:12:49.298Z	DEBUG	beat.publish	close pipeline
    logger.go:146: 2026-07-07T10:12:49.298Z	INFO	beat.processOutputController	Output shutdown started. Waiting for enqueued events to be published.
    logger.go:146: 2026-07-07T10:12:49.298Z	INFO	beat.processOutputController	Continue shutdown: Time out waiting for events to be published.
    logger.go:146: 2026-07-07T10:12:49.298Z	DEBUG	beat.publish.memqueue	ackloop: return ack to broker loop:1
    logger.go:146: 2026-07-07T10:12:49.298Z	DEBUG	beat.publish.memqueue	ackloop:  done send ack
    logger.go:146: 2026-07-07T10:12:49.298Z	DEBUG	beat.publish	client: done closing acker
--- PASS: TestPublisherStop (1.11s)
=== RUN   TestPublisherStopShutdownInactive
    logger.go:146: 2026-07-07T10:12:49.299Z	DEBUG	beat.publish	start pipeline event consumer
    logger.go:146: 2026-07-07T10:12:49.299Z	DEBUG	beat.publish	pipeline event consumer queue reader: start
    logger.go:146: 2026-07-07T10:12:49.300Z	DEBUG	beat.publish	client: close queue producer
    logger.go:146: 2026-07-07T10:12:49.300Z	DEBUG	beat.publish	client: done producer close
    logger.go:146: 2026-07-07T10:12:49.300Z	DEBUG	beat.publish	close pipeline
    logger.go:146: 2026-07-07T10:12:49.300Z	INFO	beat.processOutputController	Output shutdown started. Waiting for enqueued events to be published.
    logger.go:146: 2026-07-07T10:12:49.300Z	DEBUG	beat.publish	client: done closing acker
    logger.go:146: 2026-07-07T10:12:49.300Z	INFO	beat.processOutputController	Continue shutdown: Time out waiting for events to be published.
--- PASS: TestPublisherStopShutdownInactive (0.01s)
PASS
panic: Log in goroutine after TestPublisherStopShutdownInactive has completed: 2026-07-07T10:12:49.303Z	DEBUG	beat.publish	pipeline event consumer queue reader: stop
	

goroutine 23 [running]:
testing.(*common).log(0xc00032a008, {0xc0003a61e0, 0x56})
	/opt/hostedtoolcache/go/1.26.4/x64/src/testing/testing.go:1039 +0x1df
testing.(*common).Logf(0xc00032a008, {0x1a95ce2, 0x2}, {0xc0003980d0, 0x1, 0x1})
	/opt/hostedtoolcache/go/1.26.4/x64/src/testing/testing.go:1200 +0x8f
go.uber.org/zap/zaptest.TestingWriter.Write({{0x7f26609b57e8?, 0xc00032a008?}, 0xc0?}, {0xc000390000, 0x57, 0x400})
	/home/runner/go/pkg/mod/go.uber.org/zap@v1.28.0/zaptest/logger.go:146 +0x11a
go.uber.org/zap/zapcore.(*ioCore).Write(0xc0002fb050, {0xff, {0xc28b52285219e293, 0x429862cf, 0x24a3140}, {0xc0002e6400, 0xc}, {0x1abdfa9, 0x2a}, {0x0, ...}, ...}, ...)
	/home/runner/go/pkg/mod/go.uber.org/zap@v1.28.0/zapcore/core.go:99 +0x202
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc00038e0e0, {0x0, 0x0, 0x0})
	/home/runner/go/pkg/mod/go.uber.org/zap@v1.28.0/zapcore/entry.go:276 +0x568
go.uber.org/zap.(*SugaredLogger).log(0xc0003040b8, 0xff, {0x0, 0x0}, {0xc000121f18, 0x1, 0x1}, {0x0, 0x0, 0x0})
	/home/runner/go/pkg/mod/go.uber.org/zap@v1.28.0/sugar.go:355 +0x12d
go.uber.org/zap.(*SugaredLogger).Debug(...)
	/home/runner/go/pkg/mod/go.uber.org/zap@v1.28.0/sugar.go:149
github.com/elastic/elastic-agent-libs/logp.(*Logger).Debug(...)
	/home/runner/go/pkg/mod/github.com/elastic/elastic-agent-libs@v0.46.0/logp/logger.go:197
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*queueReader).run(0xc0003063a0, 0xc0002e47c8)
	/home/runner/go/pkg/mod/github.com/elastic/beats/v7@v7.0.0-alpha2.0.20260706144519-3b40a86c8426/libbeat/publisher/pipeline/queue_reader.go:76 +0x356
created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer in goroutine 21
	/home/runner/go/pkg/mod/github.com/elastic/beats/v7@v7.0.0-alpha2.0.20260706144519-3b40a86c8426/libbeat/publisher/pipeline/consumer.go:102 +0x3bd
FAIL	github.com/elastic/apm-server/internal/publish	1.135s

https://github.com/elastic/apm-server/actions/runs/28858474254/job/85590811442?pr=21366

[aelnahas]

Started to look into this. I think based on my understanding from https://github.com/golang/go/blob/cfdaa16d0a4f08d9b0e9b342b94e41a8d02081c2/src/testing/testing.go#L1331 it seems that a routine maybe trying to log after the test is completed.

this might be the offending line: https://github.com/elastic/apm-server/blob/main/internal/publish/pub_test.go#L100 but I am going to look into this further

[aelnahas]

I came up with two ways to address this:

1. Work in beats to wait for queueReader to be done in https://github.com/elastic/beats/blob/50ef57e75fbd5b2923a308da39a27d1ce24185c3/libbeat/publisher/pipeline/consumer.go#L237
2. work around the issues in the tests . I chose this path to help unblock releases in this pr: avoid panics in TestPublisherStopShutdownInactive by using in memory logs #21640, but I am currently trying to evaluate a fix on libbeat

[aelnahas]

I will assess the changes that need to happen on beats to gracefully shutdown the queueReader.