Running out of disk space on /tmp when running test suites

[ztamas83]

Hey!

I'm using embedded mongoDB for integration testing and when I run the full suite with a larger amount of tests they start to fail after some time.

Difficult to trace it back to flapdoodle but I found that it's the /tmp that runs out of space and most probably due to the repeated setup of new databases before actually cleaning up older runs. I can run the tests if I remount the tmp folder with more space.

My question to the community is if I can somehow tell flapdoodle not to create new DBs all the time but reuse the existing one but clear out the collections?

Caused by: com.mongodb.MongoCommandException: Command failed with error 14031 (OutOfDiskSpace): 'available disk space of 471977984 bytes is less than required minimum of 524288000' on server localhost:44145. The full response is {"ok": 0.0, "errmsg": "available disk space of 471977984 bytes is less than required minimum of 524288000", "code": 14031, "codeName": "OutOfDiskSpace", "$clusterTime": {"clusterTime": {"$timestamp": {"t": 1769092532, "i": 19}}, "signature": {"hash": {"$binary": {"base64": "AAAAAAAAAAAAAAAAAAAAAAAAAAA=", "subType": "00"}}, "keyId": 0}}, "operationTime": {"$timestamp": {"t": 1769092532, "i": 19}}}
	at com.mongodb.internal.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:210)
	at com.mongodb.internal.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:520)
	at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceiveInternal(InternalStreamConnection.java:448)
	at com.mongodb.internal.connection.InternalStreamConnection.lambda$sendAndReceive$0(InternalStreamConnection.java:375)
	at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:378)
	at com.mongodb.internal.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:111)
	at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:747)
	at com.mongodb.internal.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:61)
	at com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:208)
	at com.mongodb.internal.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:112)
	at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:82)
	at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:74)
	at com.mongodb.internal.connection.DefaultServer$OperationCountTrackingConnection.command(DefaultServer.java:298)
	at com.mongodb.internal.operation.SyncOperationHelper.lambda$executeCommand$5(SyncOperationHelper.java:209)
	at com.mongodb.internal.operation.SyncOperationHelper.lambda$withSourceAndConnection$0(SyncOperationHelper.java:131)
	at com.mongodb.internal.operation.SyncOperationHelper.withSuppliedResource(SyncOperationHelper.java:156)
	at com.mongodb.internal.operation.SyncOperationHelper.lambda$withSourceAndConnection$1(SyncOperationHelper.java:130)
	at com.mongodb.internal.operation.SyncOperationHelper.withSuppliedResource(SyncOperationHelper.java:156)
	at com.mongodb.internal.operation.SyncOperationHelper.withSourceAndConnection(SyncOperationHelper.java:129)
	at com.mongodb.internal.operation.SyncOperationHelper.executeCommand(SyncOperationHelper.java:207)
	at com.mongodb.internal.operation.CreateIndexesOperation.execute(CreateIndexesOperation.java:105)
	at com.mongodb.internal.operation.CreateIndexesOperation.execute(CreateIndexesOperation.java:60)
	at com.mongodb.client.internal.MongoClusterImpl$OperationExecutorImpl.execute(MongoClusterImpl.java:446)
	at com.mongodb.client.internal.MongoCollectionImpl.executeCreateIndexes(MongoCollectionImpl.java:941)
	at com.mongodb.client.internal.MongoCollectionImpl.createIndexes(MongoCollectionImpl.java:923)
	at com.mongodb.client.internal.MongoCollectionImpl.createIndexes(MongoCollectionImpl.java:918)
	at com.mongodb.client.internal.MongoCollectionImpl.createIndex(MongoCollectionImpl.java:903)
	at org.springframework.data.mongodb.core.DefaultIndexOperations.lambda$createIndex$0(DefaultIndexOperations.java:130)
	at org.springframework.data.mongodb.core.MongoTemplate.execute(MongoTemplate.java:616)

[michaelmosmann]

@ztamas83 hmm.. looks like mongodb needs at least 528mb disk space .. and your temp directory has less then that..

[michaelmosmann]

@ztamas83 i will have a look into some cleanup stuff so that all stuff is removed after an successful test.. maybe there i something missing..

[ztamas83]

Thanks for the response. As I reckon the cleanup is working after some time. I will actually check how many tests that run parallel, maybe that's a factor.
I have 2G on the /tmp so it's obvious that the 4th instance will fail then.

But what I'm curious about is if it was possible to reuse instances across tests somehow?

[michaelmosmann]

@ztamas83 You can do whatever you want, but to reuse instances across tests is somehow not how junit works.. out of the box. But i guess if i find the weak spots where the temp stuff is not cleaned, everything will be fine again.. a running mongodb instance does not use 512mb ad hoc.. its just a check that there is at least enough space.. so i think you can run many instances with a 2G /tmp if the cleanup works as it should.

stay tuned..

[michaelmosmann]

@ztamas83 there is a new release for de.flapdoodle.embed.mongo (4.24.0) can you try if this solves your problem?

[ztamas83]

I tested, but still the same behavior. The interesting part is that this has not happened for a few months back, something has changed. My project updates dependency versions constantly so it's difficult to find what could cause this. Can be OS, can be whatever. I understand the flapdoodle behavior and not blaming it, just wanted to check if there's anything more I could do other than increasing the /tmp size.

[michaelmosmann]

@ztamas83 what i did: i removed all test temp files and directories from /tmp .. then i picked one test and watched what is created in /tmp and why it was not removed after the finished test.. in the end, all stuff created to run a mongodb should be removed after that.. if you still find something, let me know..

[michaelmosmann]

I will make a new spring release soon, so you don't have to manual set this dependency..

[michaelmosmann]

@ztamas83 a new release of de.flapdoodle.embed.mongo.springXYZ is done.. i close this issue, reopen if you found something that i can fix:)