Multiple Critical Failures in AI Town - Timeouts, Generation Mismatches, and Conversation Loops

[ashmgdesign]

Log

Issue: Multiple Critical Failures in AI Town - Timeouts, Generation Mismatches, and Conversation Loops

Summary

The AI Town system is experiencing multiple critical failures that prevent normal operation, including function timeouts, generation number mismatches, dead engine restarts, and agents stuck in infinite conversation exit loops.

Issues Identified

1. Function Execution Timeouts

• Multiple agentGenerateMessage operations are timing out at the 600-second maximum duration
• Example: Function execution timed out (maximum duration: 600s)
• Affects conversation flow and agent interactions

2. Generation Number Mismatch Errors

• Repeated ConvexError: {"kind":"generationNumber","message":"Generation number mismatch"} errors
• Occurs during loadWorld and saveWorld operations
• Suggests synchronization issues between different parts of the system

3. Dead Engine Continuous Restarts

• Engine ks78teh5gnmjpbf1cd60m3wehx7hwm8b is repeatedly dying and being restarted
• Log shows: 'Restarting dead engine ks78teh5gnmjpbf1cd60m3wehx7hwm8b...' appearing every minute
• Indicates underlying stability issues with the game engine

4. Conversation Exit Loop

• AI agents (Alice/Pete, Bob/Kurt, Lucky/Stella) are stuck in infinite loops trying to leave conversations
• Agents repeatedly exchange farewell messages but never successfully exit
• Example: The conversation history shows 20+ farewell messages between the same agents

5. Unawaited Async Operations

• Warning: 1 unawaited operation: [fetch]. Async operations should be awaited or they might not run
• Could lead to incomplete operations and data inconsistency

6. Performance Issues

• Even successful operations take 500-580 seconds to complete
• Example: Function execution took a long time. (maximum duration: 600s, actual duration: 578.114142708s)

Expected Behavior

• Agents should be able to complete conversations and exit gracefully
• Function executions should complete within reasonable time limits
• The engine should remain stable without constant restarts
• Generation numbers should remain synchronized

Actual Behavior

• Agents get stuck in conversation exit loops
• Functions timeout after 600 seconds
• Engine crashes and restarts continuously
• Generation number mismatches prevent proper world state saves

Environment

• Convex backend
• Model: dolphin3:latest
• Multiple concurrent agent operations

Potential Root Causes

1. Inefficient message generation causing timeouts
2. State synchronization issues between different system components
3. Memory leaks or resource exhaustion causing engine crashes
4. Improper async operation handling
5. Conversation state management not properly tracking exit conditions

Suggested Fixes

1. Implement proper conversation state management to prevent exit loops
2. Optimize agent message generation to reduce execution time
3. Fix async operation handling to ensure all operations are properly awaited
4. Investigate and fix the root cause of engine crashes
5. Implement better synchronization for generation numbers
6. Add timeout handling and retry logic for long-running operations

Severity

Critical - The system is essentially non-functional with agents unable to properly interact and the engine constantly crashing.

[ianmacartney]

When you say "Convex backend" as the environment - is this on Convex cloud, self-hosted, or locally hosted? Some notes to self / whoever undertakes a refactor: - I think this repo could do with a rewrite using a few components that will drastically simplify and likely solve the noted issues. It should use the Agent & Workpool components if possible - It should store the users & game state denormalized vs all in a single game state blob. That also enables real IDs, not just the string/number hack - It should use the action cache to cache commonly used embeddings - It could hopefully generate route plans instead of animation frames, so the client can replay and not re-fetch every frame set every second & potentially miss a batch. - It should stop loading & passing around the full map, in favor of an efficient static representation of the boundaries for the default world - dynamic worlds should be opt-in not the default.

…

On Tue, Jun 17, 2025 at 12:58 PM ashmgdesign ***@***.***> wrote: *ashmgdesign* created an issue (a16z-infra/ai-town#284) <#284> Log <https://pastebin.com/wdFn7XLC> Issue: Multiple Critical Failures in AI Town - Timeouts, Generation Mismatches, and Conversation Loops Summary The AI Town system is experiencing multiple critical failures that prevent normal operation, including function timeouts, generation number mismatches, dead engine restarts, and agents stuck in infinite conversation exit loops. Issues Identified 1. *Function Execution Timeouts* - Multiple agentGenerateMessage operations are timing out at the 600-second maximum duration - Example: Function execution timed out (maximum duration: 600s) - Affects conversation flow and agent interactions 2. *Generation Number Mismatch Errors* - Repeated ConvexError: {"kind":"generationNumber","message":"Generation number mismatch"} errors - Occurs during loadWorld and saveWorld operations - Suggests synchronization issues between different parts of the system 3. *Dead Engine Continuous Restarts* - Engine ks78teh5gnmjpbf1cd60m3wehx7hwm8b is repeatedly dying and being restarted - Log shows: 'Restarting dead engine ks78teh5gnmjpbf1cd60m3wehx7hwm8b...' appearing every minute - Indicates underlying stability issues with the game engine 4. *Conversation Exit Loop* - AI agents (Alice/Pete, Bob/Kurt, Lucky/Stella) are stuck in infinite loops trying to leave conversations - Agents repeatedly exchange farewell messages but never successfully exit - Example: The conversation history shows 20+ farewell messages between the same agents 5. *Unawaited Async Operations* - Warning: 1 unawaited operation: [fetch]. Async operations should be awaited or they might not run - Could lead to incomplete operations and data inconsistency 6. *Performance Issues* - Even successful operations take 500-580 seconds to complete - Example: Function execution took a long time. (maximum duration: 600s, actual duration: 578.114142708s) Expected Behavior - Agents should be able to complete conversations and exit gracefully - Function executions should complete within reasonable time limits - The engine should remain stable without constant restarts - Generation numbers should remain synchronized Actual Behavior - Agents get stuck in conversation exit loops - Functions timeout after 600 seconds - Engine crashes and restarts continuously - Generation number mismatches prevent proper world state saves Environment - Convex backend - Model: dolphin3:latest - Multiple concurrent agent operations Potential Root Causes 1. Inefficient message generation causing timeouts 2. State synchronization issues between different system components 3. Memory leaks or resource exhaustion causing engine crashes 4. Improper async operation handling 5. Conversation state management not properly tracking exit conditions Suggested Fixes 1. Implement proper conversation state management to prevent exit loops 2. Optimize agent message generation to reduce execution time 3. Fix async operation handling to ensure all operations are properly awaited 4. Investigate and fix the root cause of engine crashes 5. Implement better synchronization for generation numbers 6. Add timeout handling and retry logic for long-running operations Severity *Critical* - The system is essentially non-functional with agents unable to properly interact and the engine constantly crashing. — Reply to this email directly, view it on GitHub <#284>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACZQWZITHZE32S3UGM6VQT3EBXN7AVCNFSM6AAAAAB7RELFP2VHI2DSMVQWIX3LMV43ASLTON2WKOZTGE2TINJVHE2DMMI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>