Feature/global memories

docs/02-usage/040_workflow.md

@@ -103,22 +103,12 @@ By default, Serena will perform an **onboarding process** when
 it is started for the first time for a project.
 The goal of the onboarding is for Serena to get familiar with the project
 and to store memories, which it can then draw upon in future interactions.
-If an LLM should fail to complete the onboarding and does not actually write the
-respective memories to disk, you may need to ask it to do so explicitly.
-
-The onboarding will usually read a lot of content from the project, thus filling
-up the context. It can therefore be advisable to switch to another conversation
-once the onboarding is complete.
-After the onboarding, we recommend that you have a quick look at the memories and,
-if necessary, edit them or add additional ones.
-
-**Memories** are files stored in `.serena/memories/` in the project directory,
-which the agent can choose to read in subsequent interactions.
-Feel free to read and adjust them as needed; you can also add new ones manually.
-Every file in the `.serena/memories/` directory is a memory file.
-Whenever Serena starts working on a project, the list of memories is
-provided, and the agent can decide to read them.
-We found that memories can significantly improve the user experience with Serena.
+
+In general, **memories** provide a way for Serena to store and retrieve 
+information about the project, relevant conventions, and other relevant aspects.
+
+For more information on this, including how to manage
+or disable these features, see [Memories & Onboarding](045-memories).
 
 
 ## Preparing Your Project

docs/02-usage/045_memories.md

@@ -0,0 +1,84 @@
+# Memories & Onboarding
+
+Serena provides the functionality of a fully featured agent, and a useful aspect of this is Serena's memory system.
+Despite its simplicity, we received positive feedback from many users who tend to combine it with their
+agent's internal memory management (e.g., `AGENTS.md` files).
+
+## Memories
+
+Memories are simple, human-readable Markdown files that both you and
+your agent can create, read, and edit. 
+
+Serena differentiates between 
+  * **project-specific memories**, which are stored in the `.serena/memories/` directory within your project folder, and
+  * **global memories**, which are shared across all projects and, by default, are stored in `~/.serena/memories/global/`
+
+The LLM is informed about the existence of memories and instructed to read them when appropriate, 
+inferring appropriateness from the file name.
+When the agent starts working on a project, it receives the list of available memories. 
+The agent should be instructed to update memories by the user when appropriate.
+
+### Organizing Memories
+
+Memories can be organized into **topics** by using `/` in the memory name (e.g. `modules/frontend`).
+The structure is mapped to the file system, where topics correspond to subdirectories.
+The `list_memories` tool can filter by topic, allowing the agent to explore even large numbers of memories in a structured way.
+
+(global-memories)=
+### Global Memories
+
+Global memories use the top-level topic `global`, i.e. whenever a memory name starts with `global/`, 
+it is stored in the global memories directory and is shared across all projects.
+
+By default, deletion and editing of global memories is allowed.
+If you want to primarily manage such memories yourself and protect them from accidental modification by the agent,
+set `edit_global_memories: False` in Serena's [global configuration](050-configuration).
+
+Since global memories are not versioned alongside your project files,
+it can be helpful to track global memories with git (i.e. to make `~/.serena/memories/` a git repository)
+in order to have a history of changes and the possibility to revert them if needed.
+
+### Manually Editing Memories
+
+You may edit memories directly in the file system, using your preferred text editor or IDE.
+Alternatively, access them via the [Serena Dashboard](060_dashboard), which provides a graphical interface for
+viewing, creating, editing, and deleting memories while Serena is running.
+
+(onboarding)=
+## Onboarding
+
+By default, Serena performs an **onboarding process** when it encounters a project
+for the first time (i.e., when no project memories exist yet).
+The goal of the onboarding is for Serena to get familiar with the project —
+its structure, build system, testing setup, and other essential aspects —
+and to store this knowledge as memories for future interactions.
+
+In further project activations, Serena will check whether onboarding was already
+performed by looking for existing project memories and will skip the onboarding
+process if memories are found.
+
+### How Onboarding Works
+
+1. When a project is activated, Serena checks whether onboarding was already
+   performed (by checking if any memories exist).
+2. If no memories are found, Serena triggers the onboarding process, which
+   reads key files and directories to understand the project.
+3. The gathered information is written into project-specific memory files (see above).
+
+### Tips for Onboarding
+
+- **Context usage**: The onboarding process will read a lot of content from the project,
+  filling up the context window. It is therefore advisable to **switch to a new conversation**
+  once the onboarding is complete.
+- **LLM failures**: If an LLM fails to complete the onboarding and does not actually
+  write the respective memories to disk, you may need to ask it to do so explicitly.
+- **Review the results**: After onboarding, we recommend having a quick look at the
+  generated memories and editing them or adding new ones as needed.
+
+## Disabling Memories and Onboarding
+
+If you do not require the functionality described in this section, you can selectively disable it.
+
+ * To disable all memory related tools (including onboarding), adding `no-memories` to the `base_modes`
+   in Serena's [global configuration](050-configuration).
+ * Similarly, to disable only onboarding, add `no-onboarding` to the `base_modes`.

src/serena/agent.py

@@ -12,7 +12,7 @@
 
 from sensai.util import logging
 from sensai.util.logging import LogTime
-from sensai.util.string import TextBuilder
+from sensai.util.string import TextBuilder, list_string
 
 from interprompt.jinja_template import JinjaTemplate
 from serena import serena_version
@@ -29,7 +29,7 @@
 )
 from serena.dashboard import SerenaDashboardAPI
 from serena.ls_manager import LanguageServerManager, LanguageServerManagerInitialisationError
-from serena.project import Project
+from serena.project import MemoriesManager, Project
 from serena.prompt_factory import SerenaPromptFactory
 from serena.task_executor import TaskExecutor
 from serena.tools import ActivateProjectTool, GetCurrentConfigTool, OpenDashboardTool, ReplaceContentTool, Tool, ToolMarker, ToolRegistry
@@ -611,12 +611,15 @@ def _format_prompt(self, prompt_template: str) -> str:
     def create_system_prompt(self) -> str:
         available_tools = self._active_tools
         available_markers = available_tools.tool_marker_names
+        global_memory_names = MemoriesManager.list_global_memories()
+        global_memories_list = list_string(global_memory_names) if global_memory_names else ""
         log.info("Generating system prompt with available_tools=(see active tools), available_markers=%s", available_markers)
         system_prompt = self.prompt_factory.create_system_prompt(
             context_system_prompt=self._format_prompt(self._context.prompt),
             mode_system_prompts=[self._format_prompt(mode.prompt) for mode in self.get_active_modes()],
             available_tools=available_tools.tool_names,
             available_markers=available_markers,
+            global_memories_list=global_memories_list,
         )
 
         # If a project is active at startup, append its activation message

src/serena/config/serena_config.py

@@ -85,6 +85,12 @@ def __init__(self) -> None:
         """
         file containing the ID of the last read news snippet
         """
+        global_memories_path = Path(os.path.join(self.serena_user_home_dir, "memories", "global"))
+        global_memories_path.mkdir(parents=True, exist_ok=True)
+        self.global_memories_path = global_memories_path
+        """
+        directory where global memories are stored, i.e. memories that are available across all projects
+        """
         self.last_returned_log_file_path: str | None = None
         """
         the path to the last log file returned by `get_next_log_file_path`. If this is not None, the logs
@@ -549,6 +555,9 @@ class SerenaConfig(SharedConfig):
     """List of paths to ignore across all projects. Same syntax as gitignore, so you can use * and **.
     These patterns are merged additively with each project's own ignored_paths."""
 
+    edit_global_memories: bool = True
+    """Whether global memories are allowed to be deleted or edited."""
+
     # settings with overridden defaults
     default_modes: Sequence[str] | None = ("interactive", "editing")
     symbol_info_budget: float = 10.0
@@ -559,7 +568,6 @@ class SerenaConfig(SharedConfig):
     If the budget is exceeded, Serena stops issuing further requests and returns partial info results.
     0 disables the budget (no early stopping). Negative values are invalid.
     """
-
     # *** fields that are NOT mapped to/from the configuration file ***
 
     _loaded_commented_yaml: CommentedMap | None = None

src/serena/dashboard.py

@@ -89,6 +89,11 @@ class RequestDeleteMemory(BaseModel):
     memory_name: str
 
 
+class RequestRenameMemory(BaseModel):
+    old_name: str
+    new_name: str
+
+
 class ResponseGetSerenaConfig(BaseModel):
     content: str
 
@@ -269,6 +274,18 @@ def delete_memory() -> dict[str, str]:
             except Exception as e:
                 return {"status": "error", "message": str(e)}
 
+        @self._app.route("/rename_memory", methods=["POST"])
+        def rename_memory() -> dict[str, str]:
+            request_data = request.get_json()
+            if not request_data:
+                return {"status": "error", "message": "No data provided"}
+            request_rename_memory = RequestRenameMemory.model_validate(request_data)
+            try:
+                result_message = self._rename_memory(request_rename_memory)
+                return {"status": "success", "message": result_message}
+            except Exception as e:
+                return {"status": "error", "message": str(e)}
+
         @self._app.route("/get_serena_config", methods=["GET"])
         def get_serena_config() -> dict[str, Any]:
             try:
@@ -556,8 +573,7 @@ def run() -> None:
             project = self._agent.get_active_project()
             if project is None:
                 raise ValueError("No active project")
-
-            project.memories_manager.save_memory(request_save_memory.memory_name, request_save_memory.content)
+            project.memories_manager.save_memory(request_save_memory.memory_name, request_save_memory.content, is_tool_context=False)
 
         self._agent.execute_task(run, logged=True, name="SaveMemory")
 
@@ -566,11 +582,22 @@ def run() -> None:
             project = self._agent.get_active_project()
             if project is None:
                 raise ValueError("No active project")
-
-            project.memories_manager.delete_memory(request_delete_memory.memory_name)
+            project.memories_manager.delete_memory(request_delete_memory.memory_name, is_tool_context=False)
 
         self._agent.execute_task(run, logged=True, name="DeleteMemory")
 
+    def _rename_memory(self, request_rename_memory: RequestRenameMemory) -> str:
+        def run() -> str:
+            project = self._agent.get_active_project()
+            if project is None:
+                raise ValueError("No active project")
+
+            return project.memories_manager.move_memory(
+                request_rename_memory.old_name, request_rename_memory.new_name, is_tool_context=False
+            )
+
+        return self._agent.execute_task(run, logged=True, name="RenameMemory")
+
     def _get_serena_config(self) -> ResponseGetSerenaConfig:
         config_path = self._agent.serena_config.config_file_path
         if config_path is None or not os.path.exists(config_path):

src/serena/generated/generated_prompt_factory.py

@@ -33,6 +33,12 @@ def create_prepare_for_new_conversation(self) -> str:
         return self._render_prompt("prepare_for_new_conversation", locals())
 
     def create_system_prompt(
-        self, *, available_markers: Any, available_tools: Any, context_system_prompt: Any, mode_system_prompts: Any
+        self,
+        *,
+        available_markers: Any,
+        available_tools: Any,
+        context_system_prompt: Any,
+        global_memories_list: Any,
+        mode_system_prompts: Any,
     ) -> str:
         return self._render_prompt("system_prompt", locals())