From a57caf0e73cc56e0a4a6196e9cb1596902938431 Mon Sep 17 00:00:00 2001 From: withchao <993506633@qq.com> Date: Fri, 12 Jun 2026 11:24:23 +0800 Subject: [PATCH] feat: enhance S3 object storage documentation with detailed upload mechanisms and benefits (cherry picked from commit 5b2c3b067136308d10f577a0c957e43759411d86) --- docs/blog/golang/architectural/3.md | 189 +++++++++++---- .../current/golang/architectural/3.md | 228 ++++++++++++++---- 2 files changed, 319 insertions(+), 98 deletions(-) diff --git a/docs/blog/golang/architectural/3.md b/docs/blog/golang/architectural/3.md index ab259dad02..277862bfff 100644 --- a/docs/blog/golang/architectural/3.md +++ b/docs/blog/golang/architectural/3.md @@ -4,121 +4,212 @@ hide_title: true sidebar_position: 3 --- +## 1. 背景 -# 基于 Go 与 S3 的高效文件上传方案 —— 支持断点续传与秒传 +在即时通信系统里，图片、语音、视频、文件、视频封面等内容都属于对象数据。它们和普通文本消息不同，通常体积更大、上传时间更长、受网络波动影响更明显。如果所有文件都经由业务 API 服务中转，服务端会同时承担业务处理和大流量文件搬运，容易影响消息、会话、群组等核心链路的稳定性。 -在现代 App 与 Web 应用中，文件上传是常见且关键的功能。为应对传输中可能出现的中断、重复上传等问题，我们设计并实现了一套基于 **Go 服务端 + S3 存储** 的文件上传机制。该方案支持**断点续传** 与**秒传** ，并具备良好的**可扩展性** 和**极简客户端 SDK** ，实现客户端与 S3 直连，**数据流不经 API 服务中转** 。 +OpenIM 的 S3 对象存储能力，就是为了解决这个问题：让业务服务只负责鉴权、签名、元数据和访问控制，让文件数据直接进入对象存储。这样既能提高上传效率，也能降低 API 服务压力。 +## 2. 核心定位 -## ✨ 设计目标 +OpenIM 的 S3 方案不是简单地把文件上传到 MinIO，而是一套面向 IM 场景的大对象管理能力。它覆盖上传、下载、访问、续传、去重、清理、迁移等环节。 +整体上可以理解为两层： -- ✅ 极简 SDK：客户端无需存储系统适配逻辑 +- 服务端负责控制面：鉴权、生成临时凭证、登记对象信息、生成统一访问地址、执行过期清理。 +- SDK 负责客户端上传体验：读取文件、计算哈希、分片上传、断点续传、并发控制、进度回调、取消上传。 -- ✅ 降低服务器负载：文件流量不经由 API 服务 +这种拆分让“业务控制”和“文件流量”分离。服务端不再搬运大文件，客户端拿到授权后直接上传到对象存储。 -- ✅ 支持断点续传与秒传，提升用户体验 +## 3. 整体上传体验 -- ✅ 兼容任意 S3 接口，易于接入多种对象存储 +用户发送图片、视频或文件时，SDK 会先在本地分析文件，判断文件大小和类型，并计算文件内容特征。然后 SDK 向服务端申请一次上传授权。服务端校验用户身份和对象归属后，返回临时上传凭证。之后文件内容由客户端直接传到对象存储，不再经过 OpenIM API 服务中转。 -- ✅ 可拓展支持非 S3 的 HTTP PUT 分片上传 +上传完成后，服务端会登记这份对象和业务对象名之间的关系，并返回一个 OpenIM 层面的访问地址。业务消息里保存的是这个稳定地址，而不是某个存储厂商暴露的原始地址。后续访问时，OpenIM 再根据对象元数据生成临时访问链接并重定向到真实对象。 +这个链路的好处是： +- 上传大文件时不会挤占业务 API 带宽。 +- 下载地址可以统一由 OpenIM 管理。 +- 底层存储后端变化时，业务层地址可以保持稳定。 +- 私有桶也可以通过临时授权安全访问。 -## 📦 上传流程 +## 4. 多存储后端兼容 +OpenIM 支持多种 S3 或 S3 兼容对象存储，包括 MinIO、腾讯云 COS、阿里云 OSS、七牛 Kodo、AWS S3 等。业务层不需要感知具体厂商差异，只需要使用 OpenIM 提供的统一对象能力。 -上传流程共分为五步： +这带来的优势是部署选择更灵活： +- 私有化部署可以使用 MinIO。 +- 公有云部署可以接入云厂商对象存储。 +- 企业后续更换存储厂商时，业务逻辑不需要整体重写。 +- 存储访问域名、内外网地址、权限策略可以按部署环境调整。 -### 1. 获取分片大小 +## 5. 客户端直传 -接口：`GET /object/part_limit` +传统文件上传常见做法是客户端先把文件传给业务服务，业务服务再转存到对象存储。这种方式实现简单，但在大文件和高并发场景下成本很高。 -客户端首次上传前，请求服务端获取推荐的分片大小（如 5MB），并进行缓存。 +OpenIM 采用客户端直传：服务端只签发短期授权，文件本体由客户端直接上传到对象存储。这样 API 服务只处理轻量请求，不承担大文件传输。 +客户端直传的优势包括： -### 2. 计算文件哈希 +- 降低 API 服务带宽和连接压力。 +- 大文件上传更稳定，不影响消息收发等核心接口。 +- 对象存储天然适合承载大对象和高并发下载。 +- 文件上传链路更短，整体吞吐能力更好。 -客户端将文件按分片大小切分，并计算每个分片的哈希值。然后将所有分片哈希拼接，最终计算出文件整体哈希（`file_hash`），用于秒传判断。 +## 6. 分片上传 +对于较大的图片、视频和文件，SDK 会使用分片上传。文件被拆成多个小块，每个分片独立上传，最后由对象存储合并成完整对象。 -### 3. 初始化上传 +分片上传带来的体验提升很明显： -接口：`POST /object/initiate_multipart_upload` +- 大文件不需要一次性完整上传。 +- 某个分片失败时，只需要重传失败部分。 +- 可以通过受控并发提升上传速度。 +- 上传过程更容易展示细粒度进度。 -客户端提交文件整体信息（如 hash、文件名、大小）： +SDK 会根据文件大小和服务端限制自动选择合适的分片大小，上层业务不需要手动处理这些细节。 +## 7. 秒传 -- 若服务器发现该文件已存在（通过 `file_hash` 判断），则返回秒传成功的下载地址； +OpenIM 支持基于文件内容的秒传。SDK 会计算文件内容特征，服务端可以判断相同内容是否已经存在。如果对象存储里已经有这份文件，就不再重复上传文件内容，而是直接登记一次新的业务引用并返回访问地址。 -- 若不存在，则返回每个分片的上传签名信息，包括： +秒传的价值主要体现在重复发送场景： +- 同一用户重复发送相同文件时，可以瞬间完成。 +- 不同消息引用同一份真实对象，减少重复存储。 +- 降低网络传输成本和对象存储写入成本。 +- 弱网环境下减少不必要的重传。 - - 分片上传 URL +秒传不是按文件名判断，而是按内容特征判断，因此更适合 IM 文件场景。 - - 必需的 Header（如 `Content-Type`, `Authorization`） +## 8. 断点续传 - - `PartNumber` 等标识 +SDK 会在本地记录上传进度。上传过程中，如果网络中断、应用退出、进程重启，已经成功上传的分片不会丢失。用户再次上传同一个文件时，SDK 可以从本地恢复状态，只补传尚未完成的分片。 +断点续传能显著改善弱网和移动端体验： -### 4. 上传分片（直传 S3） +- 大文件失败后不用从头开始。 +- 切后台、断网、重启后仍能继续。 +- 每个已成功的分片都能沉淀为有效进度。 +- 用户重试成本更低，失败体验更可控。 +这也是 OpenIM S3 方案相比普通表单上传的重要优势之一。 -客户端使用返回的签名信息，将每个分片通过 HTTP PUT 直接上传至 S3，对应： +## 9. 并发与内存控制 +SDK 支持分片并发上传，但不会无节制地并发。上传时会根据分片大小和内存预算动态调整并发数量，避免为了追求速度导致客户端内存过高。 -- 上传过程中记录每一分片的 ETag 与 PartNumber +当分片较小时，SDK 可以并发上传多个分片；当分片较大时，SDK 会降低并发；如果分片过大，还会退回更稳妥的流式上传方式。 -- SDK 持久化这些信息，实现断点续传 +这种策略在移动端、桌面端、Web/WASM 等环境中都更稳。它兼顾了上传速度、内存占用和设备稳定性。 +## 10. 进度反馈与取消上传 +SDK 会把上传过程拆成多个可感知阶段，例如打开文件、计算哈希、获得上传会话、上传分片、完成合并等。上层 UI 可以据此展示更准确的进度，而不是只有一个模糊的百分比。 -> ⚠️ 文件内容始终不经由服务端，仅签名和元数据走 API。 +对于大文件来说，这一点很重要。用户可以清楚知道当前是在本地计算、网络上传，还是等待服务端完成合并。 +SDK 也支持取消上传。业务侧可以为上传任务绑定取消标识，用户点击取消后，SDK 会中止对应上传任务，避免继续消耗网络和设备资源。 -### 5. 完成上传 +## 11. 统一访问地址 -接口：`POST /object/complete_multipart_upload` +OpenIM 上传完成后返回的是 OpenIM 统一访问地址，而不是直接暴露对象存储原始地址。访问文件时，OpenIM 会根据对象元数据生成临时访问链接，再跳转到真实存储地址。 -客户端上传所有分片成功后，调用此接口提交分片信息列表。 +这样设计有几个好处： +- 消息里保存的文件地址更稳定。 +- 底层存储迁移不会直接影响历史消息。 +- 私有对象也可以通过临时授权安全访问。 +- 可以统一处理文件名、内容类型、图片缩略图等访问策略。 -- 服务端校验每个分片哈希及整体 hash +对于图片类对象，访问层还可以支持缩略图、格式转换、宽高裁剪等派生能力，从而提升聊天图片加载效率。 -- 校验通过后，发起 S3 的合并操作 +## 12. 权限与隔离 -- 最终返回文件的访问地址（可跳转到签名 S3 URL） +普通用户上传对象时，对象会归入当前用户自己的命名空间。这样可以避免用户伪造或覆盖其他用户的文件对象。服务端在生成上传授权前会进行身份校验和对象归属校验，确保上传行为受控。 +对象存储的长期密钥不会下发给客户端。客户端拿到的只是短期上传或访问凭证，即使链接泄露，风险也会被限制在较短时间和特定对象范围内。 -## ⬇️ 下载流程 +这种方式兼顾了直传效率和安全边界。 +## 13. 元数据治理 -客户端下载时，从服务端获取临时签名下载地址（S3 Pre-signed URL），可设置权限与有效期控制。该地址可通过 API 统一跳转实现权限校验。 +对象存储只负责保存文件内容，但 IM 系统还需要知道这份文件属于哪个用户、哪个业务类型、使用哪个存储后端、什么时候创建、是否可以清理等信息。 +OpenIM 会为每个业务对象登记元数据。元数据把“业务对象名”和“真实存储对象”分离开来。多个业务对象可以引用同一个真实对象，这也是秒传和去重能够成立的基础。 +元数据治理带来的能力包括： -## ✅ 特性一览 +- 统一生成访问地址。 +- 支持内容去重和秒传。 +- 支持按业务类型做生命周期清理。 +- 支持统计真实对象是否仍被引用。 +- 支持未来迁移底层存储。 -| 特性 | 说明 | -| --- | --- | -| 秒传支持 | 基于文件 hash 实现上传秒跳过 | -| 断点续传 | 支持断线、异常退出后继续上传 | -| 极简 SDK | 客户端无需签名逻辑，仅执行 PUT | -| 存储兼容性强 | 支持任意兼容 HTTP PUT 的对象存储 | -| 高可拓展性 | 可快速接入其他上传协议和存储系统 | +## 14. 生命周期清理 +IM 系统里的文件对象并不都需要永久保存。比如图片、语音、视频、视频封面等消息对象，可以根据业务策略设置保留时间。 +OpenIM 支持按业务分类清理过期对象。清理时不会简单粗暴地删除真实文件，而是先删除过期业务引用，再判断真实对象是否仍被其他业务对象引用。只有没有任何引用时，才会删除对象存储里的真实文件。 -## ⚙️ 可扩展性设计 +这种引用计数式清理可以避免误删。它和秒传能力天然配套：同一份真实文件可能被多个消息引用，只有所有引用都过期后，真实对象才应该被删除。 +## 15. FormData 小程序兼容通道 -该方案以“服务端统一生成签名 + 客户端通用 PUT 上传”为核心架构，只要目标存储系统支持分片上传并接受带签名的 PUT 请求，即可无缝接入： +除了完整的 SDK 分片上传链路，OpenIM 还保留了 FormData 表单直传能力。这条链路主要用于小程序 JSSDK，用来适配小程序平台的文件上传能力和调用限制。 +需要特别说明的是，Go 语言提供的 SDK 不会调用 FormData。Go SDK 使用的是完整的分片上传链路。 -- 支持 Amazon S3、MinIO、阿里云 OSS、腾讯云 COS 等 +FormData 是兼容通道，不具备完整上传链路的高级能力。它不支持： -- 也可扩展支持其它上传服务 +- 分片上传。 +- 秒传。 +- 本地断点续传。 +- 分片并发。 +- 分片级校验和分片级进度回调。 +因此，普通 App、桌面端、Go SDK、Web/WASM 端应优先使用完整的 SDK 上传能力；小程序 JSSDK 在平台限制下可以使用 FormData 作为简化直传方案。 -客户端代码不变，服务端扩展签名生成逻辑即可。 +## 16. 存储迁移能力 + +由于 OpenIM 在元数据中记录了对象所属的存储后端，系统具备从一个对象存储迁移到另一个对象存储的基础能力。例如从自建 MinIO 迁移到云厂商对象存储，或者在不同云厂商之间切换。 + +迁移时，系统可以读取旧存储中的真实对象，写入新存储，并更新元数据中的存储后端信息。业务消息里保存的是 OpenIM 统一访问地址，因此迁移后历史消息仍然可以通过原有方式访问。 + +这降低了长期运维风险，也减少了被单一存储厂商或部署形态绑定的成本。 + +## 17. 相比传统上传方式的优势 + +OpenIM S3 方案的优势可以归纳为五类。 + +性能方面，文件流量直达对象存储，API 服务只处理控制请求，大文件上传不会挤占核心业务接口。 + +稳定性方面，分片上传、断点续传、分片校验、取消上传和受控并发共同提升了弱网环境下的成功率。 + +成本方面，秒传和去重减少重复上传与重复存储，生命周期清理减少无效对象长期占用空间。 + +安全方面，客户端只拿短期授权，不接触对象存储长期密钥；文件访问也可以通过临时链接进行控制。 + +演进方面，统一对象地址和元数据治理让底层存储可以迁移或替换，业务层不需要感知具体厂商变化。 + +## 18. 适用场景 + +OpenIM S3 对象存储适合以下场景： + +- 聊天图片、原图、缩略图。 +- 语音消息。 +- 视频消息和视频封面。 +- 普通文件消息。 +- ASR 等需要临时对象上传的能力。 +- 私有化部署中的文件存储。 +- 大文件、弱网、移动端上传场景。 + +如果业务只需要小程序端简单表单上传，可以使用 FormData 兼容通道；如果需要秒传、分片、断点续传和更完整的上传体验，应使用 SDK 的完整上传能力。 + +## 19. 总结 + +OpenIM 的 S3 对象存储能力，是一套面向 IM 大对象场景的文件基础设施。它通过客户端直传降低服务端压力，通过分片和断点续传提升弱网体验，通过秒传和生命周期清理降低成本，通过统一访问地址和元数据治理提升长期可维护性。 + +它的核心价值，是把文件上传从“业务 API 的沉重负担”变成“可授权、可续传、可去重、可治理、可迁移的对象服务能力”。对于企业级 IM，这种设计能同时提升性能、稳定性、安全性、成本控制和后续演进能力。 diff --git a/i18n/en/docusaurus-plugin-content-docs-blog/current/golang/architectural/3.md b/i18n/en/docusaurus-plugin-content-docs-blog/current/golang/architectural/3.md index 4049841508..c172e9b097 100644 --- a/i18n/en/docusaurus-plugin-content-docs-blog/current/golang/architectural/3.md +++ b/i18n/en/docusaurus-plugin-content-docs-blog/current/golang/architectural/3.md @@ -4,82 +4,212 @@ hide_title: true sidebar_position: 3 --- -# Efficient File Upload Solution Based on Go and S3 - Supports Resumable Upload and Instant Upload +## 1. Background -In modern App and Web applications, file upload is a common and critical feature. In order to deal with interruptions, repeated uploads, and other problems that may occur during transmission, we designed and implemented a file upload mechanism based on **Go Backend + S3 Storage**. This solution supports **resumable upload** and **instant upload**, and features good **scalability** and an **extremely simple client SDK**, enabling direct connection between the client and S3, with **the data stream not transitioning through the API service**. +In an instant messaging system, images, voice messages, videos, files, video covers, and similar content are all object data. They are different from normal text messages: they are usually larger, take longer to upload, and are more sensitive to network fluctuations. If every file is relayed through the business API service, the server has to handle both business processing and high-volume file transfer, which can easily affect the stability of core paths such as messages, conversations, and groups. -## ✨ Design Goals +OpenIM's S3 object storage capability is designed to solve this problem: business services are responsible only for authentication, signatures, metadata, and access control, while file data goes directly into object storage. This improves upload efficiency and reduces pressure on API services. -- ✅ Extremely simple SDK: Clients do not need to adapt to storage system logic. -- ✅ Reduce server load: File traffic does not go through the API service. -- ✅ Support resumable upload and instant upload to improve user experience. -- ✅ Compatible with any S3 interface, easy to access various object storages. -- ✅ Scalable to support non-S3 HTTP PUT multipart uploads. +## 2. Core Positioning -## 📦 Upload Process +OpenIM's S3 solution is not simply about uploading files to MinIO. It is a large-object management capability built for IM scenarios. It covers upload, download, access, resumable upload, deduplication, cleanup, migration, and related stages. -The upload process is divided into five steps: +At a high level, it can be understood as two layers: -### 1. Request Part Size limits +- The server is responsible for the control plane: authentication, temporary credential generation, object information registration, unified access address generation, and expired-object cleanup. +- The SDK is responsible for the client upload experience: reading files, calculating hashes, multipart upload, resumable upload, concurrency control, progress callbacks, and upload cancellation. -Interface: `GET /object/part_limit` +This split separates "business control" from "file traffic". The server no longer moves large files; after receiving authorization, the client uploads directly to object storage. -Before the client uploads for the first time, it requests the recommended part size (such as 5MB) from the backend and caches it. +## 3. Overall Upload Experience -### 2. Calculate File Hash +When a user sends an image, video, or file, the SDK first analyzes the file locally, determines its size and type, and calculates content features for the file. The SDK then requests upload authorization from the server. After the server verifies the user's identity and the object's ownership, it returns temporary upload credentials. The file content is then uploaded directly from the client to object storage, without being relayed through the OpenIM API service. -The client splits the file according to the part size, and calculates the hash value of each part. Then it concatenates all part hashes, and finally calculates the overall file hash (`file_hash`), used for instant upload judgment. +After the upload is complete, the server registers the relationship between this object and the business object name, and returns an OpenIM-level access address. Business messages store this stable address, not the raw address exposed by a storage vendor. During later access, OpenIM generates a temporary access link based on object metadata and redirects to the real object. -### 3. Initialize Upload +This flow has several benefits: -Interface: `POST /object/initiate_multipart_upload` +- Uploading large files does not consume business API bandwidth. +- Download addresses can be managed centrally by OpenIM. +- Business-layer addresses can remain stable when the underlying storage backend changes. +- Private buckets can also be accessed securely through temporary authorization. -The client submits the overall file information (such as hash, filename, size): +## 4. Compatibility With Multiple Storage Backends -- If the server finds that the file already exists (judged by `file_hash`), it returns the download address indicating successful instant upload. -- If it does not exist, it returns the upload signature information for each part, including: - - Part upload URL - - Required Headers (such as `Content-Type`, `Authorization`) - - Identifiers such as `PartNumber` +OpenIM supports multiple S3 or S3-compatible object storage services, including MinIO, Tencent Cloud COS, Alibaba Cloud OSS, Qiniu Kodo, AWS S3, and others. The business layer does not need to understand vendor-specific differences; it only needs to use the unified object capability provided by OpenIM. -### 4. Upload Part (Direct Upload to S3) +This makes deployment choices more flexible: -Using the returned signature information, the client directly uploads each part to S3 via HTTP PUT: +- Private deployments can use MinIO. +- Public cloud deployments can connect to cloud-vendor object storage. +- If an enterprise later switches storage vendors, the business logic does not need to be rewritten as a whole. +- Storage access domains, internal and external network addresses, and permission policies can be adjusted for each deployment environment. -- Record the ETag and PartNumber of each part during the upload process. -- The SDK persists this information to achieve resumable uploads. +## 5. Direct Client Upload -> ⚠️ File content never goes through the backend; only signatures and metadata go through the API. +A common approach for traditional file upload is for the client to upload the file to the business service first, and then for the business service to transfer it to object storage. This is simple to implement, but it is costly for large files and high concurrency. -### 5. Complete Upload +OpenIM uses direct client upload: the server only issues short-lived authorization, and the file itself is uploaded directly from the client to object storage. As a result, API services only process lightweight requests and do not handle large file transfer. -Interface: `POST /object/complete_multipart_upload` +Direct client upload has these advantages: -After the client successfully uploads all parts, it calls this interface to submit the list of part information. +- Reduces API service bandwidth and connection pressure. +- Makes large file upload more stable without affecting core interfaces such as message sending and receiving. +- Object storage is naturally suited for large objects and high-concurrency downloads. +- The file upload path is shorter, improving overall throughput. -- The backend verifies each part's hash and the overall hash. -- After passing verification, it starts the S3 merge operation. -- Finally, it returns the file's access address (which can jump to a signed S3 URL). +## 6. Multipart Upload -## ⬇️ Download Process +For larger images, videos, and files, the SDK uses multipart upload. The file is split into multiple smaller parts, each part is uploaded independently, and object storage merges them into a complete object at the end. -When the client downloads, it gets a temporary signed download address (S3 Pre-signed URL) from the backend, which can set permissions and valid period control. This address can unify access control verification via an API redirect. +Multipart upload brings a clear improvement in user experience: -## ✅ Feature Glance +- Large files do not need to be uploaded all at once. +- If one part fails, only the failed part needs to be retransmitted. +- Upload speed can be improved through controlled concurrency. +- The upload process can show more fine-grained progress. -| Feature | Description | -| --- | --- | -| Instant Upload Support | Achieve skipping upload based on file hash | -| Resumable Upload | Support resuming upload after disconnection or abnormal exit | -| Extremely Simple SDK | The client does not need signature logic, only executes PUT | -| Strong Storage Compatibility | Supports any object storage compatible with HTTP PUT | -| Highly Scalable | Can quickly integrate with other upload protocols and storage systems | +The SDK automatically selects an appropriate part size based on file size and server-side limits, so upper-layer business code does not need to handle these details manually. -## ⚙️ Scalability Design +## 7. Instant Upload -This solution is based on the core architecture of "unifying signature generation on the server + universal PUT upload on the client". As long as the target storage system supports multipart upload and accepts signed PUT requests, it can be seamlessly integrated: +OpenIM supports instant upload based on file content. The SDK calculates content features for the file, and the server can determine whether the same content already exists. If object storage already has this file, the file content is not uploaded again. Instead, the server directly registers a new business reference and returns an access address. -- Supports Amazon S3, MinIO, Alibaba Cloud OSS, Tencent Cloud COS, etc. -- Can also be extended to support other upload services. +Instant upload is especially valuable in repeated sending scenarios: -The client code remains unchanged; just extend the signature generation logic on the backend. +- When the same user sends the same file again, the upload can complete instantly. +- Different messages can reference the same real object, reducing duplicate storage. +- Network transfer cost and object storage write cost are reduced. +- Unnecessary retransmission is reduced in weak network environments. + +Instant upload is based on content features, not file names, which makes it better suited to IM file scenarios. + +## 8. Resumable Upload + +The SDK records upload progress locally. During upload, if the network is interrupted, the app exits, or the process restarts, successfully uploaded parts are not lost. When the user uploads the same file again, the SDK can restore the local state and upload only the remaining unfinished parts. + +Resumable upload significantly improves weak-network and mobile experiences: + +- Large files do not need to restart from the beginning after a failure. +- Uploads can continue after backgrounding, network interruption, or restart. +- Every successfully uploaded part becomes valid progress. +- Retrying is cheaper for users, and failure behavior is more controllable. + +This is also one of the important advantages of OpenIM's S3 solution compared with normal form upload. + +## 9. Concurrency and Memory Control + +The SDK supports concurrent multipart upload, but it does not allow unbounded concurrency. During upload, it dynamically adjusts the number of concurrent uploads based on part size and memory budget, avoiding excessive client memory usage for the sake of speed. + +When parts are smaller, the SDK can upload multiple parts concurrently. When parts are larger, the SDK lowers concurrency. If parts are too large, it falls back to a more conservative streaming upload method. + +This strategy is more stable across mobile, desktop, Web/WASM, and other environments. It balances upload speed, memory usage, and device stability. + +## 10. Progress Feedback and Upload Cancellation + +The SDK breaks the upload process into multiple perceivable stages, such as opening the file, calculating hashes, obtaining an upload session, uploading parts, and completing the merge. Upper-layer UI can use this to display more accurate progress instead of a vague percentage. + +This is important for large files. Users can clearly understand whether the current step is local calculation, network upload, or waiting for the server to complete the merge. + +The SDK also supports upload cancellation. The business side can bind a cancellation identifier to an upload task. After the user clicks cancel, the SDK stops the corresponding upload task to avoid continuing to consume network and device resources. + +## 11. Unified Access Address + +After an upload is complete, OpenIM returns a unified OpenIM access address instead of directly exposing the raw object storage address. When accessing a file, OpenIM generates a temporary access link based on object metadata and then redirects to the real storage address. + +This design has several benefits: + +- File addresses stored in messages are more stable. +- Underlying storage migration does not directly affect historical messages. +- Private objects can also be accessed securely through temporary authorization. +- Access policies such as file names, content types, and image thumbnails can be handled centrally. + +For image objects, the access layer can also support derived capabilities such as thumbnails, format conversion, and width or height cropping, improving image loading efficiency in chats. + +## 12. Permissions and Isolation + +When a normal user uploads an object, the object is placed in that user's own namespace. This prevents users from forging or overwriting other users' file objects. Before generating upload authorization, the server verifies identity and object ownership to ensure the upload behavior is controlled. + +Long-term object storage keys are not issued to clients. The client receives only short-lived upload or access credentials. Even if a link is leaked, the risk is limited to a short time window and a specific object scope. + +This approach balances direct upload efficiency with security boundaries. + +## 13. Metadata Governance + +Object storage is responsible only for storing file content, but an IM system also needs to know which user the file belongs to, which business type it belongs to, which storage backend it uses, when it was created, whether it can be cleaned up, and other information. + +OpenIM registers metadata for each business object. Metadata separates the "business object name" from the "real storage object". Multiple business objects can reference the same real object, which is also the foundation for instant upload and deduplication. + +Metadata governance provides the following capabilities: + +- Generate access addresses uniformly. +- Support content deduplication and instant upload. +- Support lifecycle cleanup by business type. +- Support checking whether a real object is still referenced. +- Support future migration of the underlying storage. + +## 14. Lifecycle Cleanup + +Not every file object in an IM system needs to be stored forever. For example, message objects such as images, voice messages, videos, and video covers can have retention periods based on business policies. + +OpenIM supports cleaning up expired objects by business category. During cleanup, it does not simply delete the real file. Instead, it deletes expired business references first, and then checks whether the real object is still referenced by other business objects. Only when there are no references left is the real file in object storage deleted. + +This reference-counting cleanup avoids accidental deletion. It naturally works together with instant upload: the same real file may be referenced by multiple messages, and the real object should be deleted only after all references have expired. + +## 15. FormData Compatibility Channel for Mini Programs + +In addition to the complete SDK multipart upload path, OpenIM also keeps a FormData direct form upload capability. This path is mainly used by the mini program JSSDK to adapt to file upload capabilities and invocation limits on mini program platforms. + +It is important to note that the SDK provided by Go does not call FormData. The Go SDK uses the complete multipart upload path. + +FormData is a compatibility channel and does not provide the advanced capabilities of the complete upload path. It does not support: + +- Multipart upload. +- Instant upload. +- Local resumable upload. +- Multipart concurrency. +- Part-level verification or part-level progress callbacks. + +Therefore, normal apps, desktop clients, the Go SDK, and Web/WASM clients should prioritize the complete SDK upload capability. Under platform limits, the mini program JSSDK can use FormData as a simplified direct upload solution. + +## 16. Storage Migration Capability + +Because OpenIM records the storage backend that each object belongs to in metadata, the system has the foundation for migrating from one object storage service to another. For example, it can migrate from a self-hosted MinIO deployment to cloud-vendor object storage, or switch between different cloud vendors. + +During migration, the system can read the real object from the old storage, write it to the new storage, and update the storage backend information in metadata. Business messages store OpenIM's unified access address, so historical messages can still be accessed in the original way after migration. + +This reduces long-term operations risk and lowers the cost of being tied to a single storage vendor or deployment model. + +## 17. Advantages Over Traditional Upload Methods + +The advantages of OpenIM's S3 solution can be summarized in five categories. + +In terms of performance, file traffic goes directly to object storage, and API services only process control requests. Large file uploads do not consume core business interfaces. + +In terms of stability, multipart upload, resumable upload, part verification, upload cancellation, and controlled concurrency work together to improve the success rate in weak network environments. + +In terms of cost, instant upload and deduplication reduce duplicate uploads and duplicate storage, while lifecycle cleanup reduces long-term space occupied by invalid objects. + +In terms of security, clients receive only short-lived authorization and never touch long-term object storage keys. File access can also be controlled through temporary links. + +In terms of evolution, unified object addresses and metadata governance allow the underlying storage to be migrated or replaced without requiring the business layer to understand vendor-specific changes. + +## 18. Applicable Scenarios + +OpenIM S3 object storage is suitable for the following scenarios: + +- Chat images, original images, and thumbnails. +- Voice messages. +- Video messages and video covers. +- Normal file messages. +- Capabilities that require temporary object upload, such as ASR. +- File storage in private deployments. +- Large file, weak network, and mobile upload scenarios. + +If the business only needs simple form upload for mini programs, it can use the FormData compatibility channel. If instant upload, multipart upload, resumable upload, and a more complete upload experience are required, it should use the complete SDK upload capability. + +## 19. Summary + +OpenIM's S3 object storage capability is file infrastructure built for large-object IM scenarios. It reduces server pressure through direct client upload, improves weak-network experience through multipart and resumable upload, lowers costs through instant upload and lifecycle cleanup, and improves long-term maintainability through unified access addresses and metadata governance. + +Its core value is to turn file upload from "a heavy burden on the business API" into "an object service capability that can be authorized, resumed, deduplicated, governed, and migrated". For enterprise-grade IM, this design improves performance, stability, security, cost control, and future evolution at the same time.