Skip to content

feat: add filecoin-pin-migration lib #694

Open
BravoNatalie wants to merge 16 commits intomainfrom
feat/filecoin-pin-migration-lib
Open

feat: add filecoin-pin-migration lib #694
BravoNatalie wants to merge 16 commits intomainfrom
feat/filecoin-pin-migration-lib

Conversation

@BravoNatalie
Copy link
Copy Markdown
Contributor

@BravoNatalie BravoNatalie commented Apr 8, 2026

from the README:

Context

Headless library for migrating Storacha content to Filecoin on Chain (FOC) without re-uploading data.

Filecoin storage providers pull your data directly from Storacha's storage via the Curio pull API. Your data never moves through your machine.


How it works

Migration runs in three sequential phases:

Reader → Planner → Migrator

1. Reader — inventory

Walks every space on the Storacha client, paginates all uploads, and queries the indexing service to resolve shards:

  • pieceCID — required by the Filecoin storage provider
  • sourceURL — the URL the SP will pull from (resolved by the chosen strategy)
  • sizeBytes — derived from the piece itself

The result is a SpaceInventory per space: a structured list of every upload and its resolved shards, with counts and byte totals pre-computed.

2. Planner — cost calculation and approval

Takes space inventories, creates one StorageContext per space per copy (each context binds a payer, provider, and on-chain dataset via the Synapse SDK), and computes the single USDFC deposit needed across all spaces. Returns a MigrationPlan the consumer can display for user approval before any on-chain action.

3. Migrator — execution

Executes the approved plan as an AsyncGenerator. Yields MigrationEvent objects the consumer handles at its own pace: persisting state to disk, displaying progress, logging failures.

Each space is processed upload-by-upload in configurable batches. Each batch runs three stages:

  1. Presign — EIP-712 signature for the batch
  2. Pull — SP fetches pieces from the sourceURL; pieces are partitioned into succeeded/failed
  3. Commit — on-chain registration of successfully pulled pieces and creation of datasets

State is checkpointed after every batch with at least one commit, enabling crash recovery.


Prerequisites

  • A funded USDFC wallet — required for storage payment lockup
  • FIL for gas — required for on-chain transactions
  • An authenticated @storacha/client instance with access to the spaces to migrate
  • An initialized @filoz/synapse-sdk Synapse instance
  • Access to an indexing service (for resolving shard claims)

Installation

npm install @storacha/filecoin-pin-migration

Comment thread packages/filecoin-pin-migration/.context/ARCHITECTURE.md Outdated
Comment thread packages/filecoin-pin-migration/.context/ARCHITECTURE.md Outdated
Comment thread packages/filecoin-pin-migration/.context/ARCHITECTURE.md Outdated
Comment thread packages/filecoin-pin-migration/.context/ARCHITECTURE.md Outdated
Comment thread packages/filecoin-pin-migration/.context/ARCHITECTURE.md Outdated
Comment thread packages/filecoin-pin-migration/.context/ARCHITECTURE.md Outdated
@BravoNatalie BravoNatalie force-pushed the feat/filecoin-pin-migration-lib branch from ae7ff67 to 0e83477 Compare April 13, 2026 16:10
@BravoNatalie BravoNatalie force-pushed the feat/filecoin-pin-migration-lib branch from 761736f to 957aa1e Compare April 14, 2026 15:00
@BravoNatalie BravoNatalie marked this pull request as ready for review April 16, 2026 05:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants