Raftsimple (copy of raftexample) - Simple key-value store example

contrib/raftsimple/Procfile

@@ -0,0 +1,3 @@
+raft1: ./raftsimple --id 1 --cluster http://127.0.0.1:9021,http://127.0.0.1:9022,http://127.0.0.1:9023 --port 9121
+raft2: ./raftsimple --id 2 --cluster http://127.0.0.1:9021,http://127.0.0.1:9022,http://127.0.0.1:9023 --port 9122
+raft3: ./raftsimple --id 3 --cluster http://127.0.0.1:9021,http://127.0.0.1:9022,http://127.0.0.1:9023 --port 9123

contrib/raftsimple/README.md

@@ -0,0 +1,109 @@
+# raftsimple
+
+<!-- `raftsimple` is a simplified, educational version of etcd's `raftexample`. While `raftexample` demonstrates production-grade integration with `etcd/raft`, `raftsimple` focuses on minimalism, providing a self-contained implementation with a custom HTTP transport and a basic file-based Write-Ahead Log (WAL). -->
+
+This project is designed for developers who want to understand the core mechanics of the Raft algorithm and the `Ready()` event loop without the complexity of the full etcd server ecosystem.
+
+## Getting started
+
+### Building raftsimple
+
+Build the binary:
+
+```bash
+go build -o raftsimple
+```
+
+### Running single-node raftsimple
+
+To start a single-node Raft cluster, run:
+
+```bash
+./raftsimple --id 1 --cluster http://127.0.0.1:9021 --port 9121
+```
+
+This starts a node with ID 1, listening for Raft messages on `127.0.0.1:9021`, and exposing a key-value API on port `9121`.
+
+You can now interact with the key-value store:
+
+```bash
+# Propose a value
+curl -L http://127.0.0.1:9121/foo -XPUT -d bar
+
+# Retrieve the value
+curl -L http://127.0.0.1:9121/foo
+```
+
+### Running a local cluster
+
+For a more realistic distributed setup, use [goreman](https://github.com/mattn/goreman) to run a 3-node cluster:
+
+```bash
+goreman start
+```
+
+This uses the included `Procfile` to start three independent `raftsimple` processes on different ports. Each node can be communicated with via its respective key-value API at ports `9121`, `9122`, and `9123`.
+
+### Fault Tolerance
+
+Raft's key feature is its ability to withstand node failures. You can simulate a node crash with `goreman` (it uses Procfile file):
+
+```bash
+# Stop Node 2
+goreman run stop raft2
+```
+
+The cluster will still operate with Node 1 and Node 3. Propose a new value on Node 1:
+
+```bash
+curl -L http://127.0.0.1:9121/foo -XPUT -d baz
+```
+
+Now restart Node 2:
+
+```bash
+goreman run restart raft2
+```
+
+Node 2 will automatically recover its state from its local WAL and snapshot, then catch up on any missed data from the leader. You can verify it has recovered:
+
+```bash
+curl -L http://127.0.0.1:9122/foo
+```
+
+### Dynamic Reconfiguration
+
+`raftsimple` also supports membership changes via the KV API. To add a new node (Node 4) to the cluster:
+
+```bash
+# Register Node 4 with the cluster, providing its URL in the request body
+curl -L http://127.0.0.1:9121/4 -XPOST -d http://127.0.0.1:9024
+
+# Start Node 4 in a separate terminal with the --join flag
+./raftsimple --id 4 --cluster http://127.0.0.1:9021,http://127.0.0.1:9022,http://127.0.0.1:9023,http://127.0.0.1:9024 --port 9124 --join
+```
+
+Similarly, to remove a node from the cluster:
+
+```bash
+curl -L http://127.0.0.1:9121/4 -XDELETE
+```
+
+## Design
+
+The `raftsimple` architecture is divided into three main components:
+
+1.  **KV Store (`kvstore.go`):** The application-level state machine. It manages the actual key-value data and provides methods for proposing changes to the Raft log and applying committed entries.
+2.  **REST API (`httpapi.go`):** The user-facing interface. It handles incoming HTTP requests and converts them into proposals for the Raft node.
+3.  **Raft Node (`raft.go`):** The core consensus engine. It manages the `etcd/raft` state machine, handles the `Ready()` loop, and coordinates with the transport and storage layers.
+
+## Project Philosophy: Simplicity First
+
+`raftsimple` is designed as an educational tool, not a production-grade database. To keep the code as readable and linear as possible, we have made deliberate design choices that prioritize **simplicity over robustness**:
+
+*   **Synchronous Storage:** Unlike production systems that use complex asynchronous I/O and batching, `raftsimple` writes to disk synchronously within the main Raft loop.
+*   **Minimalist WAL:** The Write-Ahead Log is a simple append-only file. It does not include checksums, CRCs, or automatic truncation after snapshots.
+*   **Simple Transport:** The inter-node communication uses standard Go `net/http` for simplicity, rather than the more complex but efficient `rafthttp` package used in `etcd`.
+*   **Direct File I/O:** Snapshots and HardState are overwritten directly on disk.
+
+These choices make the codebase significantly easier to audit and learn from, though they mean the project should not be used for storing critical data.

contrib/raftsimple/go.mod

@@ -0,0 +1,17 @@
+module raftsimple
+
+go 1.25.7
+
+require (
+	github.com/stretchr/testify v1.10.0
+	go.etcd.io/raft/v3 v3.6.0
+)
+
+require (
+	github.com/davecgh/go-spew v1.1.1 // indirect
+	github.com/gogo/protobuf v1.3.2 // indirect
+	github.com/golang/protobuf v1.5.4 // indirect
+	github.com/pmezard/go-difflib v1.0.0 // indirect
+	google.golang.org/protobuf v1.33.0 // indirect
+	gopkg.in/yaml.v3 v3.0.1 // indirect
+)

contrib/raftsimple/go.sum

@@ -0,0 +1,51 @@
+github.com/cockroachdb/datadriven v1.0.2 h1:H9MtNqVoVhvd9nCBwOyDjUEdZCREqbIdCJD93PBm/jA=
+github.com/cockroachdb/datadriven v1.0.2/go.mod h1:a9RdTaap04u637JoCzcUoIcDmvwSUtcUFtT/C3kJlTU=
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
+github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
+github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
+github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
+github.com/google/go-cmp v0.5.8 h1:e6P7q2lk1O+qJJb4BtCQXlK8vWEO8V1ZeuEdJNOqZyg=
+github.com/google/go-cmp v0.5.8/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
+github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
+github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
+github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
+github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
+github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
+go.etcd.io/raft/v3 v3.6.0 h1:5NtvbDVYpnfZWcIHgGRk9DyzkBIXOi8j+DDp1IcnUWQ=
+go.etcd.io/raft/v3 v3.6.0/go.mod h1:nLvLevg6+xrVtHUmVaTcTz603gQPHfh7kUAwV6YpfGo=
+golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
+golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
+golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
+golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
+golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
+golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
+golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
+golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
+golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+google.golang.org/protobuf v1.33.0 h1:uNO2rsAINq/JlFpSdYEKIZ0uKD/R9cpdv0T+yoGwGmI=
+google.golang.org/protobuf v1.33.0/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
+gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

contrib/raftsimple/httpapi.go

@@ -0,0 +1,107 @@
+package main
+
+import (
+	"io"
+	"log"
+	"net/http"
+	"strconv"
+
+	"go.etcd.io/raft/v3/raftpb"
+)
+
+type httpKVAPI struct {
+	store       *kvstore
+	confChangeC chan<- raftpb.ConfChange
+}
+
+func (h *httpKVAPI) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+	key := r.RequestURI
+	defer r.Body.Close()
+
+	switch r.Method {
+	case http.MethodPut:
+		v, err := io.ReadAll(r.Body)
+		if err != nil {
+			log.Printf("Failed to read on PUT (%v)\n", err)
+			http.Error(w, "Failed to PUT", http.StatusBadRequest)
+			return
+		}
+		h.store.Propose(key, string(v))
+		w.WriteHeader(http.StatusNoContent)
+
+	case http.MethodGet:
+		if v, ok := h.store.Lookup(key); ok {
+			w.Write([]byte(v))
+		} else {
+			http.Error(w, "Failed to GET", http.StatusNotFound)
+		}
+
+	case http.MethodPost:
+		nodeID, err := strconv.ParseUint(key[1:], 0, 64)
+		if err != nil {
+			log.Printf("Failed to convert ID for conf change (%v)\n", err)
+			http.Error(w, "Failed on POST", http.StatusBadRequest)
+			return
+		}
+
+		url, err := io.ReadAll(r.Body)
+		if err != nil {
+			log.Printf("Failed to read body for conf change (%v)\n", err)
+			http.Error(w, "Failed on POST", http.StatusBadRequest)
+			return
+		}
+
+		cc := raftpb.ConfChange{
+			Type:    raftpb.ConfChangeAddNode,
+			NodeID:  nodeID,
+			Context: url,
+		}
+		h.confChangeC <- cc
+
+		w.WriteHeader(http.StatusNoContent)
+
+	case http.MethodDelete:
+		nodeID, err := strconv.ParseUint(key[1:], 0, 64)
+		if err != nil {
+			log.Printf("Failed to convert ID for conf change (%v)\n", err)
+			http.Error(w, "Failed on DELETE", http.StatusBadRequest)
+			return
+		}
+
+		cc := raftpb.ConfChange{
+			Type:   raftpb.ConfChangeRemoveNode,
+			NodeID: nodeID,
+		}
+		h.confChangeC <- cc
+
+		w.WriteHeader(http.StatusNoContent)
+
+	default:
+		w.Header().Set("Allow", http.MethodPut)
+		w.Header().Add("Allow", http.MethodGet)
+		w.Header().Add("Allow", http.MethodPost)
+		w.Header().Add("Allow", http.MethodDelete)
+		http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
+	}
+}
+
+func serveHTTPKVAPI(kv *kvstore, port uint64, confChangeC chan<- raftpb.ConfChange, done <-chan struct{}) {
+	srv := http.Server{
+		Addr: ":" + strconv.FormatUint(port, 10),
+		Handler: &httpKVAPI{
+			store:       kv,
+			confChangeC: confChangeC,
+		},
+	}
+	go func() {
+		if err := srv.ListenAndServe(); err != nil {
+			if err != http.ErrServerClosed {
+				log.Fatal(err)
+			}
+		}
+	}()
+
+	// exit when raft goes down
+	<-done
+	_ = srv.Close()
+}

contrib/raftsimple/integration_test.go

@@ -0,0 +1,89 @@
+package main
+
+import (
+	"fmt"
+	"os"
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/require"
+	"go.etcd.io/raft/v3/raftpb"
+)
+
+func findLeader(nodes map[uint64]*raftNode) *raftNode {
+	for range 50 {
+		for _, rn := range nodes {
+			if rn.node != nil && rn.node.Status().Lead == rn.id {
+				return rn
+			}
+		}
+		time.Sleep(100 * time.Millisecond)
+	}
+	return nil
+}
+
+func TestIntegration_Lifecycle(t *testing.T) {
+	tmpDir := t.TempDir()
+	defer os.RemoveAll(tmpDir)
+
+	peers := []uint64{1, 2, 3}
+	peerURLs := make(map[uint64]string)
+	for _, id := range peers {
+		peerURLs[id] = fmt.Sprintf("http://127.0.0.1:%d", 10000+id)
+	}
+
+	nodes := make(map[uint64]*raftNode)
+	kvstores := make(map[uint64]*kvstore)
+
+	createNode := func(id uint64) {
+		snapdir := fmt.Sprintf("%s/node-%d", tmpDir, id)
+		ss, _ := newSnapshotStorage(snapdir)
+		proposeC := make(chan string)
+		confChangeC := make(chan raftpb.ConfChange)
+		kvs, fsm := newKVStore(proposeC)
+		rn := newRaftNode(id, peerURLs, false, fsm, ss, proposeC, confChangeC)
+		nodes[id] = rn
+		kvstores[id] = kvs
+		go rn.processCommits()
+	}
+
+	for _, id := range peers {
+		createNode(id)
+	}
+
+	defer func() {
+		for _, rn := range nodes {
+			rn.stop()
+		}
+	}()
+
+	leader := findLeader(nodes)
+	require.NotNil(t, leader, "A leader should be elected")
+	kvstores[leader.id].Propose("key1", "val1")
+
+	require.Eventually(t, func() bool {
+		v, ok := kvstores[3].Lookup("key1")
+		return ok && v == "val1"
+	}, 5*time.Second, 100*time.Millisecond, "Data should replicate to Node 3")
+
+	// Simulate node 2 crash
+	nodes[2].stop()
+
+	leader = findLeader(nodes)
+	require.NotNil(t, leader, "A leader should be elected after crash")
+	kvstores[leader.id].Propose("key2", "val2")
+
+	require.Eventually(t, func() bool {
+		v, ok := kvstores[3].Lookup("key2")
+		return ok && v == "val2"
+	}, 5*time.Second, 100*time.Millisecond, "Data should replicate to Node 3 while Node 2 is offline")
+
+	// Restart node 2
+	createNode(2)
+
+	require.Eventually(t, func() bool {
+		v1, ok1 := kvstores[2].Lookup("key1")
+		v2, ok2 := kvstores[2].Lookup("key2")
+		return ok1 && ok2 && v1 == "val1" && v2 == "val2"
+	}, 10*time.Second, 100*time.Millisecond, "Node 2 should recover and catch up on all data")
+}