ShadowVault-Decentralized-Backup-Agent

ShadowVault — Decentralized Encrypted Backup Agent

Go protobuf libp2p AES-256-GCM Content Addressed Storage Ed25519 Bbolt CLI Docker Docker Compose C C++ Rust makefile Testing Shell Scripting CLI MIT License

Table of Contents

  1. Overview
  2. Goals & Motivation
  3. Key Concepts
  4. Quickstart
  5. Installation & Build
  6. Configuration
  7. CLI Commands & Usage Reference
  8. Snapshot Lifecycle
  9. Deduplication & CAS Internals
  10. Identity & Authentication
  11. Peer Management
  12. PubSub Message Formats & Validation
  13. Restore Workflow
  14. Testing
  15. Docker & Orchestration
  16. Utility C Tool
  17. Shell Helpers & Entry Point
  18. Example File Layout After Run
  19. Troubleshooting & Common Issues
  20. Protocol Buffers
  21. Security Considerations
  22. Extension Points / Developer Notes
  23. Contributing
  24. Glossary
  25. License

Overview

ShadowVault is a privacy-preserving, decentralized backup agent written in Go. It snapshots filesystem data, chunks and deduplicates content, encrypts everything client-side, and synchronizes encrypted chunks and metadata across a peer-to-peer network using libp2p. There is no trusted central server: peers gossip what blocks they have, fetch missing pieces directly, and validate integrity and authenticity through signatures.

Goals & Motivation

Key Concepts

Core Technologies

Security Primitives

P2P Sync Model

sequenceDiagram
    participant User
    participant Daemon as "ShadowVault Daemon"
    participant Identity as "Identity Store (Ed25519 keys / ACL)"
    participant Chunker
    participant Dedup as "Deduplicator"
    participant Encryptor as "AES-256-GCM Encryptor"
    participant LocalCAS as "Local CAS (encrypted blobs)"
    participant MetaDB as "Metadata DB (bbolt)"
    participant PubSub as "PubSub / Gossip"
    participant Peer as "Remote Peer"
    participant Restore as "Restore Agent"
    participant Decryptor as "Decryptor"
    participant Reconstructor as "File Reconstructor"

    %% Snapshot creation
    User->>Daemon: request snapshot of directory
    Daemon->>Identity: load persistent identity & ACL
    Daemon->>Chunker: chunk files (content-defined)
    Chunker->>Dedup: send chunk hashes
    Dedup->>LocalCAS: check existing encrypted chunks
    alt chunk missing locally
        Dedup->>Encryptor: encrypt chunk with derived key
        Encryptor->>LocalCAS: store encrypted chunk (SHA-256 address)
    else chunk already present
        Dedup-->>LocalCAS: reuse existing blob
    end
    Daemon->>MetaDB: assemble snapshot metadata (chunk list, parent, timestamps)
    Daemon->>Identity: sign snapshot metadata with Ed25519
    Daemon->>MetaDB: persist signed snapshot descriptor
    Daemon->>PubSub: publish SnapshotAnnouncement
    Daemon->>PubSub: publish BlockAnnounce for available chunk hashes

    %% Peer synchronization
    PubSub->>Peer: receive SnapshotAnnouncement + BlockAnnounces
    Peer->>Identity: verify snapshot signature against ACL/public key
    alt signature valid and allowed
        Peer->>LocalCAS: check which announced chunks are missing
        alt has missing chunks
            Peer->>Peer: open direct libp2p stream to known holder\nrequest specific chunk
            Peer->>LocalCAS: if requested, send encrypted chunk
            Peer-->>Peer: receive encrypted chunk
            Peer->>LocalCAS: store received encrypted chunk
        end
        Peer->>MetaDB: update block availability index / peer cache
    else invalid announcement
        Peer-->>PubSub: ignore / log rejection
    end

    %% Restore workflow
    User->>Restore: request restore of snapshot ID
    Restore->>MetaDB: fetch snapshot metadata
    Restore->>Identity: verify snapshot signature
    Restore->>LocalCAS: for each chunk in snapshot, check local presence
    alt chunk present
        LocalCAS-->>Restore: provide encrypted chunk
    else chunk missing
        Restore->>PubSub: query gossip for holders
        Restore->>Peer: direct fetch chunk via libp2p stream
        Peer-->>Restore: send encrypted chunk
        Restore->>LocalCAS: cache fetched encrypted chunk
    end
    Restore->>Decryptor: decrypt chunk(s) using derived key
    Decryptor->>Reconstructor: supply plaintext pieces
    Reconstructor->>User: reassemble original files with metadata

Quickstart

# Build and start daemon, create identity and default config, snapshot a directory
./entrypoint.sh config.yaml /path/to/important/data

# List known peers
./bin/peerctl -c config.yaml -p "yourpass" list

# Add a peer (multiaddr)
./bin/peerctl add /ip4/1.2.3.4/tcp/9000/p2p/<peerID> -c config.yaml -p "yourpass"

# Restore a snapshot
./bin/restore-agent restore <snapshot-id> restored/ -c config.yaml -p "yourpass"

Installation & Build

Prerequisites

Local Build

git clone <repo-url> shadowvault
cd shadowvault
make build

This produces:

Run Tests

make test

Configuration

Primary configuration lives in config.yaml (created automatically by entrypoint.sh if absent). Example:

repository_path: "./data"
listen_port: 9000
peer_bootstrap:
  - "/ip4/127.0.0.1/tcp/9001/p2p/QmSomePeerID"
nat_traversal:
  enable_auto_relay: true
  enable_hole_punching: true
snapshot:
  min_chunk_size: 2048
  max_chunk_size: 65536
  avg_chunk_size: 8192
acl:
  admins:
    - "base64-ed25519-pubkey..."

Defaults are applied when fields are missing.

CLI Commands & Usage Reference

backup-agent (daemon & snapshot)

# Start daemon
./bin/backup-agent daemon -c config.yaml -p "passphrase"

# Take snapshot of a directory
./bin/backup-agent snapshot /path/to/dir -c config.yaml -p "passphrase"

restore-agent

# Restore snapshot by ID to target directory
./bin/restore-agent restore <snapshot-id> <target-dir> -c config.yaml -p "passphrase"

peerctl

# List stored/known peers
./bin/peerctl list -c config.yaml -p "passphrase"

# Add a peer by multiaddr
./bin/peerctl add /ip4/1.2.3.4/tcp/9000/p2p/<peerID> -c config.yaml -p "passphrase"

# Remove a stored peer
./bin/peerctl remove <peerID> -c config.yaml -p "passphrase"

Flags:

Snapshot Lifecycle

  1. Chunking: Files in the target directory are read with content-defined chunking (configurable min/avg/max) to produce variable-sized pieces.
  2. Deduplication: Each chunk is hashed (SHA-256) and if already present locally, skipped.
  3. Encryption: Chunks are encrypted with AES-256-GCM using a key derived from the user passphrase.
  4. Storage: Encrypted chunks are stored in CAS (via bbolt or on-disk object layout).
  5. Snapshot metadata: A snapshot descriptor listing chunk hashes, parent snapshot (optional), timestamps, and provenance is assembled and signed.
  6. Announcement: Signed snapshot and block availability are gossip-published to peers via pubsub.

Deduplication & CAS Internals

Identity & Authentication

Peer Management

PubSub Message Formats & Validation

Core message envelope used in gossip:

{
  "type": "snapshot_announce" | "block_announce" | "peer_add" | "peer_remove",
  "payload": { /* type-specific struct */ },
  "sig": "<base64 signature over type||payload>",
  "pubkey": "<base64-ed25519 public key of signer>"
}

Validation steps:

  1. Decode pubkey, sig.
  2. Reconstruct signing context (type + raw payload).
  3. Verify signature (Ed25519).
  4. Accept or reject based on ACL (for sensitive types).

Restore Workflow

  1. Specify snapshot ID to restore.
  2. Snapshot is loaded and its signature verified.
  3. For each chunk hash in the snapshot:

Example:

./bin/restore-agent restore snapshot-abc123 restored/ -c config.yaml -p "yourpass"

Testing

Unit tests are included for critical modules:

Run:

make test

Or directly:

go test ./... -v

Docker & Orchestration

Build Image

docker build -t shadowvault:latest .

Run ShadowVault Daemon

docker run --rm -v "$(pwd)/data":/data -v "$(pwd)/config.yaml":/app/config.yaml:ro -e PASSPHRASE=yourpass shadowvault:latest daemon -c /app/config.yaml -p "$PASSPHRASE"

Compose Two Nodes

docker compose up

(This uses docker-compose.yml to spin up node1 and node2, share bootstrap configuration and run daemons.)

Snapshot & Restore Inside Container

docker exec -it backupagent_node1 /bin/sh -c "./bin/backup-agent snapshot /data/to/backup -c /app/config.yaml -p yourpass"
docker exec -it backupagent_node1 /bin/sh -c "./bin/restore-agent restore <snapshot-id> /restored -c /app/config.yaml -p yourpass"

Persisting State

Mount host directories into container to persist:

Utility C Tool

tools/hashfile.c is a small companion compiled with OpenSSL that computes SHA-256 of arbitrary files (helpful for independent verification):

Compile:

gcc -o tools/hashfile tools/hashfile.c -lcrypto

Usage:

./tools/hashfile /path/to/snapshot.json

Outputs hex digest + filename.

Shell Helpers & Entry Point

Make executable:

chmod +x entrypoint.sh scripts/*.sh

Example File Layout After Run

.
├── config.yaml
├── entrypoint.sh
├── bin/
│   ├── backup-agent
│   ├── restore-agent
│   └── peerctl
├── data/                  # repository_path
│   ├── identity.key       # persistent libp2p key
│   ├── metadata.db        # bbolt DB (snapshots, peers, blocks)
│   └── snapshots/         # encrypted snapshot metadata
├── snapshots/             # (optional local snapshot working trees)
├── .shadowvault/          # if alternate layout used
├── scripts/
│   ├── bootstrap.sh
│   ├── snapshot.sh
│   └── restore.sh
├── tools/
│   └── hashfile           # compiled C helper
└── README.md              # this document

Troubleshooting & Common Issues

Problem Likely Cause Remedy
Snapshot fails with read errors Permissions or missing files Check file access, run with sufficient privileges
Cannot fetch chunk from peer Peer offline / no announcement Ensure peer is connected, check gossip logs, add via peerctl
Signature validation fails Passphrase mismatch / tampered snapshot Verify passphrase; reject snapshot if integrity compromised
Identity changes unexpectedly Identity key deleted or corrupted Restore identity.key backup; avoid deleting it
Peer not discovered DHT/bootstrap misconfig Ensure bootstrap addresses are correct and reachable
Cache inconsistency on restore Corrupted local chunk Delete affected chunk and allow re-fetch from another peer

Protocol Buffers

ShadowVault defines its on-wire and on-disk message formats in Protobuf, organized under proto/:


proto/
common.proto
snapshot.proto
block.proto
peer.proto
auth.proto
identity.proto
service.proto

Generating Go bindings

Install the Protobuf plugins for Go:

go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest

Then from the project root run:

protoc --go_out=. --go-grpc_out=. proto/*.proto

This will generate Go packages under github.com/yourusername/shadowvault/proto/....

File-by-file summary

Security Considerations

Extension Points / Developer Notes

Contributing

  1. Fork the repository.
  2. Create a feature branch (e.g., feature/remote-cas).
  3. Add or update tests demonstrating the new behavior.
  4. Submit a pull request with a clear description and rationale.

Areas of high impact:

Glossary

License

MIT License. See LICENSE for full terms.

Thank you for checking out ShadowVault! We hope it helps you securely back up and manage your data in a decentralized way. For any questions or contributions, please refer to the Contributing section.

This site is open source. Improve this page.