🔐 ShadowVault

Privacy-Preserving, Decentralized Encrypted Backup Agent

🔒 AES-256-GCM 🌐 P2P libp2p 🗄️ Content-Addressed Storage ✍️ Ed25519 Signatures 🐹 Built with Go 🐳 Docker Ready ⚡ Zero Trust 📦 Deduplication
🚀 Get Started 📚 Read Docs

Project Overview

A privacy-first, decentralized backup solution with military-grade encryption

🔐

Client-Side Encryption

All data is encrypted locally using AES-256-GCM before storage or transmission. Your passphrase never leaves your machine, and peers can't read your data without it.

🌐

Truly Decentralized

Built on libp2p for peer-to-peer communication. No central servers, no single point of failure. Peers gossip and sync directly via PubSub and direct streams.

📦

Smart Deduplication

Content-addressed storage with SHA-256 hashing ensures only unique chunks are stored. Saves bandwidth and storage across all snapshots.

✍️

Cryptographic Signatures

Every snapshot is signed with Ed25519 keys. Verify authenticity and prevent tampering with cryptographic proof of origin.

🔄

Automatic Synchronization

Peers automatically discover and fetch missing chunks through gossip protocols. Multi-source downloads improve resilience and speed.

⚡

NAT Traversal

Built-in support for auto-relay and hole punching. Works behind NATs and firewalls without manual port forwarding.

Technology Stack

Built with cutting-edge technologies for maximum security and performance

🐹
Go 1.21+
Core implementation language
🌐
libp2p
P2P networking & discovery
🗄️
bbolt
Embedded key-value DB
🔒
AES-256-GCM
Authenticated encryption
✍️
Ed25519
Digital signatures
📡
Protocol Buffers
Efficient serialization
🐳
Docker
Containerization
⚙️
Make
Build automation

System Architecture

Comprehensive workflow from snapshot creation to peer synchronization and restoration

sequenceDiagram participant User participant Daemon as ShadowVault Daemon participant Identity as Identity Store
(Ed25519/ACL) participant Chunker participant Dedup as Deduplicator participant Encryptor as AES-256-GCM participant LocalCAS as Local CAS
(Encrypted Blobs) participant MetaDB as Metadata DB
(bbolt) participant PubSub as PubSub/Gossip participant Peer as Remote Peer Note over User,Peer: SNAPSHOT CREATION PHASE User->>Daemon: snapshot /path/to/dir Daemon->>Identity: Load identity & ACL Daemon->>Chunker: Chunk files (CDC) Chunker->>Dedup: Check chunk hashes Dedup->>LocalCAS: Query existing chunks alt Chunk Missing Dedup->>Encryptor: Encrypt with derived key Encryptor->>LocalCAS: Store encrypted chunk else Chunk Exists Dedup-->>LocalCAS: Reuse existing end Daemon->>MetaDB: Assemble snapshot metadata Daemon->>Identity: Sign with Ed25519 Daemon->>MetaDB: Persist signed snapshot Daemon->>PubSub: Publish SnapshotAnnouncement Daemon->>PubSub: Publish BlockAnnounces Note over User,Peer: PEER SYNC PHASE PubSub->>Peer: Receive announcements Peer->>Identity: Verify signature & ACL alt Valid Signature Peer->>LocalCAS: Check missing chunks alt Has Missing Chunks Peer->>Daemon: Request via libp2p stream Daemon->>Peer: Send encrypted chunk Peer->>LocalCAS: Store chunk end Peer->>MetaDB: Update indices else Invalid Peer-->>PubSub: Reject & log end Note over User,Peer: RESTORE PHASE User->>Daemon: restore snapshot-id Daemon->>MetaDB: Fetch snapshot Daemon->>Identity: Verify signature loop For each chunk alt Chunk Local LocalCAS-->>Daemon: Return chunk else Chunk Remote Daemon->>Peer: Fetch via stream Peer-->>Daemon: Send chunk end Daemon->>Encryptor: Decrypt chunk end Daemon->>User: Reconstruct files
💡 Key Insight: The architecture ensures zero-trust operation. All sensitive data is encrypted before leaving the local machine, and cryptographic signatures guarantee authenticity at every step.

Quick Start Guide

Get up and running in minutes

📥 Installation & Build

bash 
# Clone the repository
git clone https://github.com/hoangsonww/ShadowVault-Go-Encryptor.git
cd ShadowVault-Go-Encryptor

# Build all binaries
make build

# Run tests
make test

This creates three binaries: backup-agent, restore-agent, and peerctl.

🚀 Initial Setup

bash 
# Use the entry point script (builds + configures + starts daemon)
./entrypoint.sh config.yaml /path/to/backup

# Or manually start the daemon
./bin/backup-agent daemon -c config.yaml -p "your-secure-passphrase"

📸 Creating Snapshots

bash 
# Take a snapshot of a directory
./bin/backup-agent snapshot /path/to/important/data -c config.yaml -p "passphrase"

# Using the helper script
./scripts/snapshot.sh /path/to/important/data

👥 Managing Peers

bash 
# List known peers
./bin/peerctl list -c config.yaml -p "passphrase"

# Add a peer by multiaddr
./bin/peerctl add /ip4/192.168.1.100/tcp/9000/p2p/QmPeerID -c config.yaml -p "passphrase"

# Remove a peer
./bin/peerctl remove QmPeerID -c config.yaml -p "passphrase"

♻️ Restoring Data

bash 
# Restore a snapshot by ID
./bin/restore-agent restore <snapshot-id> /path/to/restore -c config.yaml -p "passphrase"

# Using the helper script
./scripts/restore.sh <snapshot-id> /path/to/restore

🐳 Docker Deployment

Run ShadowVault in containers for easy deployment and scaling

🏗️

Build Image

bash 
# Build the Docker image
docker build -t shadowvault:latest .

# Or use make
make docker
▶️

Run Container

bash 
# Run daemon in container
docker run --rm \
  -v "$(pwd)/data":/data \
  -v "$(pwd)/config.yaml":/app/config.yaml:ro \
  -e PASSPHRASE=yourpass \
  shadowvault:latest
🔗

Multi-Node Setup

bash 
# Start multiple nodes with compose
docker compose up

# Scale to N nodes
docker compose up --scale node1=3

⚙️ Configuration

Customize ShadowVault with config.yaml

yaml 
repository_path: "./data"
listen_port: 9000

# Bootstrap peers for initial connection
peer_bootstrap:
  - "/ip4/127.0.0.1/tcp/9001/p2p/QmSomePeerID"

# NAT traversal settings
nat_traversal:
  enable_auto_relay: true
  enable_hole_punching: true

# Chunking configuration
snapshot:
  min_chunk_size: 2048
  max_chunk_size: 65536
  avg_chunk_size: 8192

# Access control list
acl:
  admins:
    - "base64-ed25519-pubkey-1"
    - "base64-ed25519-pubkey-2"
Parameter Description Default
repository_path Local storage path for snapshots and metadata ./data
listen_port Port for libp2p connections 9000
peer_bootstrap List of bootstrap peer multiaddrs []
enable_auto_relay Enable automatic relay for NAT traversal true
enable_hole_punching Enable DCUtR hole punching true
min_chunk_size Minimum chunk size in bytes 2048
max_chunk_size Maximum chunk size in bytes 65536
avg_chunk_size Target average chunk size 8192

🔒 Security Features

Military-grade security with multiple layers of protection

🔐

AES-256-GCM Encryption

All data chunks are encrypted with AES-256 in Galois/Counter Mode, providing both confidentiality and authenticated encryption. Keys are derived using Argon2id/scrypt from your passphrase.

✍️

Ed25519 Digital Signatures

Every snapshot and protocol message is cryptographically signed using Ed25519, ensuring authenticity and preventing tampering. Signatures are verified before acceptance.

🔑

Persistent Identity

Each peer has a stable libp2p identity based on a persistent private key. This enables trust establishment and reputation tracking across sessions.

🛡️

Access Control Lists

ACLs govern which Ed25519 public keys are authorized to introduce peers, create snapshots, or perform administrative operations, preventing unauthorized actions.

🔍

Content Verification

SHA-256 hashing ensures chunk integrity. Hash mismatches automatically trigger rejection and re-fetch from alternate sources, protecting against corruption.

🚫

Zero-Knowledge Architecture

Peers can't decrypt your data without the passphrase. The network operates on encrypted blobs, maintaining privacy even with malicious participants.

⚠️ Security Best Practices:
  • Use high-entropy passphrases (20+ characters with mixed case, numbers, symbols)
  • Protect identity.key file with strict permissions (chmod 600)
  • Regularly backup your identity key separately
  • Verify snapshot signatures before restoration
  • Use ACLs in production to limit peer introductions
  • Keep your Go dependencies updated for security patches

📡 Protocol Buffers

Efficient, type-safe serialization for all network communication

File Purpose Key Messages
common.proto Shared types and acknowledgments Ack
snapshot.proto Snapshot metadata and file entries FileEntry, SnapshotMetadata, SnapshotAnnouncement
block.proto Chunk transfer protocol BlockAnnounce, BlockRequest, BlockResponse
peer.proto Peer management messages PeerInfo, PeerAdd, PeerRemove, PeerList
auth.proto Authentication and authorization ACL, SignedMessage
identity.proto Identity records Identity
service.proto gRPC service definitions ShadowVault service RPCs
bash 
# Install protobuf compiler plugins
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest

# Generate Go bindings from proto files
protoc --go_out=. --go-grpc_out=. proto/*.proto

📚 CLI Documentation

Complete reference for all command-line tools

🔧 backup-agent

Main daemon and snapshot creation tool.

./bin/backup-agent daemon -c <config> -p <passphrase>

Start the daemon for continuous operation and peer communication

./bin/backup-agent snapshot <path> -c <config> -p <passphrase>

Create a new snapshot of the specified directory

./bin/backup-agent list -c <config> -p <passphrase>

List all local snapshots with metadata

♻️ restore-agent

Snapshot restoration and data recovery tool.

./bin/restore-agent restore <snapshot-id> <target-dir> -c <config> -p <passphrase>

Restore a snapshot to the target directory

./bin/restore-agent verify <snapshot-id> -c <config> -p <passphrase>

Verify snapshot integrity without restoring

👥 peerctl

Peer network management utility.

./bin/peerctl list -c <config> -p <passphrase>

List all known peers with their connection status

./bin/peerctl add <multiaddr> -c <config> -p <passphrase>

Manually add a peer by multiaddr (e.g., /ip4/1.2.3.4/tcp/9000/p2p/QmPeerID)

./bin/peerctl remove <peer-id> -c <config> -p <passphrase>

Remove a peer from the stored peer list

./bin/peerctl info <peer-id> -c <config> -p <passphrase>

Get detailed information about a specific peer

🔧 Common Flags

Flag Short Description Required
--config -c Path to config.yaml file Yes
--pass -p Encryption passphrase Yes
--verbose -v Enable verbose logging No
--help -h Show help message No

🧪 Testing

Comprehensive test suite for reliability and correctness

✅

Unit Tests

bash 
# Run all tests
make test

# Run with coverage
go test ./... -cover

# Verbose output
go test ./... -v
🔍

Test Coverage

Critical modules tested:

  • Crypto: AES-256-GCM encryption/decryption
  • Chunker: Content-defined boundaries
  • Identity: Key generation and signatures
  • Storage: CAS operations
  • P2P: Message validation
📊

Code Quality

bash 
# Format code
make fmt

# Check formatting
make check-fmt

# Run linter (if configured)
golangci-lint run

🔧 Troubleshooting

Common issues and their solutions

Problem Likely Cause Solution
Snapshot creation fails File permissions or missing files Check file access rights, run with appropriate privileges
Cannot fetch chunk from peer Peer offline or no announcement Verify peer connection with peerctl list, check network
Signature validation fails Wrong passphrase or tampered data Verify passphrase, reject corrupted snapshots
Identity changes unexpectedly identity.key deleted or corrupted Restore from backup, avoid deleting identity.key
Peer not discovered Bootstrap misconfig or network issue Check bootstrap addresses, verify firewall settings
High memory usage Large files or many concurrent ops Adjust chunk sizes, limit concurrent snapshots
Port already in use Another instance running Change listen_port in config or stop other instance
Database corruption Unclean shutdown or disk error Restore metadata.db from backup, use fsck
💡 Debug Tips:
  • Use -v flag for verbose logging
  • Check logs in repository_path/logs/
  • Verify network connectivity with ping and telnet
  • Test locally first before adding remote peers
  • Use docker logs for containerized deployments

🤝 Contributing

Help make ShadowVault better

🍴

1. Fork & Clone

bash 
git clone https://github.com/YOUR_USERNAME/ShadowVault-Go-Encryptor.git
cd ShadowVault-Go-Encryptor
🌿

2. Create Branch

bash 
git checkout -b feature/amazing-feature
✨

3. Make Changes

Write clean, tested code following the project's style guide. Add tests for new features.

✅

4. Test Thoroughly

bash 
make test
make check-fmt
💾

5. Commit Changes

bash 
git add .
git commit -m "feat: add amazing feature"
🚀

6. Submit PR

bash 
git push origin feature/amazing-feature

Then open a Pull Request on GitHub with a clear description.

🎯 High-Impact Areas:
  • Parallel snapshot creation and restoration
  • Enhanced peer reputation system
  • Plugin architecture for custom backends
  • Web dashboard for monitoring
  • Advanced chunking algorithms (Rabin fingerprinting)
  • Automated garbage collection
  • Snapshot diffing and incremental backups
  • Encrypted key rotation mechanisms

📁 Project Structure

Understanding the codebase organization

tree 
ShadowVault-Go-Encryptor/
├── cmd/                          # Command-line applications
│   ├── backup-agent/             # Main daemon & snapshot CLI
│   ├── backup-agent-restore/     # Restore agent CLI
│   └── peerctl/                  # Peer management CLI
├── internal/                     # Internal packages
│   ├── agent/                    # Core agent logic
│   ├── auth/                     # Authentication & ACL
│   ├── chunker/                  # Content-defined chunking
│   ├── crypto/                   # Encryption primitives
│   ├── identity/                 # Identity management
│   ├── p2p/                      # libp2p networking
│   ├── persistence/              # bbolt database wrappers
│   ├── protocol/                 # Protocol message handlers
│   ├── snapshots/                # Snapshot creation/restoration
│   ├── storage/                  # Content-addressed storage
│   └── versioning/               # Snapshot versioning
├── proto/                        # Protocol Buffer definitions
│   ├── common.proto
│   ├── snapshot.proto
│   ├── block.proto
│   ├── peer.proto
│   ├── auth.proto
│   ├── identity.proto
│   └── service.proto
├── scripts/                      # Helper shell scripts
│   ├── bootstrap.sh              # Initial setup
│   ├── snapshot.sh               # Snapshot wrapper
│   └── restore.sh                # Restore wrapper
├── tools/                        # Utility programs
│   └── hashfile.c                # SHA-256 verification tool
├── packages/                     # Web assets (this wiki)
│   ├── styles.css
│   └── script.js
├── config.yaml                   # Configuration file
├── docker-compose.yml            # Multi-node orchestration
├── Dockerfile                    # Container image definition
├── entrypoint.sh                 # Main entry point script
├── Makefile                      # Build automation
├── go.mod                        # Go module definition
├── LICENSE                       # MIT License
├── README.md                     # Project documentation
└── index.html                    # This wiki page

⚖️ License

Open source under the MIT License

text 
MIT License

Copyright (c) 2025 Son Nguyen

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
↑