P2P Technologies Knowledge Base

Summary

Practical, vendor-neutral overview of peer-to-peer (P2P) systems: concepts, protocols, design patterns, threats, and implementation stacks.

Focus on decentralized overlays, content routing, NAT traversal, data sync, privacy, incentives, and operations.

Scope and audience

For engineers, architects, and researchers designing or integrating P2P capabilities.

Covers application overlays (BitTorrent/libp2p/Hypercore), privacy networks (Tor/I2P), real-time (WebRTC), content networks (IPFS), and social/data sync (SSB/Matrix p2p modes).

Taxonomy of decentralization

Centralized: clients talk to a single service.

Federated: servers interoperate under distinct authorities (e.g., ActivityPub/Matrix).

Peer-to-peer: end hosts route, store, and coordinate without trusted intermediaries; may still use bootstraps or relays.

Hybrid: mixes of P2P plus assist servers (trackers, relays, rendezvous, introducers).

Core building blocks

Overlay networks: logical links over IP; structured (DHTs) vs unstructured (random graph, gossip).

Naming and addressing: location-addressed (IP:port) vs content-addressed (hashes, CIDs, Merkle DAGs).

Routing/lookup: map keys->peers (Kademlia) or peers->content via indices, trackers, or rendezvous.

Persistence and replication: availability under churn via k-replication, erasure coding.

Security: authenticated transport (TLS/Noise), message signing, capability scoping, rate limits.

Overlay networks and DHTs

Kademlia (XOR metric): iterative/recursive lookup, k-buckets for log-scaled neighbor sets; robust under churn and common in modern P2P.

Chord (ring), Pastry/Tapestry (prefix routing): historical structured overlays with logarithmic routing.

Unstructured: random walks, flooding with TTL, supernodes/ultrapeers for scalability (e.g., Gnutella).

Enhancements: proximity routing, neighbor diversity, sloppy hashing, replication strategies, DHT poisoning defenses.

Transport and connectivity

Transports: TCP, UDP, QUIC; uTP/LEDBAT for low-priority bulk transfer; WebRTC data channels on browsers.

NAT traversal: STUN (discover), TURN (relay), ICE (path selection); UDP hole punching preferred; TCP/QUIC hole punching increasingly supported.

Relays: application relays or TURN for last-resort reachability; multiplexing to conserve ports.

Congestion control: CUBIC/BBR in TCP; QUIC CC; LEDBAT for background bulk.

Data structures and content addressing

Hash-linked structures: Merkle trees/DAGs for integrity, deduplication, and partial verification.

Content IDs (CIDs): self-describing multihash/multicodec identifiers (e.g., IPFS) enabling offline verification.

Chunking strategies: fixed-size vs rolling (Rabin) for diff-friendliness; manifests and car/pack files for batching.

Versioning: immutable data plus append-only logs; references via pins, indices, or name systems.

Storage and distribution

Swarming: rarest-first, tit-for-tat, piece selection to optimize availability and fairness (BitTorrent lineage).

Caches and pins: ephemeral vs durable retention; garbage-collection policies; provider records in DHTs.

Large data: erasure coding vs replication; locality-aware fetching; opportunistic caching; disk layout considerations.

Incentives: credits, bilateral tit-for-tat, storage markets (e.g., Filecoin), bandwidth markets, vouchers.

Real-time messaging and PubSub

Gossip protocols: epidemic dissemination; mesh-based PubSub (e.g., GossipSub) with scoring and backoff.

Topics and discovery: explicit topics, content-interest filters, rendezvous points.

Ordering and reliability: at-most-once by default; application-layer acks; causal ordering via vector/lamport clocks if needed.

State sync and collaboration

Eventual consistency by default in P2P; conflict resolution via CRDTs or app-specific merges.

CRDT families: GCounter/PNCounter, OR-Set/2P-Set, LWW-Register, RGA/WOOT for sequences, Map/Delta-CRDTs.

Log shipping: append-only logs (hypercore), snapshotting, compaction; anti-entropy sync via digests.

Identity, authentication, and capabilities

Keys as identities: long-term Ed25519/SECP256k1 keys; rotation via key delegation chains.

Decentralized identifiers (DIDs) and verifiable credentials for portable identity.

Capability-based security: object capabilities (ocaps), UCAN, macaroons; least privilege and attenuated proofs.

Authorization models: allowlists, reputations, stake-bonded permissions, human-verification where appropriate.

Privacy, anonymity, and censorship resistance

Onion routing (Tor) and garlic routing (I2P) for unlinkability; target vs source anonymity trade-offs.

Metadata reduction: cover traffic, constant-rate padding, stream multiplexing, query privacy in DHTs (e.g., PIR techniques, two-phase lookups).

Name systems and indirection: .onion/.i2p, petnames, self-certifying names; rendezvous obfuscation and domain fronting (where legal and ethical).

Security threats and mitigations

Sybil attacks: admission control, proof-of-work/stake, social trust, costed identities, rate limits.

Eclipse/routing attacks: neighbor diversity, bucket hardening, signed records, path randomization.

Spam/DoS: reputation/score-based penalties, challenge puzzles, quotas, priority lanes.

Content poisoning: authenticated metadata, merkle proofs, multiple-provider verification, trust pinning.

Traffic correlation/deanonymization: entry guards, path constraints, timing obfuscation.

Design patterns and architectures

Content-addressed networks: immutable data plus mutable pointers (e.g., IPNS, signed heads).

Rendezvous and trackers: decouple discovery from transfer for scalability and flexibility.

Superpeer hybrids: elevate capable nodes for indexing/relay while preserving open participation.

Edge-first: offline-first design; delay-tolerant networking (DTN) patterns for intermittency.

Implementation stacks and notable systems

libp2p: modular transports (TCP/QUIC/WebRTC), multiplexers (mplex/yamux), crypto (Noise/TLS), DHT, PubSub, identify/autonat/hole punching.

IPFS/Filecoin: content addressing (CIDs, UnixFS), Bitswap/GraphSync, pinning, provider records; storage marketplace (Filecoin).

Hypercore/Hyperdrive (Holepunch): append-only logs, hyperswarm DHT, Hyperspace; Dat lineage for data sync.

BitTorrent/uTP: swarming downloads, trackers/DHT (BEP-5), magnet links, tit-for-tat incentives.

Secure Scuttlebutt (SSB): append-only per-user logs, gossip replication, social graph for routing/permissions.

Matrix: federated messaging with emerging p2p modes (P2P MSCs), Olm/Megolm E2E crypto.

Tor/I2P: anonymity networks with onion/garlic routing; rendezvous services; censorship resistance.

GNUnet, Freenet, Tahoe-LAFS, Syncthing, Nostr: varied takes on secure routing, storage, sync, and gossip.

Testing, metrics, and operations

Testnets/simulations: emulation (Mininet, ns-3), chaos testing under churn, NAT lab matrices.

Telemetry: per-peer session stats, DHT table health, message latency, failure rates; privacy-preserving aggregate metrics.

Upgrades and compatibility: multicodecs, negotiate features; rolling keys; split-brain and network partitions testing.

Compliance, abuse handling, and safety

Legal considerations: content liability, export controls, encryption laws, safe-harbor frameworks.

Abuse processes: takedown mechanisms for indexes/relays, community moderation, reputation-driven filtering.

Safety by design: default private data sharing, explicit consent, rate limits, resource caps.

Practical guidance and checklists

Bootstrap: multiple well-known boot peers, rotating lists, opportunistic peer exchange.

NAT: implement ICE with UDP first; fallback to relays; validate hole punching; detect symmetric NATs.

Security: sign everything that can be replayed or routed; limit unauthenticated surfaces; defend against amplification.

Performance: shard topics, bound fanout, adaptive gossip; backpressure; prioritize interactive traffic.

Resilience: diversify peers/transports; exponential backoff; periodic re-randomization of neighbors.

Glossary

DHT: Distributed Hash Table for key->peer mappings.

CID: Content Identifier (hash-based, self-describing).

ICE/STUN/TURN: NAT traversal and relay suite for peer connectivity.

CRDT: Conflict-free Replicated Data Type for mergeable state.

Gossip: epidemic dissemination protocol with probabilistic guarantees.