Quora Engineering Blog

Ideas you will meet

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Networking and Web Security Deep Dive · 12 minNetwork layers, TCP, UDP, ports and socketsHow data travels: the layered models, what TCP and UDP promise, how ports and sockets let programs talk, and why this matters for design.Read hereClose

  Layers: why networks are divided

  Networking is split into layers, each using the one below and hiding its details. Two models are used: the 7-layer OSI model (a teaching reference) and the 4-layer TCP/IP model (what the internet actually uses).

  TCP/IP layer	OSI layers	What it does	Examples
  Application	5 to 7 (session, presentation, application)	Protocols programs speak	HTTP, DNS, SMTP, TLS
  Transport	4	Delivery between programs	TCP, UDP, QUIC
  Internet	3 (network)	Addressing and routing between networks	IP, ICMP
  Link	1 and 2 (physical, data link)	Moving frames on one local network	Ethernet, Wi-Fi

  100%

  Figure SDA-NSD-01-1 · Layers wrapping a web request · © forcsguy.com

  Each layer adds its own header on the way down (encapsulation) and removes it on the way up. Routers work at the internet layer (they look at IP addresses); switches at the link layer; load balancers and proxies at transport or application (see the load balancing lesson).

  IP, in short

  • IP address identifies a network interface; IPv4 has 32 bits (about 4.3 billion addresses, now scarce, hence NAT) and IPv6 has 128 bits.
  • Packets are routed hop by hop; each router decides the next hop. IP is best effort: packets can be lost, duplicated or reordered. Higher layers fix that if needed.
  • A subnet is a block of addresses sharing a prefix (for example 10.0.1.0/24), the basis of private networks and cloud network design.
  • DNS maps names to addresses (covered in the networking lesson of the foundations course).

  TCP: reliable streams

  TCP gives a reliable, ordered byte stream:

  100%

  Figure SDA-NSD-01-2 · TCP connection life · © forcsguy.com

  • Handshake costs a round trip before data (a reason connection reuse and HTTP keep-alive matter; TLS adds more).
  • Sequence numbers and acknowledgements let it retransmit lost data and reorder.
  • Flow control (receive window) stops a fast sender overwhelming a slow receiver.
  • Congestion control slows down when the network is congested; a new connection starts slowly (slow start), so short connections to far servers are slow.
  • Head-of-line blocking: one lost segment stalls everything behind it on that connection. HTTP/3 over QUIC addresses this (see the HTTP evolution lesson).

  UDP: just datagrams

  UDP sends independent datagrams with no connection, ordering or retransmission. It is lightweight and low latency, used for DNS, voice and video, games, and as the base of QUIC, which rebuilds reliability and encryption on top of UDP with per-stream independence. Choose UDP when the application can tolerate loss or wants its own reliability rules.

  Ports and sockets

  A port (0 to 65535) identifies a program on a machine: HTTP 80, HTTPS 443, SSH 22, DNS 53, PostgreSQL 5432, Redis 6379. A socket is the endpoint a program uses: an IP address plus a port plus a protocol. A TCP connection is uniquely identified by the four values source IP, source port, destination IP, destination port; that is why one server port can serve thousands of clients.

  import net from "node:net";
  const server = net.createServer((socket) => {         // one socket per connected client
    socket.on("data", (buf) => socket.write(`echo: ${buf}`));
  });
  server.listen(9000);                                   // listening socket on port 9000
  

  Operating systems limit open sockets (file descriptors) and ephemeral ports; heavy services tune these and reuse connections.

  Why designers care

  • Latency is dominated by round trips: handshakes, DNS, TLS. Reuse connections, use CDNs, keep data close.
  • Bandwidth versus latency: a fat pipe does not make a round trip shorter (see the latency lesson).
  • Packet loss hurts TCP throughput; mobile networks need resilient protocols.
  • Choosing a transport: TCP for most request/response work, UDP/QUIC for real-time or when you need per-stream independence.
  • Debugging: ping and traceroute for reachability and path, dig for DNS, curl -v for HTTP and TLS, ss or netstat for sockets, tcpdump or Wireshark for packets.

  Real-world reading

  Related reading

  • Cloudflare BlogCloudflare engineers · CloudflareBlogs
  • Chrome Developers BlogGoogle engineers · GoogleBlogs
  • Android Developers BlogGoogle engineers · GoogleBlogs
  • The Road to QUICCloudflareBlogs

  Show 6 moreShow fewer

  • Julia Evans' BlogJulia Evans · jvns.caBlogs
  • RFC 9000: QUICIETFDocumentation
  • Web VitalsGoogle web.dev · GoogleDocumentation
  • Active-Active for Multi-Regional ResiliencyNetflix Engineering · NetflixCase Studies
  • IBM developerWorks Engineering BlogIBM developerWorks engineers · IBM developerWorksBlogs
  • Quora Engineering BlogQuora engineers · QuoraBlogs

  Open this lesson with its quiz and exercise