Syscall Auditing at Scale at Slack

Ideas you will meet

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

Web Security

OWASP risks, XSS, CSRF, CORS and secure design.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Networking and Web Security Deep Dive · 14 minWeb attacks: XSS, CSRF, injection, SSRF and clickjackingHow each major web attack works in plain terms and the few defences that stop most of them.Read hereClose

  Attacker mindset

  Most web attacks share one cause: the application trusts input, or trusts the browser, more than it should. The fixes are few and repeatable. This lesson covers five of the most important attacks; the full checklist is in the OWASP lesson. (Authentication and sessions are in the passwords and securing-an-API lessons.)

  1. Cross-site scripting (XSS)

  What: attacker-controlled text is put into a page and the browser runs it as script. The script runs as your site, so it can read the page, act as the user and steal tokens.

  Types: stored (saved in your database, such as a comment), reflected (echoed from a URL parameter), DOM-based (client-side code writes untrusted data into the page unsafely).

  // vulnerable: untrusted text inserted as HTML
  el.innerHTML = comment.text;             // <img src=x onerror=stealCookies()> runs!
  
  // safe: insert as text
  el.textContent = comment.text;
  

  Defences

  • Escape output for its context (HTML, attribute, JavaScript, URL). Modern frameworks (React, Vue and others) escape by default; avoid raw-HTML escape hatches such as dangerouslySetInnerHTML or v-html with untrusted data.
  • If you must allow rich text, sanitise with a well-maintained library and an allow-list.
  • A Content Security Policy header limits where scripts can come from, reducing damage if something slips through.
  • Mark session cookies HttpOnly so scripts cannot read them.

  2. Cross-site request forgery (CSRF)

  What: a malicious site makes the victim's browser send a request to your site, and the browser automatically attaches the victim's cookies. If the user is logged in, the action happens (change email, transfer money).

  100%

  Figure SDA-NSD-03-1 · How CSRF works and how a token stops it · © forcsguy.com

  Defences

  • SameSite cookies (Lax or Strict) stop browsers sending cookies on most cross-site requests; now the browser default for many cases but set it explicitly.
  • Anti-CSRF tokens: a random value tied to the session that must accompany state-changing requests.
  • Never change state with GET; check the Origin header; require re-authentication for sensitive actions.
  • APIs that use an Authorization header (not cookies) are not vulnerable in the same way, because the browser does not attach it automatically.

  3. Injection (SQL, command, and others)

  What: input is concatenated into an interpreter's command, so the input changes the command's meaning.

  // vulnerable
  db.query(`SELECT * FROM users WHERE email = '${email}'`);   // email = ' OR '1'='1  returns everyone
  
  // safe: a parameterised query keeps data and command separate
  db.query("SELECT * FROM users WHERE email = $1", [email]);
  

  Defences

  • Parameterised queries / prepared statements for SQL; ORMs do this for normal calls (beware raw-SQL escape hatches).
  • Avoid building shell commands from input; use library calls with argument arrays, not strings.
  • Validate input against an allow-list of expected formats, and apply least privilege to database accounts (the web app should not be able to drop tables).
  • The same pattern applies to NoSQL queries, LDAP, XML and template engines. Rule: never build code or queries by joining strings with untrusted data.

  4. Server-side request forgery (SSRF)

  What: the app fetches a URL supplied by the user (a preview, an image import, a webhook test). The attacker supplies an internal address; the server, which sits inside your network, fetches it and may return secrets, such as cloud instance credentials from the metadata endpoint, or probe internal services.

  Defences

  • Prefer an allow-list of destinations; if not possible, block private, loopback and link-local ranges after resolving the name, and re-check on every redirect (and beware of DNS tricks that change the answer between check and use).
  • Allow only http and https; set short timeouts and size limits.
  • Run fetchers in an isolated network with no route to sensitive services; use cloud settings that require a session token for metadata access.
  • Do not return the raw response to the user.

  5. Clickjacking

  What: your page is loaded in a hidden or disguised frame on an attacker's site, and the user is tricked into clicking buttons they cannot see (like "delete account").

  Defences

  • Send Content-Security-Policy: frame-ancestors 'none' (or 'self'), and the older X-Frame-Options: DENY.
  • Require confirmation for sensitive actions.

  One table to remember

  Attack	Root cause	Main defence
  XSS	Untrusted data becomes script in a page	Output escaping, CSP, HttpOnly cookies
  CSRF	Browser sends cookies on forged requests	SameSite cookies, anti-CSRF tokens
  Injection	Data joined into a command	Parameterised queries, validation, least privilege
  SSRF	Server fetches attacker-chosen URLs	Allow-lists, block internal ranges, isolation
  Clickjacking	Page framed by another site	frame-ancestors / X-Frame-Options

  Habits

  • Defence in depth: assume one layer will fail.
  • Validate on the server always; client checks are only for user experience.
  • Use framework defaults and keep dependencies updated.
  • Log and alert on suspicious input patterns and blocked requests.
  • Test: automated scanners, code review checklists and, for important systems, independent penetration tests.

  Real-world reading

  Related reading

  • MDN: Cross-Origin Resource Sharing (CORS)MDN contributors · MozillaDocumentation
  • OWASP Cheat Sheet SeriesOWASP · OWASP FoundationDocumentation
  • OWASP Top 10OWASP · OWASP FoundationDocumentation
  • Introduction to JSON Web TokensAuth0 · jwt.ioDocumentation

  Show 6 moreShow fewer

  • High Performance Browser NetworkingIlya Grigorik · O'ReillyBooks
  • RFC 9114: HTTP/3IETFDocumentation
  • Building Secure and Reliable SystemsGoogle engineers · O'Reilly / GoogleBooks
  • RFC 6749: The OAuth 2.0 Authorization FrameworkIETFDocumentation
  • Syscall Auditing at Scale at SlackScale at Slack engineers · Scale at SlackCase Studies
  • MySQL High Availability at GitHubGitHub Engineering · GitHubCase Studies

  Open this lesson with its quiz and exercise