Operating microservices: config, secrets, feature flags, tracing and logging

Configuration

Separate configuration from code. Anything that changes between environments (database address, feature limits, endpoints) should come from environment variables, config files or a config service, not be baked into the build ("build once, run anywhere").

• Keep defaults in code and overrides in the environment.
• Validate configuration at start-up and fail fast with a clear message.
• For values that must change at runtime, use a config service or watch a mounted file, and treat changes like deployments (versioned, reviewed, auditable).
• Keep config in version control (as code or templates) so changes are reviewed and reversible.

Secrets

Secrets (passwords, API keys, private keys, tokens) need stricter handling.

• Never commit them, put them in images, or print them in logs.
• Store them in a secrets manager (a cloud service, HashiCorp Vault, or an encrypted platform secret store) with access control, audit logs and encryption at rest.
• Inject at runtime (environment, mounted file or an agent), and use short-lived credentials where possible (cloud roles, dynamic database credentials).
• Rotate regularly and after any suspected leak; design services to reload secrets without downtime.
• Least privilege: each service gets only its own secrets.

Feature flags

A feature flag is a runtime switch controlling whether code runs, separate from deployment.

100%

Figure SDA-MSV-05-1 · Deploy is not release · © forcsguy.com

Uses: gradual rollout, kill switches for risky features, A/B tests, per-customer entitlements, and decoupling release from deploy (see the deployment strategies lesson). Cautions: flags multiply test combinations, can be left in place forever, and a flag service outage must fail to a safe default.

Distributed tracing

A request through a microservice system touches many services. A trace records the whole journey as a tree of spans (one per operation), joined by a trace id passed in request headers.

100%

Figure SDA-MSV-05-2 · One request traced across services · © forcsguy.com

• OpenTelemetry is the vendor-neutral standard for instrumentation (traces, metrics, logs) and the W3C Trace Context header standardises propagation.
• Propagate the id through every hop, including message queues (put it in message headers).
• Add useful attributes (user type, order size) but never secrets or personal data.
• Sampling: keep all errors and slow traces, a small share of the rest.

Centralised logging

Logs written on many short-lived instances are useless unless shipped to one searchable place.

• Write structured logs (JSON with fields: time, level, service, trace id, message, key attributes) to standard output; let the platform collect them.
• Ship to a central system (the ELK/OpenSearch stack, Loki, Splunk, cloud logging).
• Always include the trace id so logs and traces connect.
• Set levels sensibly; avoid logging personal data and secrets; set retention by value and legal need.
• Alert on log-derived signals sparingly; prefer metrics for alerting.

Health checks

Expose liveness (the process is alive; restart if not), readiness (ready to take traffic; remove from rotation if not) and optionally startup endpoints. Readiness should check what the service truly needs without cascading (a failing optional dependency should degrade, not mark the service unready for everyone).

A minimum for every service

A health endpoint, structured logs with trace ids, the standard metrics (request rate, errors, latency, saturation), propagated trace headers, config and secrets from the environment, and a runbook. The next lesson turns these signals into targets and alerts.

Real-world reading