Python Structured JSON Logging Best Practices for Production Services

Structured logging in Python means emitting each log entry as a machine-readable record, usually JSON, with consistent field names instead of free-form text. It makes logs searchable, filterable, and easy to correlate across services. The core practices are using the standard logging module, adding a request or correlation ID, logging exceptions with tracebacks, and never writing secrets to logs.

Why Is Structured Logging Better Than Plain Text?

Plain text logs are written for humans reading a terminal. In production, logs are read by systems: aggregators index them, dashboards chart them, and alerts trigger on them. Parsing free-form sentences with regular expressions is fragile and breaks the moment someone rewords a message.

With JSON, each field is addressable. You can query for all entries where user_id equals a value, or count errors by endpoint, without guessing how the message was worded. The message stays short and constant, while the variable data moves into fields, which also makes grouping similar events much easier.

The trade-off is readability during local development. A common solution is to render human-friendly output locally and JSON in production, switching with an environment setting while keeping the same logging calls in the code.

How Do You Set Up JSON Logging With the Standard Library?

Python's logging module is flexible enough to produce JSON through a custom formatter. Third-party libraries also exist for this purpose, and several structured logging packages build on the same foundation. Check the documentation of whichever you choose for current configuration options.

import json, logging

class JsonFormatter(logging.Formatter):
    def format(self, record):
        payload = {
            "ts": self.formatTime(record, "%Y-%m-%dT%H:%M:%S%z"),
            "level": record.levelname,
            "logger": record.name,
            "message": record.getMessage(),
        }
        if record.exc_info:
            payload["exception"] = self.formatException(record.exc_info)
        return json.dumps(payload)

handler = logging.StreamHandler()
handler.setFormatter(JsonFormatter())
logging.basicConfig(level=logging.INFO, handlers=[handler])

Write logs to standard output and let the platform collect them. Applications that manage their own log files and rotation add complexity that container orchestrators and log shippers already handle well.

One detail that surprises people is that the formatter above ignores any extra fields passed to the logger. In a real service you will want to copy those extra attributes into the payload, so that calls like logger.info with an extra dictionary produce searchable fields instead of silently dropping the data.

Which Fields and Practices Matter Most?

Consistency matters more than volume. Agree on a small set of field names across services so that queries work everywhere, and keep the types stable. A field that is sometimes a number and sometimes a string can break indexing in your log platform.

• Include a timestamp in a standard format, preferably UTC, plus level, logger name, and message.
• Add a correlation or request ID to every entry, and propagate it to downstream calls.
• Use contextvars to carry request context so you do not pass it through every function.
• Pass extra data through the extra argument or a structured logger, not by formatting it into the message.
• Log exceptions with logger.exception or exc_info=True so the traceback is captured.
• Use levels consistently: DEBUG for diagnostics, INFO for normal events, WARNING for recoverable problems, ERROR for failures.

What Should You Never Log?

Treat logs as data that many people and systems can read. Do not log passwords, tokens, API keys, full payment details, or personal data you do not need. Redact or hash sensitive fields before they reach the logger, and review third-party libraries that might log full request bodies at debug level.

Also watch the volume and cost. Logging every request body or running at DEBUG level in production can overwhelm storage and hide the important entries. Consider sampling noisy events and setting the level through configuration so you can raise it temporarily when investigating an issue.

Frequently Asked Questions

Should I use print or logging in production?

Use logging. It provides levels, formatting, handlers, and filtering, and it integrates with every major log platform. Print offers none of that structure.

How do I add a request ID to every log line?

Store the ID in a contextvar in middleware, then read it in a logging filter or formatter and add it as a field to each record.

Does JSON logging slow down my application?

Serialization has a small cost, usually minor compared with network and database work. For very hot paths, avoid building expensive payloads when the level is disabled.

Which log level should exceptions use?

ERROR for failures that need attention. Use WARNING for handled, recoverable problems you still want to track.

Summary

Emit JSON to standard output, keep field names consistent, attach correlation IDs, capture tracebacks, and keep secrets out of logs. Tune levels through configuration and check the documentation of your logging library for version-specific options.