How to debug and monitor your entire application using Python logging module

As a software developer, you need to debug and monitor your application to ensure that it's working correctly. One tool that can help you with this is Python's logging module. The logging module provides a way to record events that occur in your application, such as errors or warnings, and to output them to various destinations, such as the console, a file, or a remote server. In this article, we'll cover the basics of using the logging module in Python and provide code examples that demonstrate how to use it effectively.

Logging Basics:

The logging module provides a set of functions and classes that allow you to log messages in your application. The basic concept of logging is straightforward: you create a logger object, and then you use that logger object to log messages at different levels of severity.

Here's an example of how to use the logging module to log messages:

import logging

logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)

logger.debug('This is a debug message')
logger.info('This is an info message')
logger.warning('This is a warning message')
logger.error('This is an error message')
logger.critical('This is a critical message')

In this example, we first import the logging module. We then call the basicConfig function to set the logging level to DEBUG, which means that all messages of severity DEBUG and above will be logged. We also set the logging format to include the current time, the logging level, and the message.

We then create a logger object using the getLogger function and pass in the name of the current module (name). We can use this logger object to log messages at different levels of severity using the debug, info, warning, error, and critical methods.

Logging Levels:

The logging module provides several levels of severity that you can use to log messages. These levels, in order of increasing severity, are:

DEBUG
INFO
WARNING
ERROR
CRITICAL

By default, the logging module logs messages of severity WARNING and above. You can change the logging level using the basicConfig function, as shown in the previous example. 

Here's an example of how to set the logging level to INFO:

import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

logger.debug('This is a debug message')
logger.info('This is an info message')

In this example, we set the logging level to INFO, which means that only messages of severity INFO and above will be logged. When we call the debug method, the message is not logged because it's below the logging level.

Logging to a File:

You can log messages to a file using the FileHandler class in the logging module. Here's an example of how to log messages to a file:

import logging

logging.basicConfig(filename='example.log', level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)

logger.debug('This is a debug message')
logger.info('This is an info message')

In this example, we set the logging level to DEBUG and specify a file name for the log file (example.log). We then use the getLogger function to create a logger object and log messages using the debug and info methods. The messages are logged to the file specified by the filename parameter.

Logging to Multiple Destinations:

You can log messages to multiple destinations using the handlers in the logging module. For example, you might want to log messages to both the console and a file.

Here's an example of how to do this:

import logging

logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(messages)')
logger = logging.getLogger(name)

Create a console handler and set the logging level to DEBUG

ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)

Create a file handler and set the logging level to INFO

fh = logging.FileHandler('example.log')
fh.setLevel(logging.INFO)

Create a formatter and add it to the handlers

formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
ch.setFormatter(formatter)
fh.setFormatter(formatter)

Add the handlers to the logger

logger.addHandler(ch)
logger.addHandler(fh)

Log some messages

logger.debug('This is a debug message')
logger.info('This is an info message'

In this example, we first set up the basic logging configuration with a logging level of DEBUG and a message format that includes the time, logging level, and message. We then create a logger object and use it to log messages.

Next, we create a console handler and a file handler, and we set their logging levels to DEBUG and INFO, respectively. We also create a formatter and add it to the handlers. Finally, we add the handlers to the logger.

When we log messages using the logger object, the messages will be output to both the console and the file, with the formatting specified by the formatter.

Using Loggers:

In addition to using the basicConfig function to set up logging, you can also create and configure logger objects directly. This can be useful if you want more control over the logging process.

Here's an example of how to create and configure a logger object:

import logging

# Create a logger object with a specific name
logger = logging.getLogger('my_logger')

# Set the logging level to INFO
logger.setLevel(logging.INFO)

# Create a console handler and set the logging level to DEBUG
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)

# Create a formatter and add it to the handler
formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
ch.setFormatter(formatter)

# Add the handler to the logger
logger.addHandler(ch)

# Log some messages
logger.debug('This is a debug message')
logger.info('This is an info message')

In this example, we create a logger object with a specific name ('my_logger') and set its logging level to INFO. We then create a console handler with a logging level of DEBUG and a formatter, and we add it to the logger object.

When we log messages using the logger object, they will be output to the console with the formatting specified by the formatter.

we've covered the basics of using Python's logging module to debug and monitor your application. We've covered logging levels, logging to files and multiple destinations, and using logger objects directly. With this knowledge, you should be able to effectively use the logging module to debug and monitor your Python applications.

Here are some additional tips and best practices to keep in mind when using the logging module:

Use descriptive log messages: When writing log messages, make sure they are descriptive and provide enough context to understand what's happening in the application. This can be especially helpful when troubleshooting issues.

import logging

logger = logging.getLogger(__name__)

def process_data(data):
    logger.info(f"Processing {len(data)} records")
    for record in data:
        logger.debug(f"Processing record: {record}")
        # do some processing

In this example, we're using descriptive log messages to provide context about the data processing that's happening. The logger.info message logs the number of records being processed, while the logger.debug message logs the details of each individual record.

Use different logging levels for different types of messages: As we discussed earlier, there are different logging levels for different types of messages. Use these levels appropriately to ensure that you're capturing the right level of detail in your logs.

import logging

logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)

ch = logging.StreamHandler()
ch.setLevel(logging.INFO)

formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
ch.setFormatter(formatter)

logger.addHandler(ch)

def process_data(data):
    logger.info(f"Processing {len(data)} records")
    for record in data:
        logger.debug(f"Processing record: {record}")
        # do some processing

In this example, we're using different logging levels for different types of messages. We're setting the logger level to DEBUG, which means it will capture all log messages. However, we're setting the console handler level to INFO, which means it will only capture messages with a level of INFO or higher. This ensures that we're capturing the right level of detail in our logs without overwhelming our console with too much information.

Log exceptions with stack traces: When an exception is raised in your code, log the exception along with the stack trace. This can be very helpful in debugging issues and understanding how the exception was raised.

import logging

logger = logging.getLogger(__name__)

def process_data(data):
    try:
        # do some processing
    except Exception as e:
        logger.error("An error occurred while processing the data", exc_info=True)

In this example, we're using the exc_info parameter of the logger.error function to log the exception along with the stack trace. This can be very helpful in understanding how the exception was raised and where in the code it occurred.

Use rotating log files: If you're logging a large amount of data, consider using rotating log files. This will allow you to split your logs into multiple files based on size or time intervals, which can make them easier to manage.

import logging
from logging.handlers import RotatingFileHandler

logger = logging.getLogger(__name__)

handler = RotatingFileHandler('example.log', maxBytes=1000000, backupCount=5)
handler.setLevel(logging.DEBUG)

formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)

logger.addHandler(handler)

def process_data(data):
    logger.info(f"Processing {len(data)} records")
    for record in data:
        logger.debug(f"Processing record: {record}")
        # do some processing

In this example, we're using a RotatingFileHandler to split our logs into multiple files based on size. The maxBytes parameter specifies the maximum size of each log file, while the backupCount parameter specifies the maximum number of backup files to keep. This ensures that our log files don't get too large and are easy to manage.

Use a centralized logging solution: If you have multiple instances of your application running, consider using a centralized logging solution to aggregate all of your logs in one place. This can make it easier to monitor and troubleshoot issues across all instances of your application.

import logging
import logging.handlers

logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)

handler = logging.handlers.SysLogHandler(address=('localhost', 514))
handler.setLevel(logging.INFO)

formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)

logger.addHandler(handler)

def process_data(data):
    logger.info(f"Processing {len(data)} records")
    for record in data:
        logger.debug(f"Processing record: {record}")
        # do some processing

In this example, we're using a SysLogHandler to send our log messages to a centralized logging server over the network. This is useful in a distributed environment where multiple instances of an application may be running on different machines. By centralizing the logs, we can easily monitor the health of the application and troubleshoot issues.

Use custom log levels:

import logging

MY_LOG_LEVEL = 15
logging.addLevelName(MY_LOG_LEVEL, "MY_LOG_LEVEL")

logger = logging.getLogger(__name__)

def process_data(data):
    logger.log(MY_LOG_LEVEL, f"Processing {len(data)} records")
    for record in data:
        logger.log(MY_LOG_LEVEL, f"Processing record: {record}")
        # do some processing

In this example, we're using a custom log level called MY_LOG_LEVEL. We first add this log level to the logging module using the addLevelName function. We then use the logger.log function to log messages with this custom log level. This can be useful if we want to log messages at a level that's more detailed than DEBUG.

The logging module in Python provides a powerful and flexible way to debug and monitor your application. By following best practices like using descriptive log messages, using different logging levels for different types of messages, logging exceptions with stack traces, using rotating log files, using a centralized logging solution, and using custom log levels, you can create effective and easy-to-use logs that will help you diagnose problems and maintain the health of your application.