Log file management in Python is crucial for monitoring, debugging, and auditing
applications. Python provides powerful built-in support for logging via the logging module.
Effective log file management includes not only writing logs but also handling log rotation,
formatting, levels, and storage.
Here is a complete, detailed overview of Python log file management:
� 1. Basics of Python Logging
a. Importing the Logging Module
import logging
b. Basic Configuration
logging.basicConfig(
filename='app.log',
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s'
)
c. Logging Levels
Level Description
DEBUG Detailed information for diagnostics
INFO General information
WARNING Something unexpected but not fatal
ERROR A more serious problem
CRITICAL Serious error, program may not recover
d. Usage
logging.debug("Debug message")
logging.info("Info message")
logging.warning("Warning message")
logging.error("Error message")
logging.critical("Critical message")
� 2. Logging to Files
By default, logs go to the console. To write to a file:
logging.basicConfig(
filename='app.log',
filemode='a', # 'w' for overwrite, 'a' for append
format='%(name)s - %(levelname)s - %(message)s',
level=logging.DEBUG
)
� 3. Log Rotation (Avoid Infinite Growth)
Use logging.handlers for log rotation:
a. RotatingFileHandler – Based on File Size
from logging.handlers import RotatingFileHandler
logger = logging.getLogger("MyLogger")
logger.setLevel(logging.INFO)
handler = RotatingFileHandler("app.log", maxBytes=1024*1024, backupCount=3)
formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.info("This is a rotating file log.")
maxBytes=1024*1024: 1MB max per file
backupCount=3: keeps 3 backups (e.g., app.log.1, app.log.2...)
b. TimedRotatingFileHandler – Based on Time
from logging.handlers import TimedRotatingFileHandler
logger = logging.getLogger("TimedLogger")
logger.setLevel(logging.INFO)
handler = TimedRotatingFileHandler("timed_app.log", when="midnight",
interval=1, backupCount=7)
formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.info("This is a time-rotated log.")
when='midnight': rotates at midnight
interval=1: every 1 day
backupCount=7: keeps 7 days of logs
Common when values: 'S', 'M', 'H', 'D', 'midnight', 'W0' (Monday).
� 4. Customizing Log Format
formatter = logging.Formatter(
'%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
You can include:
%(asctime)s: timestamp
%(name)s: logger name
%(levelname)s: severity level
%(message)s: log message
%(filename)s, %(lineno)d, %(module)s, etc.
� 5. Logging Exceptions
try:
1 / 0
except ZeroDivisionError:
logging.exception("An error occurred:")
.exception() logs the full traceback if called inside an exception handler.
� 6. Multiple Loggers and Handlers
You can have multiple loggers for different modules/components.
logger1 = logging.getLogger("module1")
logger2 = logging.getLogger("module2")
handler = logging.FileHandler("shared.log")
formatter = logging.Formatter('%(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
logger1.addHandler(handler)
logger2.addHandler(handler)
� 7. Directory and File Management
Ensure logs go into a dedicated folder:
import os
log_dir = "logs"
os.makedirs(log_dir, exist_ok=True)
log_path = os.path.join(log_dir, "app.log")
logging.basicConfig(
filename=log_path,
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s'
)
� 8. Log Cleanup (Old Files)
You can automate deletion of old log files with backupCount, or manually with:
import glob
import os
import time
log_files = glob.glob("logs/*.log")
now = time.time()
for file in log_files:
if os.stat(file).st_mtime < now - 7 * 86400: # older than 7 days
os.remove(file)
� 9. Security and Permissions
Set proper file permissions (chmod) if logs contain sensitive data.
Avoid logging secrets, passwords, or tokens.
Use rotating handlers to avoid disk overflow (DoS protection).
� 10. Thread-safe Logging
The logging module is thread-safe by default, so it can be used in multi-threaded
applications without additional locking.
� 11. Logging Configuration via File (INI or YAML)
You can externalize config to a file:
INI-style config (logging.conf)
[loggers]
keys=root
[handlers]
keys=fileHandler
[formatters]
keys=defaultFormatter
[logger_root]
level=DEBUG
handlers=fileHandler
[handler_fileHandler]
class=FileHandler
level=DEBUG
formatter=defaultFormatter
args=('app.log', 'a')
[formatter_defaultFormatter]
format=%(asctime)s - %(name)s - %(levelname)s - %(message)s
Load it in Python:
import logging
import logging.config
logging.config.fileConfig('logging.conf')
� 12. Logging in Web Applications
For example, in Flask:
from flask import Flask
import logging
app = Flask(__name__)
handler = RotatingFileHandler('flask.log', maxBytes=10000, backupCount=3)
handler.setLevel(logging.INFO)
app.logger.addHandler(handler)
@app.route("/")
def home():
app.logger.info("Home page accessed")
return "Hello, Logging!"
� Summary Checklist
Feature Use
File logging FileHandler, basicConfig(filename=...)
Rotation by size RotatingFileHandler
Rotation by time TimedRotatingFileHandler
Custom formatting logging.Formatter(...)
Structured config logging.config.fileConfig() or dictConfig
Exception logging logging.exception()
Directory handling os.makedirs(log_dir, exist_ok=True)
Cleanup backupCount or os.remove()
Would you like a template project structure with logging built in?
