mirror of
https://github.com/grillazz/fastapi-sqlalchemy-asyncpg.git
synced 2025-11-30 13:20:40 +03:00
Merge pull request #223 from grillazz/switch-logger-to-rotoger
Switch logger to rotoger
This commit is contained in:
Ordinary Hobbit
GitHub
30
README.md
30
README.md
@@ -24,7 +24,7 @@
|
||||
<ul>
|
||||
<li><a href="#make-will-help-you">Make will help you</a></li>
|
||||
<li><a href="#how-to-feed-database">How to feed database</a></li>
|
||||
<li><a href="#rainbow-logs-with-rich">Rainbow logs with rich</a></li>
|
||||
<li><a href="#structured-asynchronous-logging-with-rotoger">Structured & Asynchronous Logging with Rotoger</a></li>
|
||||
<li><a href="#setup-user-auth">Setup user auth</a></li>
|
||||
<li><a href="#setup-local-env-with-uv">Setup local development with uv</a></li>
|
||||
<li><a href="#import-xlsx-files-with-polars-and-calamine">Import xlsx files with polars and calamine</a></li>
|
||||
@@ -102,27 +102,21 @@ Next models were generated with https://github.com/agronholm/sqlacodegen
|
||||
|
||||
<p align="right">(<a href="#readme-top">back to top</a>)</p>
|
||||
|
||||
### Rainbow logs with rich :rainbow:
|
||||
### Structured & Asynchronous Logging with Rotoger 🪵
|
||||
|
||||
To enhance the developer experience when viewing logs with extensive information from multiple emitters
|
||||
(which are particularly useful during development), this project uses the [rich](https://github.com/Textualize/rich) library.
|
||||
Event with the superpowers of [rich](https://github.com/Textualize/rich), reading logs can be challenging.
|
||||
The [rich](https://github.com/Textualize/rich) library is highly beneficial, but integrating it properly as a logger object
|
||||
and maintaining it as a singleton took some effort.
|
||||
To elevate the logging capabilities beyond simple colored output,
|
||||
this project has transitioned to [Rotoger](https://github.com/tinyplugins/rotoger).
|
||||
This powerful library provides a comprehensive, production-ready logging setup for modern asynchronous applications,
|
||||
addressing challenges like log management, performance, and readability.
|
||||
|
||||
To address the following needs:
|
||||
- Difficulty in finding specific information in logs.
|
||||
- Avoiding the complexity of setting up an ELK stack for log management.
|
||||
- Speeding up the debugging process.
|
||||
Rotoger is built upon the excellent [structlog](http://structlog.org/) library and brings several key advantages:
|
||||
|
||||
he following steps were taken to integrate [rich](https://github.com/Textualize/rich) into the project:
|
||||
1. Configure emitters using the [logging-uvicorn.json](https://github.com/grillazz/fastapi-sqlalchemy-asyncpg/blob/main/logging-uvicorn.json)
|
||||
or use [logging-granian.json](https://github.com/grillazz/fastapi-sqlalchemy-asyncpg/blob/main/logging-granian.json) for granian
|
||||
2. Eliminate duplicates, such as SQLAlchemy echo, by using separate handlers.
|
||||
3. Maintain the logger as a singleton to prevent multiple instances.
|
||||
4. Add the --log-config ./logging-uvicorn.json parameter to Uvicorn or --log-config ./logging-granian.json to Granian.
|
||||
- `Structured Logging`: By using structlog, all log entries are generated as structured data (JSON), making them machine-readable and significantly easier to query, parse, and analyze in log management systems.
|
||||
- `Asynchronous & Non-Blocking`: Designed for async frameworks like FastAPI, Rotoger performs logging operations in a non-blocking manner. This ensures that I/O-bound logging tasks do not hold up the event loop, maintaining high application performance.
|
||||
- `High-Performance JSON`: It leverages orjson for serialization, which is one of the fastest JSON libraries for Python. This minimizes the overhead of converting log records to JSON strings.
|
||||
- `Built-in Log Rotation`: Rotoger implements its own log rotation mechanism in Python, allowing you to manage log file sizes and retention policies directly within your application without relying on external tools like logrotate.
|
||||
|
||||
|
||||
This setup solves common logging pain points in production environments, such as managing large log files, ensuring logs don't impact performance, and making logs easily searchable.
|
||||
|
||||
<p align="right">(<a href="#readme-top">back to top</a>)</p>
|
||||
|
||||
|
||||
@@ -1,98 +0,0 @@
|
||||
import logging
|
||||
import os
|
||||
from logging.handlers import RotatingFileHandler
|
||||
from pathlib import Path
|
||||
|
||||
import orjson
|
||||
import structlog
|
||||
from attrs import define, field
|
||||
from whenever._whenever import Instant
|
||||
|
||||
from app.utils.singleton import SingletonMetaNoArgs
|
||||
|
||||
|
||||
class RotatingBytesLogger:
|
||||
"""Logger that respects RotatingFileHandler's rotation capabilities."""
|
||||
|
||||
def __init__(self, handler):
|
||||
self.handler = handler
|
||||
|
||||
def msg(self, message):
|
||||
"""Process a message and pass it through the handler's emit method."""
|
||||
if isinstance(message, bytes):
|
||||
message = message.decode("utf-8")
|
||||
|
||||
# Create a log record that will trigger rotation checks
|
||||
record = logging.LogRecord(
|
||||
name="structlog",
|
||||
level=logging.INFO,
|
||||
pathname="",
|
||||
lineno=0,
|
||||
msg=message.rstrip("\n"),
|
||||
args=(),
|
||||
exc_info=None,
|
||||
)
|
||||
|
||||
# Check if rotation is needed before emitting
|
||||
if self.handler.shouldRollover(record):
|
||||
self.handler.doRollover()
|
||||
|
||||
# Emit the record through the handler
|
||||
self.handler.emit(record)
|
||||
|
||||
# Required methods to make it compatible with structlog
|
||||
def debug(self, message):
|
||||
self.msg(message)
|
||||
|
||||
def info(self, message):
|
||||
self.msg(message)
|
||||
|
||||
def warning(self, message):
|
||||
self.msg(message)
|
||||
|
||||
def error(self, message):
|
||||
self.msg(message)
|
||||
|
||||
def critical(self, message):
|
||||
self.msg(message)
|
||||
|
||||
|
||||
class RotatingBytesLoggerFactory:
|
||||
"""Factory that creates loggers that respect file rotation."""
|
||||
|
||||
def __init__(self, handler):
|
||||
self.handler = handler
|
||||
|
||||
def __call__(self, *args, **kwargs):
|
||||
return RotatingBytesLogger(self.handler)
|
||||
|
||||
|
||||
@define
|
||||
class AppStructLogger(metaclass=SingletonMetaNoArgs):
|
||||
_logger: structlog.BoundLogger = field(init=False)
|
||||
|
||||
def __attrs_post_init__(self):
|
||||
_log_date = Instant.now().py_datetime().strftime("%Y%m%d")
|
||||
_log_path = Path(f"{_log_date}_{os.getpid()}.log")
|
||||
_handler = RotatingFileHandler(
|
||||
filename=_log_path,
|
||||
maxBytes=10 * 1024 * 1024, # 10MB
|
||||
backupCount=5,
|
||||
encoding="utf-8",
|
||||
)
|
||||
structlog.configure(
|
||||
cache_logger_on_first_use=True,
|
||||
wrapper_class=structlog.make_filtering_bound_logger(logging.INFO),
|
||||
processors=[
|
||||
structlog.contextvars.merge_contextvars,
|
||||
structlog.processors.add_log_level,
|
||||
structlog.processors.format_exc_info,
|
||||
structlog.processors.TimeStamper(fmt="iso", utc=True),
|
||||
structlog.processors.JSONRenderer(serializer=orjson.dumps),
|
||||
],
|
||||
logger_factory=RotatingBytesLoggerFactory(_handler),
|
||||
)
|
||||
self._logger = structlog.get_logger()
|
||||
|
||||
def get_logger(self) -> structlog.BoundLogger:
|
||||
return self._logger
|
||||
Reference in New Issue
Block a user