docs: Describe configuration and limitations

Author: ErikKalkoken

Makefile

@@ -1,3 +1,6 @@
 coverage:
 	coverage run -m unittest discover tests/
 	coverage report -m
+
+docs:
+	pdoc -d=google vlogs_handler

README.md

@@ -1,16 +1,13 @@
-# victorialogs-handler
+# python-victorialogs-handler
 
 A high-performance Python log handler for VictoriaLogs.
 
-[![release](https://img.shields.io/pypi/v/python-victorialogs-handler?label=release)](https://pypi.org/project/python-victorialogs-handler/)
-[![python](https://img.shields.io/pypi/pyversions/python-victorialogs-handler)](https://pypi.org/project/python-victorialogs-handler/)
+[![release](https://img.shields.io/pypi/v/victorialogs-handler?label=release)](https://pypi.org/project/victorialogs-handler/)
+[![python](https://img.shields.io/pypi/pyversions/victorialogs-handler)](https://pypi.org/project/victorialogs-handler/)
 [![CI/CD](https://github.com/ErikKalkoken/python-victorialogs-handler/actions/workflows/cicd.yaml/badge.svg)](https://github.com/ErikKalkoken/python-victorialogs-handler/actions/workflows/cicd.yaml)
 [![codecov](https://codecov.io/gh/ErikKalkoken/python-victorialogs-handler/graph/badge.svg?token=2pPb3lid2k)](https://codecov.io/gh/ErikKalkoken/python-victorialogs-handler)
 [![license](https://img.shields.io/badge/license-MIT-green)](https://gitlab.com/ErikKalkoken/python-victorialogs-handler/-/blob/master/LICENSE)
 
-> [!IMPORTANT]
-> STATUS: In development. The API may still change.
-
 ## Description
 
 **victorialogs-handler** is a high-performance Python log handler tailored for [VictoriaLogs](https://victoriametrics.com/products/victorialogs/). It integrates seamlessly with Python’s native logging module, allowing you to stream log events to a VictoriaLogs instance with minimal configuration.

src/vlogs_handler/__init__.py

@@ -11,6 +11,6 @@
 from .handler import VictoriaLogsHandler  # noqa: F401
 
 __title__ = "VictoriaLogs Handler"
-__version__ = "0.1.0a1"
+__version__ = "0.1.0a2"
 
 __all__ = ["VictoriaLogsHandler"]

src/vlogs_handler/handler.py

@@ -49,33 +49,43 @@ class VictoriaLogsHandler(logging.Handler):
     """A handler class which dispatches logging records to a VictoriaLogs server.
 
     Args:
-        batch_size: New logs are submitted immediately once this threshold is reached.
-        flush_interval: New logs are submitted every x seconds.
+        batch_size: Triggers flushing the buffer once this threshold is reached.
         buffer_size: Maximum number of logs the buffer can hold.
             If buffer_size <= 0, the size is unlimited (not recommended).
             When the buffer is full any new logs will be discarded.
-            100 000 logs consume approx. 80-100 MB of RAM.
-            Tip: The buffer should be large enough to hold incoming logs
-            while the log server is down for regular maintenance.
+            50 000 logs consume approx. 40-50 MB of RAM.
+            Note that each Python process the handler is running on has it's own buffer.
+            Tip: At a minimum the combined buffer of all handlers should be
+            large enough to hold incoming logs while the last chunk
+            is being transferred to the log server.
+            Ideally, it should be able to hold incoming logs while
+            the vlogs server is down for regular maintenance.
         chunk_size: Maximum number of logs send per request to the log server.
+        flush_interval: Triggers flushing the buffer at an interval of x seconds.
         record_to_stream: A function that returns the value for the `stream`field
             for a log record. The default will return the name of the top package.
         request_timeout: Timeout when sending a request to the vlogs server in seconds.
+        shutdown_timeout: Timeout when waiting for the worker
+            to flush the buffer during shutdown.
         start_worker: Whether to start the worker at initialization.
             Alternatively, the worker can be started later by calling `start()`.
-        shutdown_timeout: Timeout when waiting for the worker to shut down.
         url: URL of the vlogs server, e.g. `"http://localhost:9428"`
+
+    Raises:
+        ValueError: If the validation of an argument fails.
+        RuntimeError: If the handler is not started in a module.
     """
 
     def __init__(
         self,
+        *,
         batch_size: int = 125,
-        buffer_size: int = 100_000,
+        buffer_size: int = 50_000,
         chunk_size: int = 1_000,
         flush_interval: float = 5.0,
         record_to_stream: Optional[Callable[[logging.LogRecord], str]] = None,
         request_timeout: float = 3.0,
-        shutdown_timeout: float = 2.0,
+        shutdown_timeout: float = 4.0,
         start_worker: bool = True,
         url: str = "http://localhost:9428",
     ):

src/vlogs_handler/technical.md

@@ -1,32 +1,80 @@
-# Technical overview
+# Technical documentation
 
-This section gives a technical overview of how the vlogs handler works.
+This section describes how the handler works and how it can be configured.
 
 ## Process Flow
 
 The high-level process flow of the vlogs handler is as follows:
 
-1. When a a log event is received, it is converted into JSON and stored
-    in the buffer
-1. At the tick of an interval (e.g. 5 seconds) or when a threshold is reached
-    (e.g. 125 logs) a background worker starts the process of submitting logs
-    from the buffer to the log server
-1. Logs are combined into chunks (e.g. 1.000 logs per request)
-    and then submitted to the log server using vlog's the JSON Stream API
+- When a log event is received, it is converted into JSON and stored in the buffer.
+- When the logs are flushed, they are transferred to the log server by a background worker.
+- The flushing of the buffer is triggered when the `flush_interval` expires (e.g. every 5 seconds) or when more then `batch_size` logs (e.g. 125) have been received.
+- Logs are combined into chunks defined by `chunk_size` (e.g. 1.000 logs per request) and then submitted to the log server using the JSON Stream API.
 
 ## Failure behavior
 
 The failure behavior of the handler in key scenarios is as follows:
 
-- In case the submission to the log server fails
-    (e.g. the server is temporarily down) the logs will be stored
-    in the buffer for later retry
-- If the buffer is full, new log events will be discarded
-    and raise a exceptions (depending on logging configuration)
-- The handler will make a final attempt to submit remaining logs during shutdown.
-If that fails those logs will written to stderr.
+- In case the submission to the log server fails (e.g. the server is temporarily down) the logs will be stored in the buffer for later retry.
+- If the buffer is full, new log events will be discarded and raise a exceptions (depending on logging configuration).
+- The handler will make a final attempt to submit remaining logs during shutdown. If that fails those logs will written to stderr.
 
-## LogRecord fields
+## Configuration
+
+This section describes how the handler and it's logger can be configured.
+
+### Handler configuration
+
+> [!TIP]
+> The handler comes pre-configured with sensible defaults and often does not need to be configured.
+
+The handler can be configured through keyword argument of the handler's constructor.
+All arguments are optional and will be set to sensible defaults when not provided.
+
+For example the size of the log buffer can be configured by setting `buffer_size`:
+
+```python
+vlogs_handler = VictoriaLogsHandler(buffer_size=100_000)
+```
+
+For a full description of all arguments please see `VictoriaLogsHandler`.
+
+### Handler logging
+
+The vlogs handler has it's own logger and which can be found under the name `vlogs_handler`.
+It will log issues related to transferring logs to the vlogs server.
+
+For example this will attach a console handler to the logger with warning level:
+
+```python
+console_handler = logging.StreamHandler()
+console_handler.setLevel(logging.DEBUG)
+vlogs_logger = logging.getLogger("vlogs_handler")
+vlogs_handler.setLevel(logging.WARNING)
+vlogs_logger.addHandler(console_handler)
+```
+
+> [!NOTE]
+> A vlogs handler instance can not be attached to this logger, as this would create an infinite loop.
+
+## Limitations
+
+The handler was designed for a specific scenario:
+
+- Transfer logs from a Django server to a VictoriaLogs instance running on the same server
+- Our focus is on applications logs over server logs (i.e. logs from Django apps over logs from the Django server itself)
+- The handler should be easy to install and use
+
+Our design choices result in a few limitations:
+
+- The buffer is kept in memory, which means that there is a risk that some logs are lost when the Django server itself has a failure. Because of that limitation we recommend not to rely on the vlogs handler alone, but to always use a log file as backup.
+- The handler does not implement HTTP authentication or backoff and retry logic for HTTP requests. It is therefore currently not suitable to be used with a remote VictoriaLogs instance.
+
+## Data Model
+
+This section describes the structure of the data that is transferred to the vlogs server.
+
+### LogRecord fields
 
 The following fields will be transferred for each log event. They are derived from Python's [LogRecord](https://docs.python.org/3/library/logging.html#logrecord-objects):
 
@@ -44,7 +92,7 @@ Name | Description | Example | Optional
 
 In addition any custom `extras' fields will be added as they are encountered.
 
-## VictoriaLogs special fields
+### VictoriaLogs special fields
 
 VictoriaLogs handles three fields in a special way:
 

tests/test_handler.py

@@ -32,7 +32,7 @@ def test_should_init_with_defaults(self):
         self.assertEqual(handler._chunk_size, 1000)
         self.assertEqual(handler._flush_interval, 5.0)
         self.assertEqual(handler._request_timeout, 3.0)
-        self.assertEqual(handler._shutdown_timeout, 2.0)
+        self.assertEqual(handler._shutdown_timeout, 4.0)
         self.assertEqual(
             handler._vlogs_url,
             "http://localhost:9428/insert/jsonline"