Description
POSIX_FADV_SEQUENTIAL
we should call this as soon as you open the log file for writing or reading.
Why: It tells the kernel to double its readahead window. Even if you are writing, you often have a "tailing" reader (like a follower node in a cluster) reading what you just wrote. This ensures the kernel optimizes for that forward movement.
POSIX_FADV_DONTNEED (The Cache Cleaner)
For write-heavy logs.
The Problem: When you write 100GB of log data, the kernel keeps all that data in the page cache (RAM). Eventually, the system runs out of memory, and the kernel’s "pdflush" daemon panics, forcing a massive, blocking sync to disk that causes a huge spike in latency (the "IO stall").
The Strategy: After you have successfully synced a chunk of data to disk (using fdatasync), and figure out the chances of file to be used again, call DONTNEED on the old range.
This tells the kernel: "I've written this and synced it to the physical disk; you can safely evict it from RAM now." This keeps your memory usage flat and prevents sudden IO spikes.
Description
POSIX_FADV_SEQUENTIALwe should call this as soon as you open the log file for writing or reading.
Why: It tells the kernel to double its readahead window. Even if you are writing, you often have a "tailing" reader (like a follower node in a cluster) reading what you just wrote. This ensures the kernel optimizes for that forward movement.
POSIX_FADV_DONTNEED(The Cache Cleaner)For write-heavy logs.
The Problem: When you write 100GB of log data, the kernel keeps all that data in the page cache (RAM). Eventually, the system runs out of memory, and the kernel’s "pdflush" daemon panics, forcing a massive, blocking sync to disk that causes a huge spike in latency (the "IO stall").
The Strategy: After you have successfully synced a chunk of data to disk (using fdatasync), and figure out the chances of file to be used again, call DONTNEED on the old range.
This tells the kernel: "I've written this and synced it to the physical disk; you can safely evict it from RAM now." This keeps your memory usage flat and prevents sudden IO spikes.