Incremental ingestion
Log files only ever grow: your framework appends new lines to the end, but the millions of bytes before that never change. Incremental ingestion means Log Lens reads only the new tail of each file on every import, instead of re-reading and re-parsing the whole thing. Imports stay fast even when a log has been collecting for months.
How it tracks progress
For every file it has seen, Log Lens remembers a small set of facts, including a last_offset - the byte position where the previous import stopped. The next import opens the file, seeks to last_offset, and parses from there. When it finishes, last_offset is advanced to the file's new size.
Skipping unchanged files
Before reading anything, Log Lens compares the file against what it recorded last time. If all of these still match, the file is skipped entirely - zero work:
| Checked | Meaning |
|---|---|
| Size | File hasn't grown or shrunk |
| Modified time | Filesystem mtime is unchanged |
| Parser version | The parsing logic is the same build |
| Log type | Still detected as the same format |
| Module / source | Still assigned to the same place |
What triggers a full reparse
Sometimes reading only the tail would be wrong, so Log Lens resets last_offset to 0, discards the file's existing indexed events, and reparses from the beginning. This happens when:
- The file shrank (
size < last_offset) - a sign it was rotated or truncated. See Log rotation and generations. - The parser version changed - a Log Lens update improved how a format is read.
- The detected log type changed.
- The module or connector source assignment changed.
The parser version signature
The stored "parser version" is more than a build number. It combines the built-in parser version with a signature of which severities you have chosen to index. Changing your severity selection therefore changes the signature, which invalidates every file and forces a clean reparse on the next run - so newly included levels are picked up and excluded ones are dropped. See Choose which severities to index.
Forcing a rebuild
If you want to reprocess everything from scratch - for example after changing settings - a full reindex resets last_offset and the parser version for all known files and reparses them. See Reindex after changing settings.
Incremental ingestion pairs naturally with byte-range storage, which lets Log Lens point back to the exact bytes of each indexed event.