PHP memory exhaustion when reading large log files via LogFileModelTraits

[novastate]

Bug: PHP memory exhaustion when reading large log files via LogFileModelTraits

Description

The REST API crashes with a PHP fatal error when attempting to read large log files. The read_uncompressed_log() method in LogFileModelTraits.inc uses PHP's file() function which loads the entire file into memory, causing memory exhaustion on large rotated log files.

Environment

• pfSense Version: 2.8.1-RELEASE (amd64)
• REST API Version: pfSense-pkg-RESTAPI-2.6_8
• PHP Memory Limit: 512 MB

Error

PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted
(tried to allocate 541402264 bytes) in
/usr/local/pkg/RESTAPI/ModelTraits/LogFileModelTraits.inc on line 38

Root Cause

In LogFileModelTraits.inc, the read_uncompressed_log() method:

private function read_uncompressed_log(string $filepath): array {
    $this->check_file_exists($filepath);
    return file($filepath, FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);  // Line 38
}

This loads the entire file into memory as an array. When log rotation produces large files (500MB+ in my case due to misconfigured log limits), API calls that query logs will crash.

The same issue exists in read_bzip2_log() and read_gzip_log() which decompress entire files into memory.

Steps to Reproduce

1. Have a large rotated log file (e.g., /var/log/system.log.X > 500MB)
2. Make an API call that reads system logs
3. PHP crashes with memory exhaustion

Suggested Fix

Consider one of:

1. Add file size check - Reject files over a threshold with a clear error
2. Stream reading - Use fgets() in a loop with line limit
3. Generator pattern - Yield lines instead of returning full array
4. Pagination - Only read requested line ranges from file

Example size-limited approach:

private function read_uncompressed_log(string $filepath, int $max_bytes = 50000000): array {
    $this->check_file_exists($filepath);

    if (filesize($filepath) > $max_bytes) {
        throw new NotAcceptableError(
            message: "Log file too large to read. Size: " . filesize($filepath) . " bytes.",
            response_id: 'LOG_FILE_TRAITS_FILE_TOO_LARGE',
        );
    }

    return file($filepath, FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);
}

Workaround

Reduce log file size limits in pfSense:

• Status → System Logs → Settings → Set log file size to 500KB or less
• Or add to config.xml: <logfilesize>512000</logfilesize>

[jaredhendrickson13]

This should be relatively simple to address. Currently, Models that are not based on config values are not paginated until all objects have been loaded into memory. This was intentional as most of these models simply call internal pfSense callables that always return the full set of data, and cannot be paginated beforehand anyway. For logs however, this does demonstrate a legitimate use case for passing pagination parameters all the way down to these callables. Should be able to have that included in v2.7.0.

In the meantime, my suggestions would be:

1. Reduce the log file size (like you mentioned). pfSense in general does not handle large log files well. Better to ship those logs somewhere better equipped to handle log files of that size.
2. If you cannot reduce the log file size, increase the amount of memory allocated to PHP. In the pfSense webConfigurator, go to System > Advanced > Miscellaneous > PHP Settings and increase the memory limit if possible.

[JeremiahChurch]

I've been hitting this OOM consistently on two of my firewalls (~30MB filter.log + 30 rotated bz2 files) via an MCP integration that calls ?limit=<whatever IE 50>. I put together a fix that passes the pagination limit down to the log-reading callables so they can stop early, as suggested by the OP

What it does:

• read_log() gains an optional $limit parameter. When set, it reads newest files first using memory-efficient readers and stops once enough lines are collected
• Plain text files: reverse-reads from EOF via fseek - O(limit) memory
• Compressed files: streams via PHP stream wrappers (compress.zlib://, compress.bzip2://) with a ring buffer - O(limit) memory
• Model::get_internal_objects() forwards the limit hint to callables that accept it (checked via cached reflection, backward-compatible with callables that don't)
• Model::read_all() passes limit + offset down. Skips the optimization when $reverse=true since that needs the full dataset
• Unbounded reads ($limit=0) follow the exact same code path as before

Tested on a live pfSense CE 2.7.0 box (FreeBSD 15.0, REST API v2.5):

Metric	Before	After
?limit=50 memory	512MB+ (OOM crash)	22MB
?limit=10 response time	N/A (crash)	1ms
Rotated files touched	All 31	Zero (current log had enough)

I'm absolutely BEYOND rusty in PHP and I have zero experience with the API code (beyond using the snot out of it) so I have no idea if this is a good pattern or hallucinated trash. It generally seems reasonably limited in scope, and most importantly, it worked on real hardware for me.

Take a glance at https://github.com/JeremiahChurch/pfSense-pkg-RESTAPI/compare/fix/log-pagination-memory-exhaustion

If it looks like a workable path I'm happy to open a PR against this repo. If it's hot garbage, lemme know and I'll toss it in the trash.

[jaredhendrickson13]

@JeremiahChurch Seems like those changes should handle this okay. Definitely open to a PR into the next_minor branch so it can be tested more thoroughly.

Thanks!