Experience · Lessons Learned
The false finding that wasn't
This report is anonymised. It describes a real incident without host names, addresses or customer data. The focus is not the vulnerability but the chain of evidence: what do you check when you must decide within hours whether a system has been compromised — and how easily can the result be misread?
Starting point
Within one day the PHP project published two critical, unauthenticated vulnerabilities (CVE-2026-109618 and CVE-2026-109619, CVSS 9.2 each, jointly called “PHP2Shell”). Affected are PHP installations serving web requests. The host in question ran a reverse proxy, a Nextcloud instance on PHP-FPM and a number of static sites. The operator's question was simple:
Are we affected? Is the patch active? And do the logs show anyone exploiting the flaw?
Three questions, three very different evidence problems. That was the difficulty.
Step 1 — prove exposure instead of assuming it
The first reflex (“we run PHP, so we are affected”) is too coarse. What matters is the SAPI: only installations that process web requests are exposed. The proof was correspondingly concrete:
- Inventory of the PHP landscape: only
fpmandcliinstalled — nomod_php, nophp-cgi, no Apache. - Exposed endpoints: exactly one PHP application behind the proxy (the Nextcloud) plus a LAN-only admin port; alongside it an orphaned Adminer site whose PHP socket had not existed for weeks.
- No “PHP somewhere else”: static sites without a FastCGI block are not a PHP attack surface.
Step 2 — patch verification: package and runtime
An installed security update is not yet an active patch. Between “the package is on disk” and “the running process uses it” sits a restart of the FPM master. So it was checked twice:
| Check | Result | What it proves |
|---|---|---|
| Package version | 8.5.4-0ubuntu1.5 (Ubuntu fix, USN-8948-1/-2) | The fix is installed |
| Installation time | 2026-10-22 06:12 (automatic updates) | When the fix arrived |
| FPM restart | 2026-10-22 06:13, master active ever since | The fix runs |
| File integrity | package verification without deviation | No tampered binaries |
These four lines are the actual patch proof. Everything else is interpretation of logs.
Step 3 — four log layers, four different questions
A single log cannot prove an incident. Each layer answers a different question — and has its own blind spot:
| Layer | Answers | Blind spot |
|---|---|---|
| Proxy access log | Which requests arrived, with which status and body size? | No content; rotation limits lookback |
| Application log (Nextcloud) | What did the application do internally — errors, warnings, logins? | At info level only events, no access records |
| FPM log | Worker crashes, memory exhaustion, fatal errors | PHP level only, no HTTP context |
| Journald / kernel | OOM kills, segfaults, core dumps | No application semantics |
In addition: Fail2ban (SSH brute force) and the package manager (patch timing). Only the combination carries a statement — and only if the blind spots are named.
The false finding — “the application writes no log”
The search for the application log covered the usual directories (/srv, /var, /home, limited depth) and found nothing. The conclusion was obvious: there is no log. It was wrong.
Two things had been overlooked:
- The data directory lay outside the search space. The instance did not store its data in the program directory but on a separate volume. The search was thorough — just not there.
- The logger had a fallback. If the configured log file did not exist and its directory was not writable, the application wrote to a substitute file in the data directory. That is correct behaviour — but it hides a misconfiguration.
The actual defect sat one level deeper: the rotation job reads the log path from the configuration. Because that path did not exist, it compared the size of a non-existent file — and never rotated. Result: a single file of 438 MB instead of hourly rotation.
logfile = program-dir/data"] --> L{"File exists?"} L -->|no| F["Logger uses fallback:
data-dir/nextcloud.log"] L -->|yes| W["Logger writes configured file"] C --> R["Rotation job reads the same
configured path"] R --> N{"File there?"} N -->|no| X["No rotation
(file grows unbounded)"] F --> X
The lesson is uncomfortable because it concerns our own work: “not found” is not a finding as long as the search space is undocumented. A negative finding needs the same care as a positive one — including a statement of where one did not look.
The second false finding — the firewall
An admin port was “reachable only inside the LAN”. The check found no port-specific allow and noted: no allow present, so closed from outside. That was too short as well.
In fact a blanket rule for the entire LAN subnet existed, permitting every port. So the port was open inside the LAN — protection came from network topology, not from the port rule. The result (“not reachable from the internet”) was right, the reasoning was not. And a wrong reasoning is the beginning of the next wrong decision.
Rule: read a firewall as a whole — default policy, blanket allows, order. A glance at “is a port allow missing?” is not enough.
What the logs actually showed
After correcting the log path, the evaluation became possible — and the picture was inconspicuous:
- 18,244 HTTP requests over ten days; the PHP application answered not a single request with 5xx.
- Largest request body: 25 KB. No body ≥ 1 MB — the path typical for memory exhaustion was absent from the observed traffic.
- Application log: 14,518 entries, 0 fatal errors, a single error (app store unreachable), no failed logins, no app installations.
- Indicator search across 789,528 lines (typical webshell functions,
/tmp/paths): zero hits each. Hits on known shell names turned out to be random sequences inside request IDs. - FPM and kernel: no fatal errors, no OOM kills, no segfaults.
- Web root: no new PHP file since the start of the observation window, no unexpected change.
The only “hot” hit was a scanner sending four roughly 1 MB requests to the orphaned Adminer site — exactly the pattern an exploitation attempt produces. All four ended with 502 because the PHP socket did not exist. The attack ran against a dead backend. That too is a result: the most dangerous finding proved the effectiveness of an (unintended) hardening.
The limit of proof — and why evidence must be secured
Two retention rules limited the statement:
- Proxy logs: 5 × 10 MB rotation — about ten days of lookback. The observation window was fully covered, nothing more.
- Application log: rotation keeps exactly one previous file. On the next run the evaluation window would have been overwritten.
Therefore the evaluation window was secured before the next rotation as a separate archive: compressed, owner root, mode 640 (the file contains user names and IP addresses), with a documented checksum. Evidence that cannot be reproduced is of little value in an audit — a checksum turns a file into a verifiable statement.
(one rotated file)"] --> B["Window extract
2026-10-15 … 2026-10-25"] B --> C["gzip archive
root:root 640"] C --> D["SHA-256 in the report"] D --> E["Reproduction:
gunzip + wc -l"]
What worked well
- Patch proof kept twice (package version and runtime) — the decisive statement did not depend on logs.
- Negative proofs instead of assumptions: no new PHP file, no 5xx, no OOM kills, no fatal errors.
- Attack against a dead backend recognised — the 502 finding prevented a dramatised misjudgement.
- Decision reasoned, not asserted: a reporting obligation was examined and denied with reasons, not silently ignored.
What we will do differently next time
- Document the search space before searching. Write “not found” only when the paths and depths searched are named.
- Attach a counter-check to every config statement. Log path: not only read it, but verify the file exists, grows and rotates.
- Test rotation actively (exceed the size artificially, prove rotation in the log) — do not assume it runs.
- Read the firewall as a whole (default policy, blanket subnet rules, order).
- Dismantle orphaned sites immediately. A dead PHP site is not a risk, but a permanent noise source in the log and a blind spot in the assessment.
- Secure evidence before rotation — with owner, mode and checksum.
- Correct false findings, do not smooth them over. The report contains the correction including date and cause. A silently repaired false finding destroys trust more than the mistake itself.
Transferability
The incident itself was unspectacular: patched, not exploited, no notification required. What was valuable is the evidence apparatus: exposure via inventory, patch via package and runtime, exploitation via several log layers, assessment with explicit limits of proof, evidence with checksum — and a report with a change history that also contains our own misjudgement. This structure can be versioned in Git and still be verified years later.
Sources
- PHP project: security advisories for
CVE-2026-109618andCVE-2026-109619. - CERT-EU, Security Advisory 2026-018 (mitigations and exploitation status).
- Ubuntu Security Notices USN-8948-1/-2 (package fix for PHP 8.5).
- Manufacturer statement on the perimeter device: no PHP in use there.