Log Shipper: Descriptor Leak, Symlink Cycle, Bad Relativize, and a Brittle Serialized Config

Files.walk does not eagerly build a list: it returns a lazily populated stream backed by one open DirectoryStream per directory level, and the documentation says that if timely disposal of file system resources matters, the stream must be closed with try-with-resources. A terminal operation such as forEach consumes the stream but does not close it, so the directory handles stay open until GC finalizes them - which is far too late at 4,000 directories per minute. A heap dump shows nothing because the leak is in native descriptors, not in the heap; 'Too many open files' with a flat heap is the signature. The same rule applies to Files.list, Files.find and Files.lines.

try (Stream<Path> s = Files.walk(root)) {
    s.filter(p -> p.toString().endsWith(".log")).forEach(this::copy);
}

For the cycle: Files.walk does not follow symbolic links unless you pass FOLLOW_LINKS, so the default variant is safe here. With FOLLOW_LINKS the traversal enters the link back to an ancestor, detects the cycle and reports FileSystemLoopException (surfacing through the stream as an UncheckedIOException), which kills the whole run unless handled. Options: stay with the default non-following walk and resolve link targets explicitly when needed; or catch UncheckedIOException/FileSystemLoopException, log and continue the remaining entries; or canonicalize with toRealPath() and keep a visited set of real paths; and always bound the traversal with Files.find(root, maxDepth, ...) so an unexpected shape cannot run away.

relativize requires both paths to be of the same type: both absolute or both relative, otherwise it throws IllegalArgumentException. When the root was supplied as a relative path (for example logs/ingest) while the walked entries were absolute, the bases did not match, the computed relative path had to climb out of the base, and normalize() faithfully kept those .. segments. normalize() is purely syntactic - it removes redundant . and .. components and never touches the file system - so it cannot make a path 'inside' the root; it only tidies the result.

Correct pattern:

Path base = root.toAbsolutePath().normalize();
Path abs  = p.toAbsolutePath().normalize();
if (!abs.startsWith(base)) throw new SecurityException("outside root: " + abs);
Path key = base.relativize(abs);   // no leading '..'
Path back = base.resolve(key);     // inverse of relativize

startsWith compares path components, so /a/bc does not start with /a/b - do not use string prefix checks. If symbolic links matter (they do here), canonicalize with toRealPath() (resolves links; requires the file to exist) or toRealPath(LinkOption.NOFOLLOW_LINKS) (makes absolute and normalizes without resolving), then re-check startsWith against the canonical base. Finally, normalize the key for the manifest: key.toString().replace(File.separatorChar, '/'), and delete trees in reverse walk order because a directory can only be removed once empty.

InvalidClassException: with no explicit serialVersionUID, the JVM computes one from the class's name, modifiers, interfaces, fields and methods. Adding int retries changes that computed value, so every previously written stream is rejected with 'local class incompatible: stream classdesc serialVersionUID = X, local class serialVersionUID = Y'. Declaring private static final long serialVersionUID = 2L; pins the value; with a stable UID, adding a field is a compatible change and older streams deserialize it as 0 (or null/false). Removing a field and changing a field's type are handled differently: removing is compatible, narrowing a type is not.

NotSerializableException: serialization walks the whole object graph. SocketFactory is not Serializable and the field is not transient, so writing fails as soon as the field is non-null. Mark it transient (it reads back as null, so reinitialize it in a private void readObject(ObjectInputStream in) or lazily), make the type serializable, or use a writeReplace/readResolve proxy that writes only the state.

Rules worth remembering: static fields are never serialized, transient fields are skipped and defaulted, and the first non-serializable superclass must have an accessible no-argument constructor or deserialization throws InvalidClassException. For a configuration object, a properties or JSON file is usually the better design - it survives refactoring, is inspectable, and does not require the class hierarchy to stay serialization-compatible.