A length-prefixed sensor protocol meeting a single recv()

TCP is a byte stream, not a message stream: recv(n) returns between 1 and n bytes, at arbitrary boundaries. recv(65536) may return half a frame, or two and a half frames. Two of the reported symptoms have the same cause - a partial buffer handed to json.loads. The "300 KB never gets through" case is not a size limit; it is that 300 KB almost always arrives in multiple segments, so the very first recv is a fragment that never parses. Bench frames are 200 bytes and usually arrive whole, which is why the bug lay dormant.

Loop:

def recv_exact(sock, n):
    buf = bytearray()
    while len(buf) < n:
        chunk = sock.recv(min(n - len(buf), 65536))
        if not chunk:
            raise ConnectionError("peer closed mid-frame")
        buf += chunk
    return bytes(buf)

hdr = recv_exact(conn, 4)
(n,) = struct.unpack("!I", hdr)
payload = recv_exact(conn, n)
msg = json.loads(payload)

Avoid socket.MSG_WAITALL as the primary mechanism: it is unimplemented or non-functional on some platforms and ignored on non-blocking sockets. The explicit loop is portable and testable.

Test it properly: use socket.socketpair(), write the frame one byte at a time with sendall and a short pause, and assert the parser reassembles it. That test is deterministic and catches every framing regression - a normal loopback test with sendall(whole_frame) never will.

Three independent defects, one theme: no lifecycle ownership.

1. Leak. The except: log; continue path leaves conn open. Every JSON error leaks a socket, and at 1–3% of frames across 500 sensors you reach the default 1024 FD soft limit in about 36 hours. Wrap the handler in with conn: (sockets are context managers and close on exit, including exceptions) or try/finally: conn.close(). Same for the listening socket. Add ResourceWarning-as-error in tests (-W error::ResourceWarning) so an unclosed socket fails CI instead of production.

2. 60-second bind failure. The previous process's connections are in TIME_WAIT; the 4-tuple cannot be reused, so bind() fails with EADDRINUSE. Set SO_REUSEADDR before bind() - sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1), or allow_reuse_address = True on a socketserver.TCPServer subclass. It permits binding over TIME_WAIT sockets; it does not permit two live listeners. Do not reach for SO_REUSEPORT - that does allow two live listeners and will silently split your traffic.

3. Blocked thread. Blocking recv never returns if the peer vanishes without FIN. Set conn.settimeout(30) and treat socket.timeout as a normal disconnect; also enable SO_KEEPALIVE with TCP_KEEPIDLE/TCP_KEEPINTVL so half-open connections are reaped at the kernel level. Pair it with a bounded worker pool (ThreadingTCPServer with daemon_threads and a semaphore) so 500 slow sensors cannot exhaust threads.

N comes from a device that may be spoofed, so n = struct.unpack("!I", hdr)[0] can be 0xFFFFFFFF (4 GB). Consequences, in order of severity:

• Memory exhaustion. A naive recv_exact(conn, n) that pre-allocates, or conn.recv(n), lets one host demand gigabytes. Ten hostile devices take the collector down.
• Slowloris. A client that declares 1 MB and then dribbles one byte per minute holds a worker thread indefinitely. The settimeout(30) above helps per-read but not for a trickle that resets the clock.
• Desync. N = 0 yields an empty payload, json.loads(b"") raises, and if the handler continues on the same connection every subsequent frame is misaligned.

Validation rules to implement:

1. MAX_FRAME = 1 << 20 (1 MiB). Reject n == 0 or n > MAX_FRAME, log with the peer address, and close the connection - a framing violation means the stream cannot be trusted, so do not continue.
2. Read in bounded chunks (min(remaining, 65536)) rather than allocating n up front.
3. Enforce an absolute per-frame deadline (total elapsed, not per-read), so trickling cannot outlast the timeout.
4. Only after the payload is complete, parse with json.loads(payload.decode("utf-8")); catch UnicodeDecodeError and JSONDecodeError separately, count them as metrics, and close.
5. Validate the decoded object against a schema (device_id, ts, readings) before it reaches any database.

Better still: require TLS with per-device client certificates, so a spoofed source cannot open a connection at all - validation limits the blast radius, authentication removes most of it.