Index Saturation on a Small-File LTFS Archive
A university research archive holds 1.1 PB on 900 LTO-8 LTFS tapes. Average file size is 2 MB, and each tape carries roughly 600,000 files because instruments write many small outputs. LTFS runs on Linux with default formatting. Mount times have risen from 3 minutes to 22 minutes, several tapes fail to mount after repeated append sessions, and restores average 90 MB/s although the drives are rated at 360 MB/s. Tape alerts show shoe-shining. The archive adds 50 tapes per month and must keep an 8-hour maintenance window. Scientists like LTFS because they can drag and drop files, so management wants to keep it. The team has been told to fix performance without rehydrating the entire archive to disk. A proposal to reformat every tape with a larger index partition is on the table.
1. Diagnose the root cause. Why is a larger index partition alone unlikely to fix the problem?
Read the worked answer
The root cause is file count, not just partition size. At 2 MB average and 600,000 files per tape, the LTFS index is hundreds of megabytes of XML. Every mount parses it; every append rewrites or extends it; fragmentation and small I/O cause shoe-shining. A larger index partition helps only if the index actually exceeds the partition. It does not reduce the cost of parsing a huge index or restoring millions of small files. The proposal treats a symptom. Check index size, append count, block size, and tape alerts before reformatting. Reformatting every tape would also create a massive migration project and risk data loss.
2. Recommend format and operational changes that preserve drag-and-drop access.
Read the worked answer
Preserve drag-and-drop at the project level, not per instrument file. Containerize small files into tar or zip archives before writing to LTFS, so one project is one or a few large objects. For new tapes, use a larger index partition and a larger LTFS block size where supported. Reduce append sessions by staging a project and writing once. Keep an LTFS index cache or catalog. Run ltfsck on suspect tapes. For existing tapes, do not reformat; read and repack onto new tapes. This reduces index entries and improves restore streaming while keeping a familiar folder view for scientists.
3. How would you decide whether LTFS remains the right archive layer?
Read the worked answer
Decide by access pattern and scale. LTFS is strong for large files, simple handoff, and drag-and-drop. It is weak as a namespace for hundreds of millions of small files. If scientists need file-level browse, put a catalog or object store in front and use LTFS for transport and archive containers. If most access is project-level, containerization is enough. Measure mount time, restore rate, and file count per tape; set a threshold. Keep LTFS but change the data layout, rather than accepting 22-minute mounts as normal.