Halcyon Media: 400 TB of SMB File Data, a Nine-Month Deadline, and an 06:00 Analytics SLA

Arithmetic first 180 Mbps usable is about 22.5 MB/s, or roughly 1.9 TB/day at perfect efficiency. Realistically 60% efficiency gives 1.1-1.2 TB/day. 400 TB / 1.15 TB per day is roughly 350 days - well past the 270-day deadline, and it would saturate the circuit users also depend on. Network-only transfer is out.

Bulk path AWS Snowball Edge Storage Optimized holds about 80 TB usable, so five devices cover 400 TB with room for a re-run of a failed job. Data lands in S3 within days. Note the constraint: Snowball writes objects to S3; it cannot write to FSx for Windows File Server directly. So the path is Snowball Edge to S3, then AWS DataSync from S3 into FSx for Windows File Server.

Target service FSx for Windows File Server is the only fully managed service that gives real SMB with AD-based authentication and NTFS ACLs, and it is not a gateway sitting in a data center that is being closed. Place it in the region on the Direct Connect path so SMB round trips stay under the 15 ms budget.

Storage configuration judgement Sizing 400 TB on SSD storage is the cost mistake here. Only 6% (~24 TB) is touched monthly. Use the HDD storage type with an SSD read cache sized to the working set, Multi-AZ for automatic failover, and enable Data Deduplication for general file-share content. Verify HDD storage type availability in your chosen region and test the cache hit rate against real access traces before committing.

The ACL problem nobody plans for DataSync S3-to-FSx does not reconstruct original NTFS ACLs, because objects in S3 never carried them. Two mitigations, use both:

• Run DataSync directly from the on-premises Windows servers to FSx for the 6% hot/active data set (roughly 24 TB, about three weeks at realistic throughput, scheduled off-peak). This preserves permissions natively.
• For the Snowball-loaded cold bulk, either export an icacls manifest before shutdown and reapply it, or apply a documented inherited-ACL model per department share and accept it. Get the records-management owner to sign off on the second option in writing before cutover.

Rollback Keep the on-premises shares read-only for two weeks after cutover, and cut over via DFS Namespace referral so rolling back is a referral change, not a data move.

Do not query the file server Running nightly SQL through SMB against FSx would contend with user I/O, and Athena cannot read FSx for Windows File Server at all. The analytics path must be object storage.

The free copy already exists The Snowball-landed S3 bucket holding the 400 TB is already the analytics source of truth. Do not treat it as a staging area to be deleted. Keep it, organize it, and query it in place. This is the single biggest cost decision in the scenario: a second 40 TB copy into a separate analytics bucket is pure waste.

Incremental sync About 8,000 files change per day. A scheduled DataSync task from FSx for Windows File Server to a partitioned S3 prefix moves only changed files and completes in minutes, not hours. Partition the prefix by department and by ingest date so partition pruning is possible.

Query layer

• AWS Glue Data Catalog over the partitioned prefixes, refreshed by a crawler or, better, explicit partition registration so crawlers do not rescan 40 TB.
• Convert hot partitions to Parquet with a nightly Glue job. Athena bills $5 per TB scanned; querying raw text over 40 TB is roughly $200 per full scan, while columnar plus partitioning on a typical query drops the scanned volume by 90% or more. This single change usually decides whether the project is affordable.
• If users need familiar SQL semantics over the curated subset, put Amazon Redshift Serverless on the Parquet output; if they are analysts comfortable with Athena, skip it.

Meeting 06:00 Sequence matters: DataSync incremental (minutes, can start at 01:00), Glue transform on changed partitions only (tens of minutes), then a pre-warmed Athena workgroup. Do not re-transform all history nightly - that is what pushes past 06:00. Set a query result reuse and a per-workgroup data scan limit so one runaway query cannot blow the budget.

What must not be cut

• The Snowball bulk job. It is the only mechanism that beats the nine-month clock. Compressing the schedule later is impossible because the physics of the link do not change.
• The ACL remediation and its sign-off. Cutting this produces a go-live where the wrong people can read HR and legal shares - a security incident, not a schedule slip.
• The pilot department and read-only rollback window.

What flexes

• The SSD read cache size. Start smaller and grow it with observed hit rate; this is a resize operation on an existing file system, not a re-architecture.
• Multi-AZ to Single-AZ for the lowest-criticality department shares, accepting a documented recovery-time hit for genuinely non-critical content.
• The Parquet conversion. Ship Athena over text for week one at a known scan cost, then optimize. Analytics can be slower before it can be wrong.

The real deadline driver Decommission is gated by the last byte leaving on-premises, not by optimization. Sequence so that data egress finishes first and tuning happens after. Also book the Snowball jobs early: device availability and shipping, not network throughput, become the critical path once the window compresses. Keep a two-week buffer for a re-run of any failed device, because a re-ship costs days you will not have.