FastRP and KNN Product Similarity under Memory Pressure
An e-commerce team has 900k :Product nodes, 500k :Customer nodes, 14 million :VIEWED relationships, and 3.2 million :BOUGHT relationships. They want 20 similar products for 200k active products, refreshed monthly, on a 24 GB heap. They project all Product and Customer nodes with all relationships as UNDIRECTED, run FastRP with embeddingDimension 256, iterationWeights [0.0, 1.0, 1.0], and mutate an embedding. Then they run gds.knn.stream with topK 20 on all 1.4 million nodes. FastRP peaks at 8 GB; KNN estimate is 30 GB and fails. In a pilot on 100k products, most neighbours are bestsellers, niche products have no neighbours, and some products appear similar to themselves in the raw output before filtering.
1. How would you redesign the projection and execution so FastRP and KNN both fit in 24 GB?
Read the worked answer
Separate the embedding graph from the similarity graph. FastRP benefits from Customer context, so run it on the bipartite Product-Customer graph with VIEWED and BOUGHT relationships, weighted if the business trusts interaction strength. Mutate the embedding onto Product nodes. Do not run KNN on all 1.4 million nodes: KNN only needs the node properties, not Customers or their relationships.
After FastRP, project a Product-only graph that carries the embedding as a node property. If the GDS version requires a relationship projection, include a minimal existing product-product relationship rather than the full bipartite graph. Run gds.knn on that smaller graph with topK 20, estimate first, and use mutate mode if scores will be queried repeatedly. This typically moves KNN memory from a 1.4 million-node problem to a 900k-node problem and removes 500k Customer nodes plus 17.2 million relationships from KNN's memory footprint.
2. Why are bestsellers over-represented and niche products left without neighbours, and what would you change?
Read the worked answer
FastRP embeddings and cosine KNN tend to reflect degree and interaction volume, so products with many views and purchases sit near many other products. Niche products have sparse neighbourhoods and may fall below any reasonable similarity cutoff. This is a coverage problem as much as a ranking problem.
Measure the fraction of active products with at least 20 neighbours above the cutoff, broken down by sales decile. Use normalizationStrength in FastRP if available to reduce degree effects, and evaluate Adamic-Adar or same-category features rather than relying only on embedding cosine. Apply a similarityCutoff so weak neighbours are not presented as similar, and provide a documented fallback for uncovered products, such as category bestsellers or editorial collections. If a popularity penalty is used, tune it on held-out co-purchase data and check that niche coverage improves without destroying relevance.
3. How should the team evaluate the monthly similarity job and prevent temporal leakage?
Read the worked answer
FastRP uses graph structure, so including future VIEWED or BOUGHT edges in the projection leaks information into the embeddings. Build the training graph from interactions before a cutoff, produce embeddings and neighbour lists, and evaluate against BOUGHT edges after the cutoff. Report recall@20, coverage by product decile, and popularity concentration, not only a global similarity average.
Use a fixed randomSeed for FastRP so monthly runs are reproducible, and version the embedding property and neighbour output. Remove self-similarity and duplicate product pairs before ranking. Because the business needs only 200k active products, filter the final output to those products and confirm that the fallback path is evaluated separately. A monthly job that fits memory but recommends only bestsellers is not a successful similarity system.