Memory-Constrained Community Detection for a Person Graph
A people-analytics team has 12 million :Person nodes and 48 million :KNOWS relationships with an interaction weight from 1 to 100. There are also 6 million :WORKS_AT relationships to 900k :Company nodes and 2.4 million :LIVES_IN relationships to 40k :City nodes. The GDS host has a 16 GB heap and 4 CPUs. To find communities, they project every node label and every relationship type with UNDIRECTED orientation and properties ['weight'], then run gds.louvain.mutate. The projection estimate returns 22 GB, and the job fails with a memory-estimation error. Raising concurrency to 8 makes it worse. They need Person communities monthly and want to keep Company and City for later analysis, but not for this run. A smaller test run gave one giant community covering 38% of people and 24k tiny communities.
1. Why did the projection exceed memory, and what projection would you use for Louvain?
Read the worked answer
The projection is far larger than the algorithm needs. It includes 900k Company and 40k City nodes, 8.4 million WORKS_AT and LIVES_IN relationships, and an UNDIRECTED copy of every relationship, plus the weight property. Even if Company and City are small relative to Person, the extra relationships and property arrays consume memory. UNDIRECTED roughly doubles relationship storage, which is significant at 48 million KNOWS relationships.
For Person communities, project only :Person and :KNOWS with orientation UNDIRECTED and relationshipWeightProperty 'weight'. Do not include WORKS_AT, LIVES_IN, Company, or City in this graph; keep them in separate graphs for later analysis. Run gds.graph.project.estimate and gds.louvain.mutate.estimate before executing, use mutate mode with a mutateProperty, and set concurrency to 2 or 4 rather than 8. If the estimate is still too high, use gds.graph.project.cypher to aggregate parallel KNOWS relationships and filter low-weight interactions, then re-estimate. Drop old graphs with gds.graph.drop after checking gds.graph.list.
2. The Louvain result has one giant community and many tiny ones, and community counts change between runs. What configuration or algorithm changes would you make?
Read the worked answer
First verify that the weight property is actually projected and used; an ignored weight can collapse structure. Then treat the giant/tiny split as a resolution problem. Leiden is generally preferable to Louvain for well-connected communities and exposes gamma, where higher gamma produces more, smaller communities and lower gamma produces fewer, larger ones. If Leiden is available in the installed GDS version, test a small gamma range and compare modularity, community-size distribution, and the share of the largest community. Use includeIntermediateCommunities to inspect hierarchy rather than accepting the first level.
Louvain and Leiden community IDs are not stable identifiers across runs. Persist the IDs with a run timestamp and compare memberships with a metric such as adjusted Rand index instead of comparing raw IDs. Do not tune only for modularity; a single giant community can have acceptable modularity while being useless operationally. Evaluate against the business definition of a community.
3. How should the team handle monthly refreshes without recomputing on an ever-growing graph?
Read the worked answer
There is no safe incremental Louvain/Leiden mode that can simply patch last month's communities. Recompute on a deliberately scoped graph. For monthly communities, project Persons and KNOWS relationships active within a rolling window, such as the last 90 or 180 days, using gds.graph.project.cypher with a cutoff parameter. Aggregate multiple interactions per pair into one weighted relationship, and filter weights below a threshold if low-weight edges add noise. This keeps node and relationship counts bounded and makes the estimate predictable.
Write community IDs to a versioned property such as communityId_2025_06, or write to a separate node label/property, so downstream users are not surprised by changing IDs. Compare successive months using adjusted Rand index and community-size drift. If a full historical graph is required for regulatory reasons, run that less frequently in a larger maintenance window and keep the monthly operational graph separate.