A migration that silently drops 12% of nodes

Two separate things are going on.

Why re-running does nothing. Migrate records each successfully imported row in the migration's map table, keyed by source ID. On a second run it skips anything already in the map. The 5,900 rows were never mapped because they failed, not because they were skipped by a limit. Raising --limit changes how many rows are attempted per run; it does not cause failed rows to be retried, because their failure state is tracked separately. The correct command is drush migrate:import <id> --update combined with clearing the failure status, or drush migrate:reset-status <id> to move failed rows back to unprocessed.

Why they failed silently. At default verbosity, Migrate logs row-level failures at a level that is suppressed. The likely causes given the setup:

• NULL body values hitting a process plugin that assumes a string (e.g. a callback doing strlen()), throwing a TypeError per row.
• Duplicate nid values causing the map table's unique constraint on source_ids_hash to reject the second insert, so the duplicate is never mapped.
• The paragraph reference failing when the referenced paragraph was itself skipped, leaving an unmet migration_lookup.

The diagnostic step is drush migrate:import <id> --update -vvv (or check the migrate_message_* tables) to see the actual per-row exception before changing anything.

Duplicates must be resolved before the map table sees them, because the map's uniqueness is on the source ID hash - two rows with the same nid are indistinguishable to Migrate and the second is always rejected.

Options, in order of correctness:

1. De-duplicate in the query. Change the source plugin's query to return one row per nid, e.g. with a GROUP BY nid or a window function selecting the latest revision. This is the cleanest and keeps the source ID stable.

2. Synthesize a unique source ID. If the duplicates are genuinely distinct records that happen to share a legacy nid (common after a botched prior import), derive a composite key such as nid . ':' . vid and declare it as the source ID in getIds(). The map then stores both. Note this changes the identity of the row, so any downstream migration_lookup must use the same composite key.

3. Throw on duplicates. If duplicates indicate corrupt data, have the source plugin detect them and fail loudly during prepareRow() so the migration stops rather than silently importing half the set.

What not to do: import the first copy and hand-fix later. That leaves the map table inconsistent and makes the next incremental run unpredictable.

They should be imported, with the NULL normalized, unless the field is genuinely required by the destination.

Reasoning: the migration's job is to represent the source faithfully. A NULL body in D7 is a valid state - the node existed and was published. Dropping it changes the site's content inventory, and a 12% loss on a content migration is a business problem, not a technical footnote. The default get plugin already handles NULL by skipping the destination property, so a NULL body usually just means an empty body field, which is fine if the field is optional.

If the destination field is required, the correct fix is a default_value or a callback that maps NULL to an explicit empty value - not to let the row throw. The exception is if the business rule says every article must have a body; then the failure is meaningful and should be surfaced as a validation report rather than a silent skip, so editors can triage the 5,900 rows before cutover.

The general principle: decide per-field whether NULL is data or an error, and encode that decision in the process pipeline. Never let it be decided by whichever plugin happens to throw.