Data preparation and integration for generative AI
Domain 8 - Data preparation and integration for generative AI
Data questions test whether you can turn messy enterprise content into governed, retrieval-ready and training-ready assets.
Where data lives
watsonx.data is an open lakehouse: object storage (IBM COS or S3) with open table formats such as Iceberg, Hudi and Delta, queried through Presto and Spark, with governed connections to sources like Db2 and Snowflake. Data can be queried in place rather than copied.
Preparing content for RAG
- Parse PDFs, HTML and scans (OCR where needed).
- Deduplicate near-identical documents; duplicates crowd out diverse results.
- Redact PII before anything is embedded.
- Chunk with overlap and attach metadata: source, page, section, version, access control.
- Embed with the chosen embedding model and load into the vector store.
- Track lineage so a retrieved chunk can be traced to its source document.
Preparing data for tuning
Convert to JSONL instruction/input/output records, normalise formatting, filter low-quality and unsafe samples, and split train/validation/test before any augmentation.
Worked example
Ten thousand scanned HR PDFs must power a policy assistant. Pipeline: OCR → dedupe by hash → redact employee identifiers → chunk at about 500 tokens with 10% overlap → embed → index with department and effective-date metadata. Reasoning: effective-date metadata lets retrieval filter out superseded policies, the most common source of wrong HR answers.
Traps
- Skipping deduplication, so top-k returns five copies of one paragraph.
- No metadata, so answers cannot be cited or filtered by permission.
- Embedding before redaction leaves PII in the vector store.
- Reusing a stale index after documents change.
- Mixing document versions in one index without version metadata.