Genie Is Confidently Wrong and Two Analysts Have Quit On It
A Genie space was launched over a silver schema containing 41 tables. Business users ask questions like "What were APAC card revenue bookings last quarter versus plan?" Observed failures: Genie answers from silver.txn_raw instead of gold.revenue_monthly, joins dim_merchant on the wrong key, mixes USD and local-currency columns, and treats booking_date and revenue_date as interchangeable. The space currently contains all 41 tables, zero general instructions, and one example question. Nine of the 41 tables are intermediate and two are deprecated. Two analysts have publicly said they will not use Genie again, and an unrelated team is about to build a second space over the same schema.
1. Redesign the space. Decide what belongs in table selection, general instructions, trusted assets, and join specifications, and explain what you deliberately leave out.
Read the worked answer
Prune first - this fixes most of the reported symptoms. Genie selects among the assets in the space, so putting 41 tables in front of it, including raw and deprecated intermediates, guarantees wrong-table answers. Cut to the 8-12 assets a business question could legitimately need: the gold revenue mart, the plan mart, and the conformed dimensions. Remove txn_raw entirely - if a question truly needs transaction detail, it is not a business-user question for this space. The deprecated tables should be dropped or renamed out of the schema, not just hidden, because the second team will hit the same trap.
Split by subject, not by schema. The second team should not build a second space over the same schema with the same tables; agree a division - revenue and bookings in this space, the other team's subject elsewhere - and share the conformed dimensions with identical instructions in both, otherwise the same dimension will be described two different ways and produce two different answers.
General instructions - short, imperative, only for rules that apply across questions: which table is authoritative for revenue; that booking_date is the order date while revenue_date is the recognized-revenue date and reporting defaults to revenue_date; that gold currency columns are already USD-normalized and local-currency columns must not be mixed into a total; and that "plan" means the plan mart, never a column on the actuals table. Keep this tight. Long prose instructions dilute, and every rule that could live in an example query should live there instead.
Trusted assets - define each metric once. Add SQL expressions/measures for net_revenue, bookings, and card_revenue so the definition cannot be re-invented per question, and add 4-6 example SQL queries taken verbatim from real analyst answers to the most common questions, showing the correct join key and the correct date filter. Example queries are stronger than prose because Genie imitates structure, not adjectives.
Join specifications - declare each legitimate relationship explicitly (dim_merchant.merchant_sk = fct_revenue.merchant_sk, dim_date.date_key = fct_revenue.revenue_date). Most of the wrong-join failures disappear once the correct key is stated rather than inferred.
What I leave out: instructions about formatting or tone until the numbers are right; example queries for questions nobody asks; and any attempt to keep the raw tables "just in case." If building the space has taught you that the gold mart is missing a column, fix the mart.
2. Design the ongoing quality and governance process so the space stops degrading, including how you would measure whether it is working.
Read the worked answer
Build a benchmark set before changing anything else. Collect 25-40 real questions users have actually asked, write the verified correct SQL and expected numbers for each, and get an analyst to sign off on every answer. Run the full set against the current space to get a baseline - expect it to be poor, which is useful, because it converts "Genie is bad" into a number and also protects you from making it worse. Re-run after every instruction or asset change; a change that lifts three questions and quietly breaks five is the normal failure mode.
Close the loop from real usage. Triage thumbs-down ratings with comments weekly. Each failure gets classified as wrong table, wrong filter, wrong join, wrong metric definition, or genuinely ambiguous question - and each class has a different fix. Wrong metric definition goes to a measure, wrong join goes to a join spec, wrong table goes to pruning or an instruction, ambiguity goes back to the user as a data-modeling problem. Capture the fixed questions as new example queries and add them to the benchmark so they never regress.
Ownership and permissions. Limit CAN MANAGE to two named people, give CAN EDIT to a small analyst group who may propose example queries, and CAN RUN to the business groups. Route instruction and asset edits through a review - not bureaucracy, but Genie changes are invisible in testing and catastrophic in production. Record why each asset exists in the space description so the next owner does not silently re-add 41 tables.
Metrics that matter: benchmark pass rate (the primary number), thumbs-down rate excluding low-volume noise, share of questions answered without an analyst follow-up, and count of repeated failure classes. Watch for the specific regression where a metric answer drifts after an upstream table change - reconcile the benchmark numbers against the source on a schedule, because a space can be perfectly configured and still be answering from stale or redefined columns.