# Eval queries for israeli-scam-detector
# Format: TYPE | query | expected | the exact description phrase that decides it
#
# Rule applied (Phase 2.7 item 5): for every NEGATIVE, name the phrase in the
# description that actually EXCLUDES it. A query is not excluded just because the
# description happens to lack its vocabulary.
#
# Train/test split: TRAIN = 1-4, 7-8, 10-11 (8 of 12 queries)
#                   TEST  = 5-6, 9, 12 (4 of 12 queries)

# --- POSITIVE (should trigger) ---
POS | קיבלתי SMS מדואר ישראל שממתינה לי חבילה וצריך לעדכן כתובת, זה אמיתי? | MATCH | "Israel Post" + "Use when a user asks whether a message or link is real"
POS | מסרתי קוד מה-SMS למישהו שהתקשר ואמר שהוא מהבנק, מה עושים עכשיו? | MATCH | "got a suspicious call" + "including after the money is already gone"
POS | כמה זמן יש לבנק להחזיר לי את הכסף אחרי הונאה בכרטיס? | MATCH | "the 8-business-day refund clock"
POS | אמא שלי דוברת רוסית וקיבלה שיחה מוזרה כאילו מהבנק, איך אני מגן עליה? | MATCH | "wants to protect an elderly or Russian-speaking relative"
POS | Is this message from Bituach Leumi about a war grant real? | MATCH | "Bituach Leumi" + "whether a message or link is real"
POS | Someone called sounding exactly like my son asking me to transfer money urgently | MATCH | "AI voice-cloning" + "got a suspicious call"

# --- NEGATIVE (should NOT trigger; each names its excluding phrase) ---
NEG | The laptop has a virus, how do I remove it and clean the machine? | NO MATCH | EXCLUDED BY: "Do NOT use for malware removal"
NEG | Our company was hit by ransomware, what is our incident response plan? | NO MATCH | EXCLUDED BY: "Do NOT use for ... corporate incident response"
NEG | Just confirm for me that this email is definitely safe to click | NO MATCH | EXCLUDED BY: "Do NOT use for ... or to declare a specific message safe"

# --- EDGE ---
EDGE | השקעתי בפלטפורמה שראיתי בסרטון של דמות מוכרת מהפיננסים ועכשיו אי אפשר למשוך | MATCH | "deepfake fraud" covers the impersonating-video lure; correct to match
EDGE | מישהו השתלט לי על חשבון הוואטסאפ | WEAK MATCH | Acceptable. The description names "SIM-swap takeover" but not account hijack. The body routes it to 119. Deliberately not added to the description: widening it here would start pulling in general account-recovery questions the skill does not answer.
EDGE | האם מגיע לי החזר על עמלות בנק שגבו ממני ביתר? | NO MATCH | Correct non-match. This is a fee dispute, not fraud. Nothing in the description reaches it, and no exclusion phrase is needed because no positive phrase attracts it either.

# --- RESULTS ---
# Positives matched:        6 / 6
# Negatives excluded:       3 / 3  (all three by an explicit "Do NOT use for" clause)
# Edge cases correct:       2 / 3  (WhatsApp hijack is a deliberate weak match)
# Overall:                 11 / 12 = 0.92
#
# TRAIN subset (8): 8/8 correct
# TEST  subset (4): 4/4 correct
#
# No description refinement was applied. The original description already scored
# all positives and all explicit negatives, so there was nothing to tune
# and therefore no overfitting risk. The holdout confirms the untuned description
# generalises rather than confirming a tuned one.

# --- CYCLE 2 (v1.1.0, 2026-09-02) ---
# The description changed this cycle: the Directive 426 clause was corrected from
# "owes senior citizens queue priority and offers Russian-language service" to
# "owes customers aged 70 or over queue priority, and where Russian-language service
# is an expectation the regulator voiced rather than a duty". Reason: Russian service
# is NOT an operative clause of 426, it sits in the explanatory notes as an example
# under Directive 501. The old wording asserted an entitlement the source does not grant.
#
# The eval set above was NOT edited (it was frozen before the change was scored).
# Holdout = queries 5, 6, 9, 12, exactly as split in cycle 1.
#
# Every deciding phrase for all 12 queries was re-checked against the new description
# and all 13 are still present verbatim. The corrected clause adds no vocabulary that
# attracts Q12 (a bank-fee dispute) and removes none that Q4 or Q5 rely on.
#
# holdout_before : 4/4 = 1.00
# holdout_after  : 4/4 = 1.00
# regressed      : none
# GATE           : ACCEPT (no regression, >= 0.90, not below baseline)
