{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.10.0"
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Notebook 2 — Evaluating Conversational Agents with StrataSynth\n",
    "\n",
    "**What you'll learn:**\n",
    "- How to evaluate a conversational AI system using StrataSynth's 10 deterministic metrics\n",
    "- What `belief_consistency`, `identity_stability`, and `behavioral_entropy` actually measure\n",
    "- How to interpret evaluation results and compare runs\n",
    "- Why deterministic metrics matter (vs. LLM self-evaluation)\n",
    "\n",
    "**Requirements:**\n",
    "```\n",
    "pip install requests pandas matplotlib seaborn\n",
    "```\n",
    "\n",
    "---\n",
    "> **API endpoint:** `https://api.stratasynth.com`  \n",
    "> Get your API key at [app.stratasynth.com](https://app.stratasynth.com)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import json\n",
    "import time\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.patches as mpatches\n",
    "import numpy as np\n",
    "\n",
    "API_KEY  = \"ss_live_...\"\n",
    "BASE_URL = \"https://api.stratasynth.com\"\n",
    "\n",
    "TOKEN = requests.post(\n",
    "    f\"{BASE_URL}/auth/token\",\n",
    "    json={\"api_key\": API_KEY}\n",
    ").json()[\"data\"][\"token\"]\n",
    "HEADERS = {\"Authorization\": f\"Bearer {TOKEN}\"}\n",
    "print(\"Authenticated.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Understanding the 10 metrics\n",
    "\n",
    "StrataSynth computes **all metrics without LLM** — only numpy, scikit-learn, and sentence-transformers.\n",
    "This matters because LLM-evaluated metrics share the same bias as LLM-generated data.\n",
    "\n",
    "There are two evaluation modes:\n",
    "\n",
    "- **Auto-evaluation** (`POST /evaluate` with just `jobId`): the dataset is evaluated against its own\n",
    "  ground truth. Only the three **intrinsic** metrics are computable; the seven **comparative** metrics\n",
    "  come back as `null` (they need an external system's output to compare against).\n",
    "- **System evaluation** (`system_output` provided): all 10 metrics are computed.\n",
    "\n",
    "| Metric | What it measures | Auto-eval |\n",
    "|---|---|---|\n",
    "| `behavioral_entropy` | Variety in communication acts (too low = robotic, too high = erratic) | ✅ computed |\n",
    "| `belief_consistency` | Do beliefs and acts correlate correctly? | ✅ computed |\n",
    "| `belief_volatility` | Do beliefs shift at a realistic rate? | ✅ computed |\n",
    "| `fact_f1` | Does your system extract the embedded facts? | `null` |\n",
    "| `noise_rejection_rate` | Does your system reject deliberately noisy turns? | `null` |\n",
    "| `identity_stability` | Do your system's identity embeddings stay stable? | `null` |\n",
    "| `affinity_smoothness` | Does your system track affinity without jumps? | `null` |\n",
    "| `reembedding_drift` | Embedding drift across re-encodings | `null` |\n",
    "| `cross_model_consistency` | Output divergence across models | `null` |\n",
    "| `episodic_segmentation_recall` | Does your system find the labeled episodes? | `null` |\n",
    "\n",
    "> Treat every metric as **nullable** in your parsers (API reference v2.6.2+).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1 — Evaluate a completed dataset (self-computed metrics)\n",
    "\n",
    "You can evaluate any completed job. The 5 self-computed metrics run on the generated dataset itself — no system output needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Replace with a COMPLETED job_id from your account\n",
    "JOB_ID = \"job_...\"\n",
    "\n",
    "# Launch evaluation\n",
    "eval_resp = requests.post(\n",
    "    f\"{BASE_URL}/evaluate\",\n",
    "    headers=HEADERS,\n",
    "    json={\"jobId\": JOB_ID}\n",
    ").json()\n",
    "\n",
    "eval_id = eval_resp[\"data\"][\"evalId\"]\n",
    "print(f\"Evaluation started: {eval_id}\")\n",
    "\n",
    "# Poll until done\n",
    "while True:\n",
    "    result = requests.get(f\"{BASE_URL}/evaluate/{eval_id}\", headers=HEADERS).json()[\"data\"]\n",
    "    status = result[\"status\"]\n",
    "    print(f\"  {status}\", end=\"\\r\")\n",
    "    if status == \"COMPLETED\":\n",
    "        print(\"\\nEvaluation complete.\")\n",
    "        metrics = result[\"metrics\"]\n",
    "        break\n",
    "    elif status == \"FAILED\":\n",
    "        raise RuntimeError(\"Evaluation failed\")\n",
    "    time.sleep(10)\n",
    "\n",
    "print(json.dumps(metrics, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2 — Interpret the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Healthy ranges (from StrataSynth documentation)\n",
    "HEALTHY_RANGES = {\n",
    "    \"noise_rejection_rate\":         (0.70, 1.00),\n",
    "    \"identity_stability\":           (0.60, 1.00),\n",
    "    \"behavioral_entropy\":           (0.40, 0.85),\n",
    "    \"belief_consistency\":           (0.50, 1.00),\n",
    "    \"belief_volatility\":            (0.05, 0.30),\n",
    "    \"fact_f1\":                      (0.60, 1.00),\n",
    "    \"reembedding_drift\":            (0.10, 0.50),\n",
    "    \"affinity_smoothness\":          (0.60, 1.00),\n",
    "    \"cross_model_consistency\":      (0.70, 1.00),\n",
    "    \"episodic_segmentation_recall\": (0.60, 1.00),\n",
    "}\n",
    "\n",
    "def score_metric(name, value):\n",
    "    if value is None:\n",
    "        return \"N/A\"\n",
    "    low, high = HEALTHY_RANGES.get(name, (0, 1))\n",
    "    if low <= value <= high:\n",
    "        return \"✅ healthy\"\n",
    "    elif value < low:\n",
    "        return \"⚠️  low\"\n",
    "    else:\n",
    "        return \"⚠️  high\"\n",
    "\n",
    "print(f\"{'Metric':<35} {'Value':>8}  {'Status'}\")\n",
    "print(\"-\" * 60)\n",
    "for name, value in metrics.items():\n",
    "    status = score_metric(name, value)\n",
    "    val_str = f\"{value:.3f}\" if isinstance(value, float) else str(value)\n",
    "    print(f\"{name:<35} {val_str:>8}  {status}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualize as radar chart\n",
    "available = {k: v for k, v in metrics.items() if isinstance(v, float)}\n",
    "names = list(available.keys())\n",
    "values = list(available.values())\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(10, 6))\n",
    "colors = [\"#22c55e\" if HEALTHY_RANGES.get(n, (0,1))[0] <= v <= HEALTHY_RANGES.get(n, (0,1))[1]\n",
    "          else \"#f97316\" for n, v in zip(names, values)]\n",
    "\n",
    "bars = ax.barh(names, values, color=colors, height=0.6)\n",
    "ax.set_xlim(0, 1)\n",
    "ax.set_xlabel(\"Score\")\n",
    "ax.set_title(\"StrataSynth Evaluation Metrics\", fontsize=14, fontweight=\"bold\")\n",
    "\n",
    "for bar, v in zip(bars, values):\n",
    "    ax.text(v + 0.01, bar.get_y() + bar.get_height()/2, f\"{v:.3f}\",\n",
    "            va=\"center\", fontsize=9)\n",
    "\n",
    "green_patch = mpatches.Patch(color=\"#22c55e\", label=\"Healthy range\")\n",
    "orange_patch = mpatches.Patch(color=\"#f97316\", label=\"Outside healthy range\")\n",
    "ax.legend(handles=[green_patch, orange_patch], loc=\"lower right\")\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3 — Deep dive: belief_consistency\n",
    "\n",
    "`belief_consistency` measures whether a speaker's **communication acts correlate with their belief state**.  \n",
    "For example: if `belief.partner_supportive` is low (0.2), you'd expect accusation or defensiveness — not reassurance.  \n",
    "A high value means the generated data is internally coherent. A low value suggests the LLM ignored the cognitive state."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the raw dataset to inspect belief_consistency manually\n",
    "# (assumes you already downloaded the JSONL from Notebook 1)\n",
    "import urllib.request\n",
    "\n",
    "job_info = requests.get(f\"{BASE_URL}/jobs/{JOB_ID}\", headers=HEADERS).json()[\"data\"]\n",
    "urllib.request.urlretrieve(job_info[\"download_url\"], \"dataset_for_eval.jsonl\")\n",
    "\n",
    "conversations = []\n",
    "with open(\"dataset_for_eval.jsonl\") as f:\n",
    "    for line in f:\n",
    "        conversations.append(json.loads(line))\n",
    "\n",
    "# Extract turns with belief_state and communication_act\n",
    "belief_rows = []\n",
    "for conv in conversations:\n",
    "    for turn in conv[\"turns\"]:\n",
    "        bs = turn.get(\"belief_state\", {})\n",
    "        belief_rows.append({\n",
    "            \"speaker\": turn[\"speaker\"],\n",
    "            \"communication_act\": turn.get(\"communication_act\"),\n",
    "            \"partner_supportive\": bs.get(\"partner_supportive\", {}).get(\"value\"),\n",
    "            \"relationship_trust\": turn.get(\"relationship_state\", {}).get(\"trust\"),\n",
    "            \"relationship_tension\": turn.get(\"relationship_state\", {}).get(\"tension\"),\n",
    "        })\n",
    "\n",
    "df_beliefs = pd.DataFrame(belief_rows).dropna()\n",
    "print(f\"Total turns with belief data: {len(df_beliefs)}\")\n",
    "\n",
    "# Average partner_supportive by communication_act\n",
    "df_beliefs.groupby(\"communication_act\")[\"partner_supportive\"].mean().sort_values().plot(\n",
    "    kind=\"barh\",\n",
    "    title=\"Avg. belief.partner_supportive by communication act\",\n",
    "    color=\"#7c3aed\",\n",
    "    figsize=(10, 6)\n",
    ")\n",
    "plt.axvline(0.5, color=\"gray\", linestyle=\"--\", alpha=0.5)\n",
    "plt.xlabel(\"belief.partner_supportive (0=distrust, 1=trust)\")\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4 — Deep dive: identity_stability\n",
    "\n",
    "`identity_stability` measures whether persona A and B behave consistently with their PsycheGraph profile across the conversation.  \n",
    "A score below 0.6 suggests the LLM drifted from the defined persona — common in long conversations or when the model ignores system prompts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualize communication act distribution per speaker\n",
    "# A persona with anxious attachment should show more seeking/accusation than a secure one\n",
    "\n",
    "act_by_speaker = df_beliefs.groupby([\"speaker\", \"communication_act\"]).size().unstack(fill_value=0)\n",
    "\n",
    "act_by_speaker.T.plot(\n",
    "    kind=\"bar\",\n",
    "    figsize=(14, 6),\n",
    "    title=\"Communication act distribution by speaker (identity consistency check)\",\n",
    "    colormap=\"Set2\"\n",
    ")\n",
    "plt.xticks(rotation=45, ha=\"right\")\n",
    "plt.ylabel(\"Turn count\")\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5 — Deep dive: behavioral_entropy\n",
    "\n",
    "`behavioral_entropy` is the Shannon entropy of the `communication_act` distribution.  \n",
    "- **Too low** (< 0.4): conversations are monotone — all accusations, or all reassurances. Boring and unrealistic.\n",
    "- **Too high** (> 0.85): completely random — no clear personality emerging. Also unrealistic.\n",
    "- **Healthy range** (0.4–0.85): varied but patterned — like real human communication."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import entropy\n",
    "\n",
    "# Compute entropy per conversation\n",
    "entropies = []\n",
    "for conv in conversations:\n",
    "    acts = [t.get(\"communication_act\") for t in conv[\"turns\"] if t.get(\"communication_act\")]\n",
    "    counts = pd.Series(acts).value_counts(normalize=True)\n",
    "    h = entropy(counts, base=2)\n",
    "    entropies.append({\"conversation_id\": conv[\"conversation_id\"], \"entropy\": h})\n",
    "\n",
    "df_ent = pd.DataFrame(entropies)\n",
    "print(f\"Mean entropy: {df_ent['entropy'].mean():.3f}\")\n",
    "print(f\"Std entropy:  {df_ent['entropy'].std():.3f}\")\n",
    "\n",
    "df_ent[\"entropy\"].hist(bins=15, color=\"#7c3aed\", edgecolor=\"white\", figsize=(8,4))\n",
    "plt.axvline(0.40, color=\"orange\", linestyle=\"--\", label=\"Healthy min (0.40)\")\n",
    "plt.axvline(0.85, color=\"orange\", linestyle=\"--\", label=\"Healthy max (0.85)\")\n",
    "plt.title(\"Behavioral entropy distribution across conversations\")\n",
    "plt.xlabel(\"Shannon entropy of communication_act\")\n",
    "plt.legend()\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6 — Compare two runs\n",
    "\n",
    "A key advantage of StrataSynth: you can compare evaluation results across different configurations (complexity, scenario, language) to understand how dataset quality varies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Replace with two eval IDs from different jobs\n",
    "EVAL_RUN_A = \"eval_...\"\n",
    "EVAL_RUN_B = \"eval_...\"\n",
    "\n",
    "results_a = requests.get(f\"{BASE_URL}/evaluate/{EVAL_RUN_A}\", headers=HEADERS).json()[\"data\"][\"metrics\"]\n",
    "results_b = requests.get(f\"{BASE_URL}/evaluate/{EVAL_RUN_B}\", headers=HEADERS).json()[\"data\"][\"metrics\"]\n",
    "\n",
    "# Build comparison DataFrame\n",
    "common_metrics = [k for k in results_a if isinstance(results_a[k], float) and k in results_b]\n",
    "df_compare = pd.DataFrame({\n",
    "    \"Metric\": common_metrics,\n",
    "    \"Run A\": [results_a[m] for m in common_metrics],\n",
    "    \"Run B\": [results_b[m] for m in common_metrics],\n",
    "})\n",
    "\n",
    "df_compare.set_index(\"Metric\").plot(\n",
    "    kind=\"barh\", figsize=(10, 6),\n",
    "    title=\"Metric comparison: Run A vs Run B\",\n",
    "    color=[\"#7c3aed\", \"#0ea5e9\"]\n",
    ")\n",
    "plt.axvline(0, color=\"black\", linewidth=0.5)\n",
    "plt.xlabel(\"Score\")\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "df_compare"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Summary\n",
    "\n",
    "StrataSynth's evaluation is **deterministic and reproducible**. The same dataset always produces the same metrics. This makes it suitable as a benchmark baseline — not a moving target.\n",
    "\n",
    "Key takeaways:\n",
    "- Use `belief_consistency` to check if cognitive state is driving behavior (or being ignored by the LLM)\n",
    "- Use `identity_stability` to detect persona drift in long conversations\n",
    "- Use `behavioral_entropy` to check for monotone outputs — a common failure mode in fine-tuned models\n",
    "- Combine multiple metrics: a system can have high `identity_stability` but low `belief_consistency` — different failure modes\n",
    "\n",
    "**Next:** See Notebook 3 to build synthetic users for product testing and agent stress-testing."
   ]
  }
 ]
}