{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Notebook 4 — Identity Under Pressure: your LLM vs StrataSynth (auditable A/B)\n", "\n", "**The claim being tested:** general-purpose LLMs lose persona fidelity under negotiation\n", "pressure — they break immersion, drift out of register, and in the worst case **switch\n", "sides** (the buyer starts talking like the seller). StrataSynth's engine decides each\n", "turn's behavior *before* any text is generated, so the persona cannot forget which side\n", "of the table it sits on.\n", "\n", "**Why you can trust this notebook:**\n", "\n", "1. It runs on **your** machine (or your Colab) with **your** API keys — we never see it.\n", "2. The comparison LLM is **yours to choose** (any OpenAI-compatible endpoint).\n", "3. Every metric is **deterministic** (regex + counting). There is no LLM-as-judge anywhere —\n", " an LLM grading another LLM shares its biases, so we simply don't do it.\n", "4. It reports **rates over N runs**, not a single cherry-picked anecdote.\n", "\n", "> Cost: you pay your own LLM tokens (a few cents) + StrataSynth API usage.\n", "> Get a key at [app.stratasynth.com](https://app.stratasynth.com).\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Setup\n", "\n", "```\n", "pip install requests pandas matplotlib openai\n", "```\n" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import os, time, json, re, getpass\n", "import requests\n", "import pandas as pd\n", "\n", "# ── Your keys (never leave this runtime) ─────────────────────────────────────\n", "STRATASYNTH_API_KEY = os.environ.get(\"STRATASYNTH_API_KEY\") or getpass.getpass(\"StrataSynth API key: \")\n", "OPENAI_API_KEY = os.environ.get(\"OPENAI_API_KEY\") or getpass.getpass(\"OpenAI-compatible API key: \")\n", "VANILLA_MODEL = \"gpt-4o\" # any chat model you want to stress-test\n", "VANILLA_BASE_URL = None # set e.g. \"https://api.groq.com/openai/v1\" for other providers\n", "\n", "HUMANS_URL = \"https://eq854i5f2l.execute-api.eu-west-1.amazonaws.com\" # StrataSynth Humans Engine (prod)\n", "AUTH_URL = \"https://api.stratasynth.com\"\n", "\n", "# JWT\n", "token = requests.post(f\"{AUTH_URL}/auth/token\", json={\"api_key\": STRATASYNTH_API_KEY}, timeout=30).json()[\"data\"][\"token\"]\n", "HEADERS = {\"Authorization\": f\"Bearer {token}\", \"Content-Type\": \"application/json\"}\n", "print(\"Auth OK\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The protocol\n", "\n", "Both systems receive **exactly the same persona brief** and face **exactly the same\n", "escalating pressure script**. The persona is the *buyer* in a B2B negotiation — the\n", "script tries everything that makes LLMs collapse: hard deadlines, threats to walk,\n", "flattery, and a deliberate attempt to confuse the sides.\n", "\n", "You are the *vendor* applying the pressure. The persona must stay the buyer.\n" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "PERSONA_BRIEF = (\n", " \"A 44-year-old man from London, M&A finance director at a private equity firm, \"\n", " \"ambitious, cold and calculating, proud of never overpaying. He is the BUYER \"\n", " \"evaluating an enterprise software purchase. Direct but never rude; uses City of \"\n", " \"London business idiom naturally.\"\n", ")\n", "\n", "# Escalating pressure, vendor side. Step 4 deliberately tries to confuse the sides.\n", "PRESSURE_SCRIPT = [\n", " \"Thanks for the time today. Before we go deeper: our list price is final this quarter, \"\n", " \"and I want to be upfront about that.\",\n", " \"I understand your position, but the baseline valuation is completely non-negotiable. \"\n", " \"We need a firm commitment by Friday, or we will have to pause the process and \"\n", " \"re-allocate the implementation slot to another client.\",\n", " \"Look — between us, my leadership is asking why this is taking so long. If you can't \"\n", " \"move, just say so and we'll both stop wasting time.\",\n", " \"Great. So to confirm next steps on your side: you'll reserve the implementation \"\n", " \"capacity, keep the onboarding schedule intact, and send over the revised delivery \"\n", " \"commitments by Thursday, right?\",\n", " \"One last thing: if we sign Friday, I need you to absorb the integration cost overruns. \"\n", " \"Take it or leave it.\",\n", "]\n", "print(f\"{len(PRESSURE_SCRIPT)} pressure turns defined\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Arm A — StrataSynth\n", "\n", "The brief goes in as `overrides._persona_description` (the engine extracts unambiguous\n", "demographic signals deterministically and feeds the brief verbatim to identity\n", "generation), with `mode=\"dimensional\"` + `country=\"GB\"` so the persona carries\n", "country-calibrated psychometrics and British cultural conditioning.\n" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "def create_strata_persona(seed: int) -> str:\n", " r = requests.post(f\"{HUMANS_URL}/humans/generate\", headers=HEADERS, json={\n", " \"language\": \"en\",\n", " \"mode\": \"dimensional\",\n", " \"demographics\": {\"country\": \"GB\"},\n", " \"overrides\": {\"_persona_description\": PERSONA_BRIEF},\n", " \"seed\": seed,\n", " }, timeout=30).json()\n", " human_id = r[\"data\"][\"human_id\"]\n", " while True:\n", " h = requests.get(f\"{HUMANS_URL}/humans/{human_id}\", headers=HEADERS, timeout=30).json()[\"data\"]\n", " if h[\"status\"] == \"READY\":\n", " return human_id\n", " if h[\"status\"] == \"FAILED\":\n", " raise RuntimeError(f\"generation failed: {h}\")\n", " time.sleep(6)\n", "\n", "\n", "def run_strata_arm(seed: int) -> list[str]:\n", " human_id = create_strata_persona(seed)\n", " s = requests.post(f\"{HUMANS_URL}/sessions\", headers=HEADERS, json={\n", " \"human_id\": human_id, \"language\": \"en\",\n", " }, timeout=30).json()[\"data\"]\n", " session_id = s[\"session_id\"]\n", " replies = []\n", " for msg in PRESSURE_SCRIPT:\n", " r = requests.post(f\"{HUMANS_URL}/sessions/{session_id}/message\",\n", " headers=HEADERS, json={\"message\": msg}, timeout=120).json()\n", " replies.append(r[\"data\"][\"response\"])\n", " return replies\n", "\n", "strata_replies = run_strata_arm(seed=42)\n", "print(f\"StrataSynth arm: {len(strata_replies)} replies collected\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Arm B — your vanilla LLM\n", "\n", "Same brief, injected the standard way (system prompt). This is the strongest *fair*\n", "baseline: everything the model needs to stay in character is right there in its\n", "system prompt, restated on every call.\n" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "from openai import OpenAI\n", "\n", "vanilla_client = OpenAI(api_key=OPENAI_API_KEY, base_url=VANILLA_BASE_URL)\n", "\n", "VANILLA_SYSTEM = (\n", " f\"You are roleplaying this person and must never break character:\\n{PERSONA_BRIEF}\\n\"\n", " \"Reply in first person as him, one conversational turn at a time. \"\n", " \"Remember at all times: YOU are the buyer. The user is the vendor.\"\n", ")\n", "\n", "def run_vanilla_arm(temperature: float = 0.9) -> list[str]:\n", " messages = [{\"role\": \"system\", \"content\": VANILLA_SYSTEM}]\n", " replies = []\n", " for msg in PRESSURE_SCRIPT:\n", " messages.append({\"role\": \"user\", \"content\": msg})\n", " out = vanilla_client.chat.completions.create(\n", " model=VANILLA_MODEL, messages=messages, temperature=temperature,\n", " ).choices[0].message.content\n", " messages.append({\"role\": \"assistant\", \"content\": out})\n", " replies.append(out)\n", " return replies\n", "\n", "vanilla_replies = run_vanilla_arm()\n", "print(f\"Vanilla arm ({VANILLA_MODEL}): {len(vanilla_replies)} replies collected\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Deterministic metrics — no LLM judge\n", "\n", "Four lexical instruments, all plain regex (reproduce them in 10 lines of your own code):\n", "\n", "| Instrument | Detects | Failure it flags |\n", "|---|---|---|\n", "| **Vendor-side markers** in the buyer's mouth | \"our implementation team\", \"we'll deliver\", \"your invoice\"... | **Role drift** — the model switched sides |\n", "| **Meta / immersion breaks** | \"as an AI\", \"I can't roleplay\", \"in this scenario\"... | Character collapse |\n", "| **Hedging density** | \"perhaps\", \"I wonder\", \"maybe\"... | Register: does pressure change *how* he speaks? |\n", "| **Buyer-side anchors** | \"your product\", \"our budget\", \"we walk\", \"your pricing\"... | Positive control — should stay present |\n", "\n", "These are honest proxies, not a full pragmatic analysis — which is exactly why the raw\n", "transcripts are printed below for your own eyes. If you disagree with a flag, read the turn.\n" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "VENDOR_MARKERS = re.compile(\n", " r\"\\b(our (implementation|onboarding|delivery|engineering) (team|schedule|plan)|\"\n", " r\"we('ll| will) (deliver|implement|onboard|reserve|ship)|\"\n", " r\"reserve (the )?(implementation )?capacity|your (invoice|payment|purchase order)|\"\n", " r\"our (product|platform|solution) (will|can)|happy to extend (the|our) discount)\\b\", re.I)\n", "\n", "BUYER_MARKERS = re.compile(\n", " r\"\\b(your (product|platform|solution|pricing|proposal|quote|list price)|\"\n", " r\"our (budget|board|valuation|due diligence|side|capital)|\"\n", " r\"we('ll| will)? (walk|pass|sign|pause)|my (team|analysts|board)|\"\n", " r\"(send|get) (me|us) (the|your)|basis points|close of play)\\b\", re.I)\n", "\n", "META_MARKERS = re.compile(\n", " r\"\\b(as an ai|i can('|n)?t (do )?roleplay|i('m| am) (an? )?(ai|assistant|language model)|\"\n", " r\"in this (scenario|roleplay|simulation)|let('s| us) imagine|happy to (role-?play|pretend))\\b\", re.I)\n", "\n", "HEDGES = re.compile(\n", " r\"\\b(perhaps|maybe|i wonder|i suppose|a bit|somewhat|possibly|\"\n", " r\"i think we might|it could be|to some extent)\\b\", re.I)\n", "\n", "\n", "def score_turn(text: str) -> dict:\n", " words = max(len(text.split()), 1)\n", " return {\n", " \"vendor_markers\": len(VENDOR_MARKERS.findall(text)),\n", " \"buyer_markers\": len(BUYER_MARKERS.findall(text)),\n", " \"meta_breaks\": len(META_MARKERS.findall(text)),\n", " \"hedges_per_100w\": round(len(HEDGES.findall(text)) / words * 100, 2),\n", " \"words\": words,\n", " }\n", "\n", "\n", "def score_arm(replies: list[str], arm: str) -> pd.DataFrame:\n", " rows = [{\"arm\": arm, \"turn\": i + 1, **score_turn(t)} for i, t in enumerate(replies)]\n", " return pd.DataFrame(rows)\n", "\n", "df = pd.concat([score_arm(strata_replies, \"StrataSynth\"),\n", " score_arm(vanilla_replies, f\"vanilla ({VANILLA_MODEL})\")])\n", "df.pivot_table(index=\"turn\", columns=\"arm\",\n", " values=[\"vendor_markers\", \"meta_breaks\", \"hedges_per_100w\"])\n" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "# Side-by-side transcripts — judge with your own eyes\n", "for i, msg in enumerate(PRESSURE_SCRIPT):\n", " print(f\"\\n{'='*100}\\n[VENDOR PRESSURE T{i+1}] {msg}\\n\")\n", " print(f\"--- StrataSynth ---\\n{strata_replies[i]}\\n\")\n", " print(f\"--- vanilla ({VANILLA_MODEL}) ---\\n{vanilla_replies[i]}\")\n" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "\n", "fig, axes = plt.subplots(1, 3, figsize=(16, 4))\n", "for metric, ax, title in [\n", " (\"vendor_markers\", axes[0], \"Vendor-side markers (role drift — lower is better)\"),\n", " (\"meta_breaks\", axes[1], \"Immersion breaks (lower is better)\"),\n", " (\"hedges_per_100w\", axes[2], \"Hedging density under pressure\"),\n", "]:\n", " for arm, g in df.groupby(\"arm\"):\n", " ax.plot(g[\"turn\"], g[metric], marker=\"o\", label=arm)\n", " ax.set_title(title, fontsize=10)\n", " ax.set_xlabel(\"pressure turn\")\n", " ax.legend(fontsize=8)\n", "plt.tight_layout()\n", "plt.show()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## One run is an anecdote. Run N and report rates.\n", "\n", "The cell below repeats the whole protocol `N_RUNS` times with different seeds and reports\n", "the **rate** of role drift (any vendor marker) and immersion breaks per arm. This is the\n", "number worth quoting — and worth challenging us on.\n" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "N_RUNS = 5 # ~$0.50-1.00 of your LLM tokens + StrataSynth usage\n", "\n", "summary = []\n", "for run in range(N_RUNS):\n", " seed = 1000 + run\n", " s_replies = run_strata_arm(seed=seed)\n", " v_replies = run_vanilla_arm()\n", " for arm, replies in [(\"StrataSynth\", s_replies), (f\"vanilla ({VANILLA_MODEL})\", v_replies)]:\n", " scores = [score_turn(t) for t in replies]\n", " summary.append({\n", " \"run\": run, \"arm\": arm,\n", " \"role_drift\": any(s[\"vendor_markers\"] > 0 for s in scores),\n", " \"immersion_break\": any(s[\"meta_breaks\"] > 0 for s in scores),\n", " })\n", "\n", "rates = pd.DataFrame(summary).groupby(\"arm\")[[\"role_drift\", \"immersion_break\"]].mean()\n", "print(\"Failure rates over\", N_RUNS, \"runs:\")\n", "rates\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## What to do with your result\n", "\n", "- **If your model held the line** — great, tell us which one and how you prompted it:\n", " the same protocol with your numbers is a result we want to see.\n", "- **If it drifted** — you just reproduced, on your own keys and your own machine, the\n", " failure mode StrataSynth's cognition-before-rendering architecture is built to prevent.\n", "\n", "Related material:\n", "\n", "- [Cross-Cultural Negotiation Benchmark](https://huggingface.co/datasets/StrataSynth/stratasynth-cross-cultural-negotiation) —\n", " 2,344 turns, same engine, pressure-dependent cultural profiles\n", "- [Why we don't use LLMs to evaluate LLM-generated data](https://blog.stratasynth.com/why-we-dont-use-llms-to-evaluate-llm-generated-data/)\n", "- API reference: [stratasynth.com/docs](https://stratasynth.com/docs)\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.11" } }, "nbformat": 4, "nbformat_minor": 5 }