# Phase 5 Summary — AI Modules & External Integrations

Built on Phases 1-4 without redesigning anything: one additive
migration (`0016`), the same MVC conventions. Like Phase 4, several
tables here (`conversations`, `conversation_messages`,
`chatbot_settings`, `integrations`) have existed since Phase 1 —
this phase is mostly the AI Assistant, Chatbot, and Voice Agent
application code finally using them, plus the genuinely new pieces
(prompt management, knowledge base, training data, webhook audit log).

## A note on testing in this environment

This sandbox has **no outbound internet access and no `curl`
extension installed**. Every external API client
(`OpenAiClient`, `TwilioClient`, `ElevenLabsClient`, `GoogleOAuthService`,
`GoogleCalendarClient`) was built to the real, documented API contract
of its provider and verified as thoroughly as this constraint allows:

- **Everything that doesn't require a live network call was tested
  directly and is fully verified**: tool dispatch logic, conversation
  storage, lead/duplicate handling, TwiML XML generation, OAuth
  authorization URL construction, and — critically — **webhook
  signature verification, checked against Twilio's own officially
  published test vector** (not just self-consistency: the exact
  input/output pair from Twilio's docs, byte for byte).
- **Every client fails gracefully, by design**, the same pattern
  established for `MailService` in Phase 1: `isConfigured()` checks
  first, and if `curl` itself is unavailable (as it is in this
  sandbox) or a request fails, the client returns a structured
  `['success' => false, 'error' => ...]` rather than crashing. This
  was itself verified live — asking the AI Assistant a question with
  no `curl` extension present correctly returned "Server is missing
  the PHP curl extension" instead of a fatal error.
- **What genuinely cannot be verified here**: an actual successful
  round-trip to OpenAI, Twilio, ElevenLabs, or Google's servers. A
  real deployment needs `composer require phpmailer/phpmailer` (for
  mail, per Phase 1) and the `curl` PHP extension enabled — both
  standard on any real hosting environment — plus real API
  credentials entered via Admin > Integrations.

## What was built

### Modules
| Module | Where |
|---|---|
| AI Assistant | `AiAssistantService` (tool-calling loop) + `AiToolService` (the 6 real tools) + `AiAssistantController` |
| Website Chatbot | `ChatbotService` + `ChatbotController` — public widget endpoints, no auth |
| AI Voice Agent | `VoiceAgentService` + `TwilioClient` (TwiML) + `VoiceAgentController` (Twilio webhooks, signature-verified) |
| Conversation Logs | `ConversationModel` (unifies all 3 channels) + `ConversationController` |
| Knowledge Base | `KnowledgeBaseModel` (FULLTEXT + LIKE fallback) + `KnowledgeBaseController` |
| Prompt Management | `AiPromptModel` (per-feature, per-business-line) + `AiPromptController` |
| Training Data | `TrainingExampleModel` (+ JSONL export) + `TrainingDataController` |
| AI Settings | Folded into Integrations (below) — every provider's config, including OpenAI's model choice, lives in one hub rather than a separate settings page |

### AI Assistant tools (all 6 real, tested independently of the LLM)
Search Customers, Search Leads, Book Appointments (reuses Phase 3's
`AppointmentService::book()` — same conflict detection), Generate
Reports (reuses `ReportService`, the same data a human sees on the
Reports page), Answer Questions (searches the Knowledge Base), and
Summarize Conversations (reads real `conversation_messages` rows).
`AiToolService::call()` was exercised directly through `ReportService`/
`KnowledgeBaseModel`/`AppointmentService`'s own test coverage — the
dispatch layer itself has no LLM dependency.

### Website Chatbot features
Capture Leads and Collect Contact Information (regex-based email/phone/
name extraction — a real bug was caught and fixed here, see below),
Book Appointment (hands off to the AI Assistant's tool loop if OpenAI
is configured, otherwise safely captures the request as a lead rather
than dropping it), FAQ (Knowledge Base search), Escalate to Human
(keyword-triggered, notifies staff via the existing
`NotificationService`).

### AI Voice features
Incoming Calls and Outgoing Calls (`TwilioClient::initiateCall()`),
Lead Qualification and Appointment Booking (`VoiceAgentService` reuses
`ChatbotService`'s exact logic — a phone call transcribed to text is
the same kind of "visitor turn"), Conversation Recording (Twilio's
`RecordingUrl` stored on the conversation), Transcripts (both
turn-by-turn `conversation_messages` and a flattened
`transcript_text`). The full call flow — incoming call, TwiML, speech
input, TwiML, call end, transcript — was run end-to-end live and
produces valid, well-formed TwiML at every step.

### Integrations (all 15: 6 from Phase 1 + 9 new this phase)
OpenAI, ElevenLabs, Twilio, WhatsApp Business API, Google OAuth,
Google Calendar, Facebook Lead Ads, Instagram Lead Ads, TikTok Lead
Generation, Meta Pixel, Google Analytics — plus Phase 1's Gmail,
QuickBooks, Slack, Google Business Profile. Every credential is
AES-256-CBC encrypted at rest (verified round-trip). API-key-style
providers (OpenAI, ElevenLabs, Twilio, WhatsApp, Meta Pixel, Google
Analytics) use a simple form; OAuth providers (Google OAuth/Calendar,
Facebook, Instagram, TikTok) use the standard authorization-code flow
— `buildAuthorizationUrl()` was verified to produce a correct,
Google-spec-compliant URL with the right scope, `access_type=offline`
(needed for a refresh token), and CSRF `state` round-tripped through
session storage.

## What was actually tested (not just written)

- **TwiML generation**: every response type (`Say`+`Hangup`,
  `Gather`+`Say`, `Record`) validated as well-formed XML via
  `DOMDocument`, including correct escaping of `&`, `<`, `>`, and
  quotes in dynamic text.
- **Twilio webhook signature verification**: checked against Twilio's
  own officially published test vector (Auth Token `12345`, a known
  URL and POST params, expected signature
  `RSOYDt4T1cUTdK1PDd93/VVr8B8=`) — matched exactly. A tampered
  signature was confirmed rejected.
- **Google OAuth URL building**: correct host, `client_id`, `scope`,
  `access_type=offline`, and `state` round-trip, verified against a
  configured test integration.
- **Full chatbot conversation flow**: FAQ match, human-escalation
  (with a real staff notification created), lead capture from a
  message containing an email, and the booking-intent fallback both
  with and without contact info already captured.
- **Full voice call flow**: incoming call, valid TwiML, speech input
  (reusing chatbot logic), valid TwiML, call end, transcript correctly
  assembled, duration/recording URL stored.
- **Knowledge base search**: FULLTEXT match, and the LIKE fallback for
  queries FULLTEXT can't handle.
- **Integration credential encryption**: connect/disconnect round-trip,
  decrypted value matches the original exactly, encrypted-at-rest
  value is never the plaintext.
- **Meta lead-ad webhook processing**: a fake field-fetcher (standing
  in for the real Graph API call) was injected to prove the
  lead-creation and duplicate-detection logic independent of network
  access — processing the same `leadgen_id` twice correctly created
  one lead, not two.
- **Full HTTP flow**: real login, every AI admin page returns 200,
  the public chatbot widget (no auth) starts a conversation and
  answers an FAQ, a Knowledge Base article is created via AJAX, an
  integration is connected/disconnected via AJAX, and the resulting
  conversation is visible and correctly stored via Conversation Logs.

### Bugs caught by testing, not assumed away
1. **Two controllers (`AiPromptController`, `TrainingDataController`)
   didn't match their views' expected variable names** — a real
   integration bug between two files, invisible to `php -l` (which
   only checks syntax, not cross-file contracts) and only surfaced by
   actually loading the pages over HTTP, where both returned HTTP 500.
   Fixed by aligning the controllers and views to the same shape
   (grouped-by-feature) and re-verified both pages return 200.
2. **A regex for extracting a caller/visitor's name from free text was
   catastrophically greedy** — "my name is Sarah Test and my email
   is..." captured the ENTIRE rest of the sentence as the "name"
   instead of stopping at two words. Caught by inspecting the actual
   lead record created during a live chatbot test, not by reading the
   regex. Fixed by capping the capture to at most two words.
3. **`Request::all()` merges query-string and POST-body params**,
   which is wrong for Twilio webhook signature verification — Twilio's
   signature only ever covers the POST body, since the query string is
   already part of the signed URL. Using the merged version would have
   made every gather-webhook (which includes `?conversation_id=X` in
   its URL) fail signature verification for a legitimate Twilio
   request. Caught during code review before it ever ran, by tracing
   through exactly what Twilio's algorithm signs — fixed by adding a
   `Request::bodyOnly()` accessor and using it specifically for
   signature verification.

## Design decisions worth flagging explicitly

- **AI Settings has no dedicated page** — every provider's
  configuration (including OpenAI's model choice) lives in the
  Integrations hub. A separate "AI Settings" page would just be a
  subset of the same connect/configure UI already there.
- **The Voice Agent's "brain" is the Chatbot's brain.** Rather than
  writing lead-qualification/booking/escalation logic twice,
  `VoiceAgentService::handleSpeechInput()` hands the caller's
  (Twilio-transcribed) speech straight to
  `ChatbotService::handleMessage()` and converts the resulting reply
  to TwiML. A phone call and a chat message are the same kind of
  "visitor turn" once transcribed to text — duplicating the logic
  would just be two places to keep the same behavior in sync.
- **Appointment Reminder stays a Phase 3 service call, not a generic
  engine workflow** (same decision as Phase 4's automation rules) —
  it's listed as a workflow row for visibility, but
  `AppointmentService::sendDueReminders()` still does the actual work.

## Phase 6 readiness (deeper reporting/analytics, or further integration depth)

1. **`ReportService`** already returns the same structured data to a
   human (Reports page) and to the AI (`generate_report` tool) — any
   future reporting/BI work extends this one service rather than
   building parallel query logic.
2. **`webhook_events`** is a ready-made audit trail for any future
   provider's webhook (Stripe, QuickBooks, additional social
   platforms) — the same `WebhookEventModel::record()`/
   `markProcessed()`/`markFailed()` pattern applies unchanged.
3. **`GoogleCalendarClient`** already handles token refresh
   transparently; wiring it into `AppointmentService::book()`/
   `reschedule()`/`cancel()` (create/update/delete the matching Google
   Calendar event, storing the id on
   `appointments.google_calendar_event_id`) is the one piece of
   "Google Calendar" from the brief not yet wired end-to-end — the
   client is ready, the appointment lifecycle hooks are the remaining
   work.
4. **Meta Pixel / Google Analytics** tracking IDs are stored and
   retrievable (`MetaIntegrationService::pixelId()`/
   `googleAnalyticsId()`) but have no public-facing page to embed into
   yet, since this application is an internal CRM, not the marketing
   website itself — the values are ready for whichever public site
   embeds the snippet.
