Orbital
ArticlesAbout

Why Your AI Chatbot Failed: Building Bespoke AI Applications That Actually Solve Problems

The pattern is predictable: Company spins up an AI pilot. Impressive demo in week one. Promising initial feedback. Then... crickets. Six months later, usage has flatlined at 3% and the project quietly dies in the backlog. The chatbot that was supposed to "transform how we work" gets one question every three days, usually from the person who built it.

I've seen this play out dozens of times across different companies, different industries, different use cases. The superficial diagnosis is always "people don't trust AI" or "change management is hard." But that's not the real problem.

The real problem is you built a demo, not a solution.

The Demo Trap: Why Generic AI Tools Fail

Let's start by examining what most companies actually build when they say they're "integrating AI":

Pattern 1: The Generic Chatbot

// What most "AI integrations" look like
async function handleUserQuery(query: string) {
  const response = await fetch('https://api.anthropic.com/v1/messages', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      model: 'claude-sonnet-4-20250514',
      max_tokens: 1000,
      messages: [
        { role: 'user', content: query }
      ]
    })
  });

  return response.json();
}

// That's it. That's the whole integration.

This is a search bar with extra steps. It looks impressive in demos because LLMs are impressive. But it provides no business value because:

  1. No context: The AI knows nothing about your systems, your data, or your workflows
  2. No actions: It can only generate text, not actually do anything
  3. No integration: It sits separately from where people do actual work
  4. No specificity: It's trying to be helpful for everything, which means it's optimized for nothing

The demo works because the person demonstrating it asks carefully crafted questions that showcase the LLM's general knowledge. In production, users ask about their specific situation with their specific data in their specific workflow — and the chatbot has no access to any of that context.

Pattern 2: "Talk to Your Data"

The slightly more sophisticated version adds retrieval:

async function handleQueryWithRetrieval(query: string) {
  // Step 1: Find possibly relevant documents
  const relevantDocs = await vectorDB.search(query, { limit: 5 });

  // Step 2: Jam them into the prompt
  const context = relevantDocs.map(doc => doc.content).join('\n\n');

  // Step 3: Hope for the best
  const response = await llm.query(`
    Context: ${context}

    Question: ${query}
  `);

  return response;
}

This fails for different reasons:

  1. Semantic search isn't magic: Embeddings capture general similarity, not domain-specific relevance
  2. No understanding of data relationships: Your customer record links to orders, tickets, invoices — those relationships matter
  3. Static retrieval: Doesn't adapt based on what the LLM actually needs
  4. No verification: Wrong retrieved context → wrong answers → users stop trusting it

Again, the demo looks great. Someone asks "what are our Q3 numbers?" and gets back Q3 numbers (maybe). In production, someone asks "why did the Chicago deal stall and what should we do about it?" and gets back a summary of the Chicago office's snack preferences because that's what matched the embedding.

Pattern 3: The Everything Dashboard

Then there's the "AI-powered dashboard" approach:

interface DashboardState {
  // Every possible data point
  customers: Customer[];
  orders: Order[];
  analytics: Analytics;
  tickets: Ticket[];
  forecasts: Forecast[];
  // ... 50 more fields
}

async function generateInsights(data: DashboardState) {
  // Serialize 10MB of JSON
  const prompt = `Given this data: ${JSON.stringify(data)}, provide insights.`;
  return await llm.query(prompt);
}

This approach:

  1. Overwhelms the context window with irrelevant data
  2. Costs a fortune in tokens (you're paying for 10MB of serialized JSON every query)
  3. Provides generic insights ("your top customers generate the most revenue!")
  4. Still can't take action — it can only comment on what's happening

The fatal flaw in all three patterns: they're solutions looking for problems, not problems getting solved.

What Bespoke AI Integration Actually Means

Real AI integration starts with a specific problem in a specific workflow. Not "let's add AI to our product" but "our fraud analysts spend 6 hours per day manually triaging alerts — can we make that faster?"

The Fraud Detection Example

Here's what bespoke integration looks like in practice:

interface FraudAlert {
  id: string;
  accountId: string;
  transactionAmount: number;
  merchant: string;
  location: GeoCoordinates;
  timestamp: Date;
  riskScore: number;
  flaggedRules: string[];
}

interface AnalystContext {
  currentAlert: FraudAlert;
  accountHistory: Transaction[];
  similarPatterns: FraudPattern[];
  recentDecisions: Decision[];
  openAlerts: FraudAlert[];
}

// This is NOT a generic chatbot
class FraudAnalysisAgent {
  async analyzeAlert(alert: FraudAlert): Promise<AnalysisResult> {
    // Get ONLY the context needed for THIS specific alert
    const context = await this.buildRelevantContext(alert);

    // Call LLM with structured prompt designed for fraud analysis
    const analysis = await this.llm.analyze({
      systemPrompt: this.buildFraudAnalysisPrompt(),
      alert: alert,
      context: context,
      outputSchema: FraudAnalysisSchema,
    });

    // Return structured data, not free text
    return {
      recommendation: analysis.recommendation, // "approve" | "decline" | "escalate"
      confidence: analysis.confidence,
      reasoning: analysis.reasoning,
      similarCases: analysis.similarCases,
      suggestedActions: analysis.suggestedActions,
    };
  }

  private async buildRelevantContext(alert: FraudAlert): Promise<AnalystContext> {
    // Parallel fetch of only relevant data
    const [account, history, patterns, recent] = await Promise.all([
      this.db.accounts.findById(alert.accountId),
      this.db.transactions.getRecent(alert.accountId, { days: 90 }),
      this.patterns.findSimilar(alert, { limit: 5 }),
      this.db.decisions.getRecent({ days: 7 }),
    ]);

    return {
      currentAlert: alert,
      accountHistory: this.filterRelevantTransactions(history, alert),
      similarPatterns: patterns,
      recentDecisions: recent,
      openAlerts: await this.db.alerts.getOpen(alert.accountId),
    };
  }

  private buildFraudAnalysisPrompt(): string {
    return `You are a fraud analyst assistant. Your job is to:

    1. Analyze the current alert in context of account history
    2. Compare to known fraud patterns
    3. Consider recent similar decisions for consistency
    4. Recommend one of: approve, decline, escalate
    5. Provide specific reasoning based on evidence

    You have access to:
    - Current transaction details and risk score
    - 90 days of account transaction history
    - Similar fraud patterns from our database
    - Recent analyst decisions on similar cases

    Focus on patterns like:
    - Velocity (rapid succession of transactions)
    - Geographic anomalies (location jumps)
    - Merchant category changes
    - Amount anomalies vs. typical behavior

    Be specific. Reference actual data points.`;
  }
}

Notice what's different:

  1. Deeply integrated: Pulls from multiple data sources (accounts, transactions, patterns, decisions)
  2. Contextually aware: Only fetches data relevant to THIS alert
  3. Domain-specific: The prompts, data structures, and logic are all designed for fraud analysis
  4. Structured output: Returns typed data that drives UI and workflow, not just text
  5. Actionable: Generates recommendations that can be acted on

This isn't a generic chatbot. It's a specialized tool that makes fraud analysts more effective at their specific job.

The Distributed Systems Example

Or consider making complex distributed systems more explorable:

interface ServiceHealth {
  serviceName: string;
  region: string;
  instanceCount: number;
  errorRate: number;
  latencyP95: number;
  dependencies: string[];
  recentDeploys: Deploy[];
}

class SystemAnalysisAgent {
  async diagnoseIssue(
    symptoms: string,
    affectedServices: string[]
  ): Promise<DiagnosisResult> {
    // Get real-time system state
    const systemState = await this.observability.getCurrentState({
      services: affectedServices,
      timeWindow: '1h',
      includeMetrics: true,
      includeLogs: true,
      includeTraces: true,
    });

    // Get recent changes
    const recentChanges = await this.deployments.getRecent({
      services: affectedServices,
      hours: 24,
    });

    // Check known issues
    const knownIssues = await this.knowledge.findSimilar(symptoms);

    const analysis = await this.llm.diagnose({
      systemPrompt: this.buildDiagnosisPrompt(),
      currentState: systemState,
      recentChanges: recentChanges,
      knownPatterns: knownIssues,
      symptoms: symptoms,
      outputSchema: DiagnosisSchema,
    });

    return {
      likelyCause: analysis.cause,
      affectedComponents: analysis.components,
      suggestedInvestigation: analysis.investigationSteps,
      similarIncidents: analysis.historicalMatches,
      remediationOptions: analysis.remediation,
      visualizations: this.generateVisualizations(systemState, analysis),
    };
  }

  private generateVisualizations(
    state: SystemState,
    analysis: Analysis
  ): Visualization[] {
    // Generate React components showing:
    // - Service dependency graph highlighting problem areas
    // - Timeline of error rate spikes correlated with deploys
    // - Latency distribution across service boundaries
    // - Trace waterfall of slow requests

    return [
      this.createDependencyGraph(state, analysis.components),
      this.createTimelineLlChart(state.metrics, analysis.timeRange),
      this.createLatencyHeatmap(state.latency),
      this.createTraceWaterfall(state.traces),
    ];
  }
}

This agent:

  1. Understands your architecture: Knows about services, dependencies, deployments
  2. Correlates multiple signals: Metrics, logs, traces, recent changes
  3. Generates visual explanations: Returns React components, not just text
  4. Provides investigation paths: Tells you specifically what to check next
  5. Learns from history: References similar past incidents

It's not trying to be helpful for everything. It's built specifically to help engineers debug distributed systems.

The Three Critical Components

Bespoke AI applications that actually work share three characteristics:

1. Deep Integration with Specific Systems

Not "connected to" but "deeply integrated with." This means:

You understand the data model:

// Bad: Generic data access
async function getData(query: string) {
  const results = await db.query(query);
  return results;
}

// Good: Domain-specific data access
async function getCustomerJourneyContext(customerId: string) {
  const [customer, purchases, support, interactions] = await Promise.all([
    this.db.customers.findById(customerId),
    this.db.purchases.getHistory(customerId, { limit: 50 }),
    this.db.support.getTickets(customerId, { status: 'open' }),
    this.db.marketing.getInteractions(customerId, { days: 90 }),
  ]);

  return {
    profile: {
      lifetimeValue: customer.ltv,
      cohort: customer.cohort,
      segment: customer.segment,
      riskLevel: customer.churnRisk,
    },
    recentPurchases: this.summarizePurchases(purchases),
    supportIssues: this.categorizeSupportTickets(support),
    engagementPattern: this.analyzeEngagement(interactions),
  };
}

You expose the right capabilities:

// Bad: Generic actions
interface Actions {
  create(type: string, data: any): Promise<void>;
  update(id: string, data: any): Promise<void>;
  delete(id: string): Promise<void>;
}

// Good: Domain-specific actions
interface FraudActions {
  approveTransaction(alertId: string, reason: string): Promise<void>;
  declineTransaction(alertId: string, reason: string): Promise<void>;
  escalateToHuman(alertId: string, context: string): Promise<void>;
  addToWatchlist(accountId: string, duration: Duration): Promise<void>;
  updateRiskRules(ruleId: string, params: RuleParams): Promise<void>;
}

You maintain context across interactions:

// Bad: Stateless requests
async function handleQuery(query: string) {
  return await llm.query(query);
}

// Good: Contextual sessions
class AnalysisSession {
  private context: SessionContext;
  private history: Interaction[];

  async query(input: string): Promise<Response> {
    // LLM has full conversation history and session context
    const response = await this.llm.query({
      history: this.history,
      context: this.context,
      input: input,
    });

    // Update session state based on response
    this.updateContext(response);
    this.history.push({ input, response });

    return response;
  }

  private updateContext(response: Response) {
    // Track what entities we're discussing
    if (response.mentionedEntities) {
      this.context.entities.push(...response.mentionedEntities);
    }

    // Track what actions were taken
    if (response.actionsTaken) {
      this.context.actions.push(...response.actionsTaken);
    }

    // Track what the user is trying to accomplish
    if (response.inferredGoal) {
      this.context.goals.push(response.inferredGoal);
    }
  }
}

2. Human-in-the-Loop UX Patterns

AI shouldn't be autonomous. It should be a tool that amplifies human judgment. This requires careful UX design:

Suggestion → Review → Approve:

interface FraudRecommendation {
  action: 'approve' | 'decline' | 'escalate';
  confidence: number;
  reasoning: string[];
  evidencePoints: Evidence[];
  similarCases: Case[];
}

// The UI shows:
// 1. The recommendation prominently
// 2. The reasoning clearly
// 3. The evidence interactively
// 4. Similar cases for comparison
// 5. One-click approve OR manual override

function FraudReviewUI({ alert, recommendation }: Props) {
  return (
    <div className="fraud-review">
      <RecommendationCard
        action={recommendation.action}
        confidence={recommendation.confidence}
      />

      <ReasoningSection
        points={recommendation.reasoning}
        evidence={recommendation.evidencePoints}
      />

      <SimilarCasesCarousel
        cases={recommendation.similarCases}
      />

      <ActionButtons>
        <Button
          variant="primary"
          onClick={() => approve(alert.id)}
        >
          Accept Recommendation
        </Button>
        <Button
          variant="secondary"
          onClick={() => openManualReview(alert)}
        >
          Review Manually
        </Button>
      </ActionButtons>
    </div>
  );
}

Partial Commitment:

// Don't make users commit to the full AI response
// Let them accept parts and modify others

interface AnalysisResponse {
  sections: AnalysisSection[];
}

interface AnalysisSection {
  id: string;
  content: string;
  confidence: number;
  editable: boolean;
}

function AnalysisReviewUI({ analysis }: Props) {
  const [sections, setSections] = useState(analysis.sections);

  const acceptSection = (id: string) => {
    setSections(sections.map(s =>
      s.id === id ? { ...s, accepted: true } : s
    ));
  };

  const editSection = (id: string, newContent: string) => {
    setSections(sections.map(s =>
      s.id === id ? { ...s, content: newContent, edited: true } : s
    ));
  };

  return (
    <div className="analysis-review">
      {sections.map(section => (
        <SectionCard
          key={section.id}
          section={section}
          onAccept={() => acceptSection(section.id)}
          onEdit={(content) => editSection(section.id, content)}
        />
      ))}
    </div>
  );
}

Undo-Friendly Actions:

// Make AI actions reversible

interface Action {
  id: string;
  type: string;
  timestamp: Date;
  reversible: boolean;
  reverseAction?: () => Promise<void>;
}

class ActionManager {
  private actionHistory: Action[] = [];

  async executeAction(action: Action): Promise<void> {
    // Execute the action
    await action.execute();

    // Track it
    this.actionHistory.push(action);

    // Show undo notification
    if (action.reversible) {
      this.showUndoNotification(action);
    }
  }

  async undoAction(actionId: string): Promise<void> {
    const action = this.actionHistory.find(a => a.id === actionId);
    if (!action || !action.reversible) {
      throw new Error('Action cannot be undone');
    }

    await action.reverseAction?.();
    this.actionHistory = this.actionHistory.filter(a => a.id !== actionId);
  }
}

3. Solving ONE Problem Extremely Well

This is the hardest part. You have to resist the temptation to build a general-purpose AI assistant. Pick one workflow, one pain point, one job to be done — and nail it.

  • Fraud analysis: Make fraud analysts 3x faster at triaging alerts
  • Customer support: Reduce time to first response by 50%
  • Incident response: Cut MTTR in half for common incident types
  • Sales qualification: Automate 80% of initial lead research

Not all of the above. Pick one. Build it deeply integrated with your specific systems, with carefully designed human-in-the-loop patterns, for that one specific job.

Once it works and people actually use it, then consider expanding scope.

Why Bespoke Integration Wins

The difference between generic AI tools and bespoke integration is the difference between a calculator app and Excel.

The calculator can do arithmetic. Excel can do arithmetic AND financial modeling AND data analysis AND charting AND pivots AND... you get the idea. Excel isn't trying to be helpful for everything; it's deeply optimized for working with tabular data and formulas. That specificity is what makes it powerful.

Same with bespoke AI integration:

Generic chatbot: "I can answer questions!"

Bespoke fraud agent: "I can analyze this alert against your account history, compare it to known fraud patterns, check recent similar decisions for consistency, and recommend approve/decline/escalate with specific reasoning."

Which one actually helps your fraud analysts do their job?

Generic RAG: "I can search your documents!"

Bespoke support agent: "I can pull this customer's purchase history, active support tickets, and previous interactions, identify the root cause of their issue, suggest resolution steps based on similar cases, and draft a response in your company's tone."

Which one actually helps your support team?

Generic dashboard: "I can visualize your data!"

Bespoke system analyzer: "I can correlate your error rates with recent deployments, identify which service is causing the latency spike, show you the trace of a slow request, and suggest which component to investigate first."

Which one actually helps your on-call engineers?

What This Means for Your AI Strategy

If you're planning an AI initiative, here's how to avoid building another abandoned chatbot:

Start with the Problem, Not the Technology

Don't ask "how can we use AI?" Ask "what are our most time-consuming manual processes?" or "where are our teams spending hours on work that feels automatable?"

Find the workflow where your team says "I wish there was a better way to do this."

Build for a Specific User in a Specific Context

Not "our customers" but "Sarah in fraud operations when she's triaging alerts in the morning."

Not "our engineering team" but "Dev on-call when they're debugging a production incident at 3 AM."

The more specific you are about WHO will use this and WHEN and WHY, the better the solution will be.

Make Integration Deep, Not Broad

Don't try to connect to every system. Pick the 2-3 systems that matter for THIS workflow and integrate deeply:

  • Understand the data model
  • Expose domain-specific capabilities
  • Maintain workflow context
  • Generate structured outputs

Ten shallow integrations are worth less than one deep integration.

Design Human-in-the-Loop Patterns

AI isn't replacing humans in most business workflows. It's augmenting their judgment. Your UX should reflect this:

  • Show AI reasoning, don't hide it
  • Make suggestions, not decisions
  • Allow partial acceptance of AI outputs
  • Make actions reversible
  • Provide escape hatches for edge cases

Measure Real Adoption, Not Engagement

"Monthly active users" is a vanity metric. What matters is:

  • Are people using this in their actual workflow?
  • Are they faster at their job because of it?
  • Do they trust the recommendations enough to act on them?
  • Would they be upset if you took it away?

If the answer to any of those is "no," you've built a demo, not a solution.

The Path Forward

The chatbot wave is cresting. Companies are realizing that generic AI tools don't deliver business value. The next wave will be bespoke integrations: AI deeply integrated into specific workflows, designed for specific use cases, solving specific problems.

This requires different skills than chatbot-building:

  • Deep understanding of business workflows
  • Systems integration expertise
  • UX design for human-AI collaboration
  • Domain knowledge in the problem space
  • Pragmatic approach to AI capabilities and limitations

It's not about impressive demos. It's about building tools that people actually use because they make their jobs materially better.

If your AI chatbot failed, it's probably because you built a demo, not a solution. The good news: you can learn from that. The next iteration can be bespoke, integrated, specific, and actually useful.

The companies that figure this out — that stop building generic AI assistants and start building deeply integrated, workflow-specific AI tools — will have a genuine competitive advantage. Not because they "use AI" but because they've made their teams faster, their processes more efficient, and their operations more effective.

That's the real opportunity. Not chatbots. Not "AI-powered search." Not generic assistants. Bespoke AI integration that solves real problems in real workflows for real people.

Building bespoke AI applications that solve specific problems is what I do. If you've tried generic AI tools and been disappointed, let's talk about what actually works. Get in touch to discuss your requirements.

AIDevelopment