Architecture

Stoneturner Components

React Client

UI to monitor syncs, view markdown, and authenticate to data sources

  <Route element={<Layout />}>
    <Route path="/" element={<KnowledgeBasePage />} />
    <Route path="knowledge" element={<KnowledgeBasePage />} />
    <Route path="knowledge/config/:integration" element={<KnowledgeBasePage />} />
    <Route path="knowledge/data/:integration" element={<IntegrationDataPage />} />
    <Route path="monitoring" element={<SyncMonitoringPage />} />
  </Route>

MCP Server

Tools for agents to search all of your context efficiently

export const tools: McpTool[] = [
  {
    name: "semantic_search",
    description:
      `Semantic search across indexed call content,
      extracted key points, and questions answered.
      Returns the most relevant artifacts ranked by similarity,
     ...`

Integrations

Integrated data sources to sync

export const gongIntegration: Integration = {
  config: gongConfig,
  sync: async(db: SqliteDb) => await syncGongPipeline(false, db),
  syncUpdates: async(db: SqliteDb) => await syncGongPipeline(true, db),
  deleteSync: async(db: SqliteDb) => {
    await deleteSyncTasksByIntegration("Gong", db);
    await deleteMdArtifactsByIntegration("Gong", db);
    await deleteEmbeddingByIntegration("Gong", db);
    await deleteAllGongData(db);
  }
}

Core Services

Turso DB storage, vector embeddings, cron schedulers, and rate limiting

// Three midnight UTC crons in src/index.ts
Bun.cron("0 0 * * *", () => syncNewCron(db));      // scheduled incremental syncs
Bun.cron("0 0 * * *", () => retryFailedTasks(db));  // failed step retries
Bun.cron("0 0 * * *", () => deleteStaleTasks());   // 14-day task cleanup

Project structure

src/
  core/
    db/              # Database connection, queries, and schemas (relational + vector)
    handlers/        # HTTP + MCP request handlers
    middleware/      # CORS middleware (note: directory is "middlware")
    models/          # Shared type definitions
    services/        # Embedding, vector indexing, MCP server, tools
  integrations/
    config-registry.ts       # Frontend UI config for all integrations
    integration-registry.ts  # Sync dispatch for all integrations
    step-registry.ts         # Step lookup for retrying failed sync tasks
    gong/                     # Gong integration
      config.ts           # IntegrationConfig definition
      integration.ts      # Integration object (sync pipeline, delete)
      db/                 # Gong-specific schemas and queries
      models/             # Gong API response types
      sync-steps/         # Individual sync pipeline steps
    .../
  client/               # React SPA (monitoring + credential config)
  index.ts              # Bun.serve() entry point with all routes

Design Decisions

Turso’s SQLite Rewrite

Stoneturner uses Turso for local database storage and retrieval. With the embeddable advantages of SQLite, Turso extends SQLite by adding concurrent writes and vector search.

Rather than run a separate vector database, Turso keeps the database layer lightweight, local, and unified to search semantically or with SQL.

Background jobs

CRON-based jobs to enabled schedule syncs and failed steps when downstream APIs fail or get rate-limited.

Job	Purpose
Scheduled syncs	Allow users to schedule sync updates on a schedule cadence (i.e. every day, week, or month)
Retries	Re-invokes failed sync steps (max 3 retries)
Cleanup	Deletes `syncTask` rows older than 14 days

For full details on scheduling, retries, cleanup, and FAQs, see Sync scheduling & retries.

Vercel AI Gateway for Embedding Models and LLM Extraction

Many “LLM Gateway” products exist. Vercel supports both embedding models use to process text to index to Turso, and LLMs to run a quick task to extract insights from synced data. I borrowed this idea from enzyme.garden, where semantic data should have a complilation step to prep semantic data for search. When vector searching, matching questions and insights provide better search quality over matching content alone. Stoneturner also extract entities (like tags) for an agent to filter and follow connections like a graph database.

Bottleneck for Rate Limit Control

Bottleneck is a lightweight javascript library to schedule jobs and “bottleneck” function calls. This is extremely useful for respecting external API rate limits. Syncs call 3rd-party APIs over and over. Bottleneck lets us control concurrent requests and backoffs.

import Bottleneck from "bottleneck";

export const aiGatewayBottleneck = new Bottleneck({
  maxConcurrent: 5,
  minTime: 200
});

Unified Integration Interface

Integrating with 3rd-party APIs are notoriously cumbersome. We keep it simple by boiling down 3rd-party syncs to their primitives, providing a recipe for building reliable 3rd-party syncs.

export type Integration = {
  config: IntegrationConfig,
  sync: (db: SqliteDb) => Promise<void> | void,
  syncUpdates: (db: SqliteDb) => Promise<void> | void,
  deleteSync: (db: SqliteDb) => Promise<void> | void,
  handleRedirect?: (req: BunRequest, db: SqliteDb) => Promise<Response> | Response,
  refreshAccessTokens?: (db: SqliteDb) => Promise<void> | void,
}

This process is so repeatable, you can extend Stoneturner in minutes with a coding agent, 3rd-party docs, and the Stoneturner Skill.

Let’s set up Stoneturner locally and build your own integration sync in just a few steps!

Getting Started

For Builders

Reference

Stoneturner Components

React Client

MCP Server

Integrations

Core Services

Project structure

Design Decisions

Turso’s SQLite Rewrite

Background jobs

Vercel AI Gateway for Embedding Models and LLM Extraction

Bottleneck for Rate Limit Control

Unified Integration Interface

​Stoneturner Components

React Client

MCP Server

Integrations

Core Services

​Project structure

​Design Decisions

​Turso’s SQLite Rewrite

​Background jobs

​Vercel AI Gateway for Embedding Models and LLM Extraction

​Bottleneck for Rate Limit Control

​Unified Integration Interface

Stoneturner Components

Project structure

Design Decisions

Turso’s SQLite Rewrite

Background jobs

Vercel AI Gateway for Embedding Models and LLM Extraction

Bottleneck for Rate Limit Control

Unified Integration Interface