Plugin Development Guide

Introduction

This comprehensive guide walks you through extending Spry, a Markdown-based notebook system built on the unified ecosystem. Spry allows you to create powerful, executable documents by combining Markdown with custom processing logic.

What you’ll learn:

• How to create remark plugins that transform Markdown AST (Abstract Syntax Tree)
• How to build task directive inspectors for custom executable cell types
• How to integrate with Spry’s event system for reactive programming
• Best practices for type-safe, performant plugin development

Core Concepts

What is Spry?

Spry is a programmable notebook system that treats Markdown files as executable documents. It leverages:

• unified/remark — Industrial-strength Markdown processing
• Task Directives — Turn code blocks into executable tasks
• Type Safety — Full TypeScript support with runtime validation
• Event-Driven — React to execution events in real-time

Extension Points

Spry provides three main extension mechanisms:

1. Remark Plugins — Process and enrich Markdown during parsing
2. Task Directive Inspectors — Define how code cells become executable
3. Event Bus Listeners — React to lifecycle events (start, complete, error)

---

Part 1: Remark Plugin Development

Remark plugins transform the Markdown Abstract Syntax Tree (AST) as documents are processed. They can add metadata, validate structure, or prepare content for execution.

Basic Plugin Structure

Every remark plugin follows this pattern:

import { Plugin } from "unified";
import { Root } from "types/mdast";

interface MyPluginOptions {
  option1?: string;
  option2?: boolean;
}

export const myPlugin: Plugin<[MyPluginOptions?], Root> = (options = {}) => {
  return function transformer(tree: Root) {
    // Transform the tree
  };
};

export default myPlugin;

Key Points:

• Plugins are factory functions that return a transformer
• The transformer receives the AST tree and modifies it in place
• Options provide configuration flexibility
• The Root type represents the top-level Markdown document

Traversing the AST with visit

The unist-util-visit utility makes tree traversal simple:

import { visit } from "unist-util-visit";
import { Code, Root } from "types/mdast";

export const myPlugin: Plugin<[], Root> = () => {
  return (tree) => {
    visit(tree, "code", (node: Code) => {
      // Process each code block
      console.log(`Found code block: ${node.lang}`);
    });
  };
};

Common node types:

• "code" — Code blocks (fenced with backticks)
• "paragraph" — Text paragraphs
• "heading" — Headings (h1-h6)
• "list" — Ordered and unordered lists
• "link" — Hyperlinks

The visitor pattern lets you process specific node types without manually recursing through the tree.

Attaching Metadata to Nodes

Plugins often need to attach computed data to nodes for later use:

import { visit } from "unist-util-visit";

export const myPlugin: Plugin = () => {
  return (tree) => {
    visit(tree, "code", (node) => {
      // Initialize data object if needed
      node.data ??= {};

      // Attach custom metadata
      node.data.myCustomData = {
        processedAt: new Date().toISOString(),
        someComputation: computeSomething(node),
      };
    });
  };
};

Why attach data?

• Downstream plugins can read this metadata
• Task inspectors use it to determine executability
• Renderers can use it for custom formatting

---

Type-Safe Node Data

Type safety prevents runtime errors and improves developer experience. Spry provides utilities for attaching validated data to AST nodes.

Using nodeDataFactory

This creates a typed accessor for node data:

import { nodeDataFactory } from "./lib/remark/mdast/safe-data.ts";

// Define your data type
interface MyMetadata {
  processed: boolean;
  value: number;
}

// Create a typed data accessor
const MY_KEY = "myMetadata" as const;
export const myMetadataNDF = nodeDataFactory<typeof MY_KEY, MyMetadata>(MY_KEY);

// Use in plugin
export const myPlugin: Plugin = () => {
  return (tree) => {
    visit(tree, "code", (node) => {
      // Attach typed data
      myMetadataNDF.set(node, {
        processed: true,
        value: 42,
      });
    });
  };
};

// Check and read data elsewhere
if (myMetadataNDF.is(node)) {
  const data = node.data.myMetadata; // TypeScript knows the type
  console.log(data.value); // 42
}

Benefits:

• TypeScript autocompletion for data fields
• Type checking at compile time
• No need for type assertions

Using safeNodeDataFactory with Zod

For runtime validation, combine with Zod schemas:

import { z } from "@zod/zod";
import { safeNodeDataFactory } from "./lib/remark/mdast/safe-data.ts";

// Define schema
const myMetadataSchema = z.object({
  processed: z.boolean(),
  value: z.number().min(0),
});

type MyMetadata = z.infer<typeof myMetadataSchema>;

// Create validated data accessor
export const myMetadataSNDF = safeNodeDataFactory<"myMetadata", MyMetadata>(
  "myMetadata",
  myMetadataSchema,
  {
    onAttachSafeParseError: ({ node, error }) => {
      console.error("Validation failed:", error);
      return null; // Don't attach invalid data
    },
  }
);

When to use:

• Parsing metadata from node attributes
• Validating user-provided configuration
• Ensuring data integrity across plugin boundaries

---

Complete Plugin Example: Code Statistics

Let’s build a practical plugin that calculates statistics for code blocks:

lib/remark/plugin/node/code-stats.ts

import type { Code, Root } from "types/mdast";
import { Plugin } from "unified";
import { visit } from "unist-util-visit";
import { nodeDataFactory } from "../../mdast/safe-data.ts";

/** Statistics for a code block */
export interface CodeStats {
  lineCount: number;
  charCount: number;
  hasShebang: boolean;
  language: string;
}

export const CODE_STATS_KEY = "codeStats" as const;
export const codeStatsNDF = nodeDataFactory<typeof CODE_STATS_KEY, CodeStats>(
  CODE_STATS_KEY
);

export interface CodeStatsOptions {
  /** Callback when stats are computed */
  onStats?: (node: Code, stats: CodeStats) => void;
}

export const codeStats: Plugin<[CodeStatsOptions?], Root> = (options = {}) => {
  const { onStats } = options;

  return function transformer(tree: Root) {
    visit(tree, "code", (node: Code) => {
      const value = node.value || "";
      const lines = value.split("\n");

      const stats: CodeStats = {
        lineCount: lines.length,
        charCount: value.length,
        hasShebang: value.startsWith("#!"),
        language: node.lang || "unknown",
      };

      // Attach to node
      codeStatsNDF.set(node, stats);

      // Optional callback
      onStats?.(node, stats);
    });
  };
};

export default codeStats;

Using the Code Statistics Plugin

import { remark } from "remark";
import codeStats, { codeStatsNDF } from "./code-stats.ts";

const processor = remark().use(codeStats, {
  onStats: (node, stats) => {
    console.log(`${stats.language}: ${stats.lineCount} lines`);
  },
});

const tree = processor.parse(`
\`\`\`bash
#!/usr/bin/env bash
echo "hello"
\`\`\`
`);

processor.runSync(tree);

// Access stats on nodes
visit(tree, "code", (node) => {
  if (codeStatsNDF.is(node)) {
    console.log(node.data.codeStats.hasShebang); // true
  }
});

Plugin Features:

• Counts lines and characters
• Detects shebang lines for executability
• Tracks language for syntax highlighting
• Provides callback for real-time processing

---

Part 2: Task Directive Inspectors

Task Directive Inspectors determine which code blocks become executable tasks and how they should run.

Understanding the Inspector Interface

type TaskDirectiveInspector<Provenance, Frontmatter, CellAttrs, I> = (
  ctx: {
    cell: PlaybookCodeCell<Provenance, CellAttrs>;
    pb: Playbook<Provenance, Frontmatter, CellAttrs, I>;
    registerIssue: (message: string, error?: unknown) => void;
  }
) => TaskDirective | false;

Context provided:

• cell — The code cell being inspected
• pb — The entire playbook (notebook) for cross-referencing
• registerIssue — Report problems without throwing errors

Return values:

• TaskDirective — Instructions for executing this cell
• false — This inspector doesn’t handle this cell (try next inspector)

Creating a Custom Inspector

Here’s an inspector for a custom deno-task language:

lib/task/deno-task-inspector.ts

import { TaskDirectiveInspector } from "./cell.ts";
import { languageRegistry } from "../universal/code.ts";

// Define the language spec
const denoTaskLangSpec = {
  id: "deno-task",
  extensions: [".ts"],
  comment: { line: ["//"], block: [] },
};

export function denoTaskInspector<Provenance>(): TaskDirectiveInspector<
  Provenance
> {
  return ({ cell, registerIssue }) => {
    // Only handle deno-task language
    if (cell.language !== "deno-task") {
      return false;
    }

    // Require a task identity
    const pi = cell.parsedPI;
    if (!pi?.firstToken) {
      registerIssue("deno-task cells require a task name");
      return false;
    }

    // Return the task directive
    return {
      nature: "TASK",
      identity: pi.firstToken,
      language: denoTaskLangSpec,
      deps: pi.flags.dep
        ? Array.isArray(pi.flags.dep)
          ? pi.flags.dep
          : [pi.flags.dep]
        : undefined,
    };
  };
}

Key Concepts:

• nature: "TASK" — This cell should be executed
• identity — Unique task name (from first token after language)
• deps — Task dependencies (will wait for these to complete)
• language — Defines syntax and execution environment

Registering Inspectors

Inspectors are checked in order until one returns a directive:

import { TaskDirectives, partialsInspector, spawnableTDI } from "./lib/task/mod.ts";
import { denoTaskInspector } from "./deno-task-inspector.ts";
import { fbPartialsCollection } from "./lib/markdown/notebook/mod.ts";

const partials = fbPartialsCollection();
const td = new TaskDirectives(partials, [
  partialsInspector(),      // Handle PARTIAL blocks first
  denoTaskInspector(),      // Our custom inspector
  spawnableTDI(),           // Default shell tasks
]);

Order matters:

• Earlier inspectors have priority
• Return false to pass to the next inspector
• The first inspector returning a directive wins

---

Part 3: Event Bus Integration

Spry uses an event bus for reactive programming. Plugins can listen to task execution events.

Task Execution Events

import { eventBus } from "./lib/universal/event-bus.ts";
import { TaskExecEventMap } from "./lib/universal/task.ts";

// Create or get the task bus
const tasksBus = eventBus<TaskExecEventMap>();

tasksBus.on("task:start", ({ task, ctx }) => {
  console.log(`[${ctx.runId}] Starting: ${task.taskId()}`);
});

tasksBus.on("task:complete", ({ task, result }) => {
  console.log(`Completed: ${task.taskId()}`);
  console.log(`Exit code: ${result.exitCode}`);
});

tasksBus.on("task:error", ({ task, error }) => {
  console.error(`Failed: ${task.taskId()}`, error);
});

Shell Execution Events

For more granular control, listen to shell events:

import { ShellBusEvents } from "./lib/universal/shell.ts";

const shellBus = eventBus<ShellBusEvents>();

shellBus.on("spawn", ({ command }) => {
  console.log(`Spawning: ${command}`);
});

shellBus.on("stdout", ({ data }) => {
  process.stdout.write(data);
});

shellBus.on("stderr", ({ data }) => {
  process.stderr.write(data);
});

Use cases:

• Real-time output streaming
• Progress tracking
• Performance monitoring
• Debugging and logging

---

Advanced Example: Diagram Plugin

Let’s build a complete plugin system for diagrams, including both a remark plugin and task inspector.

The Remark Plugin

lib/remark/plugin/node/diagram.ts

import type { Code, Root } from "types/mdast";
import { Plugin } from "unified";
import { visit } from "unist-util-visit";
import { z } from "@zod/zod";
import { safeNodeDataFactory } from "../../mdast/safe-data.ts";

// Schema for diagram metadata
const diagramSchema = z.object({
  type: z.enum(["mermaid", "plantuml", "graphviz"]),
  title: z.string().optional(),
  width: z.number().optional(),
  height: z.number().optional(),
});

type DiagramMetadata = z.infer<typeof diagramSchema>;

export const DIAGRAM_KEY = "diagram" as const;
export const diagramSNDF = safeNodeDataFactory<typeof DIAGRAM_KEY, DiagramMetadata>(
  DIAGRAM_KEY,
  diagramSchema,
  {
    onAttachSafeParseError: ({ error }) => {
      console.warn("Invalid diagram metadata:", error);
      return null;
    },
  }
);

export interface DiagramPluginOptions {
  /** Default diagram type if not specified */
  defaultType?: DiagramMetadata["type"];
  /** Callback for each diagram found */
  onDiagram?: (node: Code, meta: DiagramMetadata) => void;
}

export const diagramPlugin: Plugin<[DiagramPluginOptions?], Root> = (
  options = {}
) => {
  const { defaultType = "mermaid", onDiagram } = options;

  const diagramLanguages = ["mermaid", "plantuml", "graphviz", "diagram"];

  return function transformer(tree: Root) {
    visit(tree, "code", (node: Code) => {
      if (!node.lang || !diagramLanguages.includes(node.lang)) {
        return;
      }

      // Parse metadata from node.meta if present
      let meta: Partial<DiagramMetadata> = {};
      if (node.meta) {
        try {
          // Simple key=value parsing
          const pairs = node.meta.split(/\s+/).filter(Boolean);
          for (const pair of pairs) {
            const [key, value] = pair.split("=");
            if (key && value) {
              if (key === "width" || key === "height") {
                meta[key] = parseInt(value, 10);
              } else if (key === "title") {
                meta.title = value;
              }
            }
          }
        } catch {
          // Ignore parse errors
        }
      }

      // Determine type
      const type = node.lang === "diagram"
        ? (meta as any).type || defaultType
        : node.lang as DiagramMetadata["type"];

      const diagramMeta: DiagramMetadata = {
        type,
        ...meta,
      };

      // Attach to node (with validation)
      diagramSNDF.attach(node, diagramMeta);

      // Callback
      if (diagramSNDF.is(node)) {
        onDiagram?.(node, node.data.diagram);
      }
    });
  };
};

export default diagramPlugin;

The Task Inspector

lib/task/diagram-inspector.ts

import { TaskDirectiveInspector } from "./cell.ts";

export function diagramInspector<Provenance>(): TaskDirectiveInspector<
  Provenance
> {
  return ({ cell }) => {
    const diagramLangs = ["mermaid", "plantuml", "graphviz"];

    if (!diagramLangs.includes(cell.language)) {
      return false;
    }

    const pi = cell.parsedPI;
    if (!pi?.firstToken) {
      return false; // No identity, skip
    }

    return {
      nature: "CONTENT",
      identity: pi.firstToken,
      language: {
        id: cell.language,
        extensions: [`.${cell.language}`],
        comment: { line: [], block: [] },
      },
      content: {
        diagramSource: cell.source,
      },
    };
  };
}

Complete workflow:

1. Remark plugin attaches metadata to diagram code blocks
2. Task inspector identifies them as content tasks
3. Executor renders diagrams using the appropriate tool
4. Event bus notifies listeners of rendering progress

---

Testing Plugins

Thorough testing ensures plugins work correctly and don’t break under edge cases.

Unit Test Structure

lib/remark/plugin/node/code-stats_test.ts

import { assertEquals, assert } from "@std/assert";
import { remark } from "remark";
import codeStats, { codeStatsNDF } from "./code-stats.ts";
import { visit } from "unist-util-visit";

Deno.test("codeStats plugin", async (t) => {
  const processor = remark().use(codeStats);

  await t.step("calculates line count", () => {
    const md = "```bash\nline1\nline2\nline3\n```";
    const tree = processor.parse(md);
    processor.runSync(tree);

    visit(tree, "code", (node) => {
      assert(codeStatsNDF.is(node));
      assertEquals(node.data.codeStats.lineCount, 4);
    });
  });

  await t.step("detects shebang", () => {
    const md = "```bash\n#!/usr/bin/env bash\necho hi\n```";
    const tree = processor.parse(md);
    processor.runSync(tree);

    visit(tree, "code", (node) => {
      assert(codeStatsNDF.is(node));
      assertEquals(node.data.codeStats.hasShebang, true);
    });
  });

  await t.step("tracks language", () => {
    const md = "```python\nprint('hi')\n```";
    const tree = processor.parse(md);
    processor.runSync(tree);

    visit(tree, "code", (node) => {
      assert(codeStatsNDF.is(node));
      assertEquals(node.data.codeStats.language, "python");
    });
  });
});

Running Tests

# Run all tests
deno test --parallel --allow-all

# Run specific test file
deno test lib/remark/plugin/node/code-stats_test.ts --allow-all

# Watch mode for development
deno test --watch --allow-all

Testing best practices:

• Test edge cases (empty code blocks, missing languages)
• Use descriptive test names
• Group related tests with t.step
• Test both success and failure paths

---

Best Practices

Plugin Design Principles

1. Single Responsibility — Each plugin should do one thing well

   • ✅ Good: codeStats only calculates statistics
   • ❌ Bad: Plugin that calculates stats AND executes code

2. Idempotent — Running a plugin multiple times should be safe

   • Check if data already exists before recomputing
   • Don’t accumulate state across runs

3. Type-Safe Data — Always use nodeDataFactory or safeNodeDataFactory

   • Prevents typos in property names
   • Enables autocompletion
   • Catches errors at compile time

4. Options Pattern — Accept configuration via options object

   • Provides sensible defaults
   • Makes plugins flexible without breaking changes
   • Documents available configuration

5. No Side Effects — Plugins should be pure transformers

   • Don’t write files directly
   • Don’t make network requests
   • Emit data for later processing

Error Handling

1. Validation Errors — Use registerIssue for recoverable errors

   if (!isValid(cell)) {
     registerIssue("Cell is missing required field");
     return false;
   }

2. Schema Errors — Use safeNodeDataFactory with error handlers

   onAttachSafeParseError: ({ error }) => {
     console.warn("Invalid data:", error);
     return null; // Don't attach
   }

3. Fatal Errors — Only throw for truly unrecoverable situations

   if (systemResourcesExhausted()) {
     throw new Error("Cannot continue");
   }

Performance Optimization

1. Early Return — Check node type early in visitors

   visit(tree, "code", (node) => {
     if (node.lang !== "javascript") return; // Fast rejection
     // Expensive processing...
   });

2. Avoid Reprocessing — Check if data already exists

   if (myDataNDF.is(node)) {
     return; // Already processed
   }

3. Batch Operations — Collect during visit, process after

   const nodes = [];
   visit(tree, "code", (node) => nodes.push(node));
   // Now process all at once with optimized algorithm
   processBatch(nodes);

Documentation Standards

1. JSDoc Comments — Document all public exports

   /**
    * Calculates statistics for code blocks.
    *
    * @param options - Plugin configuration
    * @returns A remark transformer
    */
   export const codeStats: Plugin = (options) => {
     // ...
   };

2. Examples — Include usage examples in comments

   /**
    * @example
    * ```typescript
    * remark().use(codeStats, {
    *   onStats: (node, stats) => console.log(stats)
    * });
    * ```
    */

3. Type Exports — Export TypeScript interfaces for consumers

   export interface CodeStats { /* ... */ }
   export interface CodeStatsOptions { /* ... */ }

---

Next Steps

Now that you understand plugin development, explore these resources:

• Architecture Overview — Understand how Spry’s internals work together
• API Reference — Detailed documentation of all modules and functions
• Examples Gallery — See real-world plugins in action
• Community Plugins — Browse plugins created by other developers

Quick Start Checklist

• Set up a Deno development environment
• Clone the Spry repository
• Read through existing plugins in lib/remark/plugin/
• Create a simple plugin following the code-stats example
• Write tests for your plugin
• Add JSDoc documentation
• Submit a pull request or publish as a separate package

Getting Help

• GitHub Issues — Report bugs or request features
• Discussions — Ask questions and share ideas
• Discord — Real-time chat with the community
• Documentation — Keep this guide handy as a reference

Happy plugin development! 🚀