Sources

Sources define where versioned files come from. A pipeline definition lists sources in inputs, and the executor resolves those sources before any route runs.

Source definitions are backend-driven

A source definition wraps a backend:

import { definePipelineSource } from "@ucdjs/pipeline-core";

const mySource = definePipelineSource({
  id: "custom-source",
  backend: {
    async listFiles(version) {
      return [];
    },
    async readFile(file) {
      return "";
    },
  },
});

A backend can implement:

• listFiles(version) to enumerate available files
• readFile(file) to read content
• readFileStream(file, options) when streaming is needed
• getMetadata(file) when metadata is available

Source definitions can also add:

• includes to keep only matching files
• excludes to drop matching files
• kind: "pipeline-output" for synthetic sources built from another pipeline's outputs

Use helpers when possible

createMemorySource(...)

This is the best authoring helper for examples, tests, and playground pipelines:

import { createMemorySource } from "@ucdjs/pipeline-core/sources";

const colorsSource = createMemorySource({
  id: "colors",
  files: {
    "1.0.0": [{
      path: "data/colors.txt",
      content: "FF0000; Red",
    }],
  },
});

createHttpSource(...) and createUnicodeOrgSource(...)

These helpers create HTTP-backed sources:

import { createUnicodeOrgSource } from "@ucdjs/pipeline-core/sources";

const unicodeOrgSource = createUnicodeOrgSource();

Use them when HTTP transport is the right abstraction for your environment. They are lower-level than the in-memory helper, and they are best suited to setups where file enumeration is controlled intentionally.

Filter files at the source boundary

Source-level includes and excludes run before route matching. That makes them useful for broad source shaping, such as limiting a large source to one directory or excluding generated files.

import { byGlob, definePipelineSource } from "@ucdjs/pipeline-core";

const source = definePipelineSource({
  id: "custom-source",
  backend,
  includes: byGlob("**/*.txt"),
});

Consume outputs from another pipeline

pipelineOutputSource(...) creates a synthetic source backed by outputs from an upstream pipeline:

import { pipelineOutputSource } from "@ucdjs/pipeline-core";

const publishedColorsInput = pipelineOutputSource({
  pipelineId: "traced-outputs",
  outputId: "filesystem-archive",
});

Use this when one pipeline should read a published output from another pipeline in the same executor batch.