Document Ingestion

MemoryLayer can ingest documents, extract text content, and automatically create memories from the extracted content. Supported formats include PDF, DOCX, Markdown, HTML, PPTX, and plain text.

Document ingestion ships in the open-source server. Pair it with the optional [documents]/[pdf]/[office] extras (pip install "memorylayer-server[documents]") to pull in the file-format parsers you need. ColPali visual indexing is also available in OSS via the memorylayer-embed-server peer’s [colpali] extra (or [gpu] for the full bundle). Cross-cluster orchestration and the SaaS dashboards remain enterprise add-ons.

Upload a Document

Upload a document using multipart/form-data. The server automatically detects the file type, extracts text content, and creates memories — no configuration required.

curl -X POST "$MEMORYLAYER_URL/v1/documents" \
  -H "Authorization: Bearer $API_KEY" \
  -F "file=@report.pdf"

You can optionally specify a context and importance level:

curl -X POST "$MEMORYLAYER_URL/v1/documents" \
  -H "Authorization: Bearer $API_KEY" \
  -F "file=@report.pdf" \
  -F "target_context_id=project-docs" \
  -F "importance=0.7"

Parameter	Default	Description
`file`	(required)	The document file
`target_context_id`	`_default`	Memory context for extracted memories
`importance`	`0.5`	Default importance for extracted memories (0.0–1.0)

The response includes both the document record and the ingestion job:

{
  "document": {
    "id": "doc_abc123",
    "workspace_id": "ws_001",
    "filename": "report.pdf",
    "document_type": "pdf",
    "status": "pending",
    "size_bytes": 204800,
    "page_count": 0,
    "chunk_count": 0,
    "created_at": "2026-04-01T12:00:00Z"
  },
  "job": {
    "id": "job_xyz789",
    "status": "queued",
    "progress_percent": 0,
    "created_at": "2026-04-01T12:00:00Z"
  }
}

Python SDK

from memorylayer import MemoryLayerClient

client = MemoryLayerClient(base_url=MEMORYLAYER_URL, api_key=API_KEY)

with open("report.pdf", "rb") as f:
    result = client.documents.upload(
        file=f,
        target_context_id="project-docs",
        importance=0.7,
    )

print(f"Document: {result.document.id}, Job: {result.job.id}")

The server automatically selects the best chunking strategy based on the document type (page-based for PDFs/PPTX, semantic for prose documents).

Job Tracking

Check job status

curl "$MEMORYLAYER_URL/v1/documents/jobs/$JOB_ID" \
  -H "Authorization: Bearer $API_KEY"

Response fields: status (queued, running, completed, failed, cancelled), progress_percent, documents_processed, total_memories_created, and errors.

List all jobs

curl "$MEMORYLAYER_URL/v1/documents/jobs?status=running&limit=20" \
  -H "Authorization: Bearer $API_KEY"

Cancel a job

curl -X POST "$MEMORYLAYER_URL/v1/documents/jobs/$JOB_ID/cancel" \
  -H "Authorization: Bearer $API_KEY"

Retrieve Extracted Memories

After processing completes, fetch the memories created from a document:

curl "$MEMORYLAYER_URL/v1/documents/$DOC_ID/memories" \
  -H "Authorization: Bearer $API_KEY"

Each memory includes its id, content, type, importance, tags, and created_at.

Search Document Pages

Search ingested document pages using ColPali MaxSim (ColBERT-style late interaction) visual similarity:

curl -X POST "$MEMORYLAYER_URL/v1/documents/search" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "quarterly revenue chart",
    "limit": 5,
    "doc_ids": ["doc_abc123"]
  }'

Field	Type	Required	Description
`query`	string	yes	Natural-language search query
`limit`	integer	no (default `10`, max `100`)	Maximum results to return
`doc_ids`	string[]	no	Restrict the search to a specific list of document IDs

Each result entry is a DocumentPageResponse with a relevance_score (sum of per-query-token max cosine similarities) on top of the usual page fields (id, document_id, page_no, transcript, metadata, etc.).

Requirements:

A running memorylayer-embed-server peer with [colpali] (or [gpu]) installed — the core server calls /v1/embeddings/multi to embed the query as a multi-vector.
An ingestion path that populated the multivector column on document_pages. ColPali multivectors are produced for image-bearing documents (PDFs, image-only files) when the embed server is configured to do so.

OSS vs Enterprise performance: OSS and Enterprise expose the same POST /v1/documents/search endpoint with the same request/response shape. The OSS SQLite backend scores candidate pages in Python (after loading their multivectors from disk), which is correct and works well for workspaces with up to a few thousand multi-vector pages. The Enterprise PostgreSQL backend pushes MaxSim scoring into the database via a pgvector lateral subquery, which holds up under much larger corpora and concurrent query load. The application-facing API and behaviour are identical; only the storage-backend implementation differs.

List and Inspect Documents

# List documents with optional filters
curl "$MEMORYLAYER_URL/v1/documents?status=completed&document_type=pdf&limit=50" \
  -H "Authorization: Bearer $API_KEY"

# Get a single document
curl "$MEMORYLAYER_URL/v1/documents/$DOC_ID" \
  -H "Authorization: Bearer $API_KEY"

Reprocess a Document

Re-extract memories from a document without re-uploading the file:

curl -X POST "$MEMORYLAYER_URL/v1/documents/$DOC_ID/reprocess" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"importance": 0.8}'

This creates a new ingestion job and regenerates memories from the stored document.

Delete a Document

# Delete document only
curl -X DELETE "$MEMORYLAYER_URL/v1/documents/$DOC_ID" \
  -H "Authorization: Bearer $API_KEY"

# Delete document and its extracted memories
curl -X DELETE "$MEMORYLAYER_URL/v1/documents/$DOC_ID?delete_memories=true" \
  -H "Authorization: Bearer $API_KEY"

Document Statuses

Status	Meaning
`pending`	Uploaded, waiting for processing
`processing`	Extraction in progress
`completed`	All chunks processed and memories created
`failed`	Processing encountered an error
`partial`	Some chunks succeeded, others failed