Skip to content
memorylayer.ai Documentation

Data Providers

Data providers connect MemoryLayer to external data sources for automated document ingestion. Configure a provider once, set a sync schedule, and MemoryLayer pulls new and updated content automatically into the documents pipeline.

This page covers the open-source /v1/data-providers REST API and the OSS connector set. For the full enterprise connector pack (S3, GitHub, Google Drive, Dropbox, Slack, Teams, Discord, web scraper, manual upload, and the VFS-backed local_fs watcher), see Enterprise Data Connectors.

The REST API surface is identical in OSS and Enterprise. The only thing the enterprise package adds is more provider_type values backed by the memorylayer-data-connectors peer service.

OSS Connector Set

The open-source server ships with a single connector type:

TypeDescriptionTypical Connection Args
localLocal filesystem path on the server hostroot_path, glob, recursive

local is the default provider_type value if you omit it. A _default local provider is auto-created in each workspace on first access, so listing providers in a fresh workspace will always show at least one entry.

OSS scope note: in the OSS build, the local provider is a registry / metadata surface only — the /v1/data-providers/{provider_id}/sync endpoint succeeds but is implemented as a no-op (returns an empty document list) and the schedule field is stored but not honored by any built-in scheduler. The endpoints exist so OSS clients can persist provider records and stay forward-compatible with a Forge / connector-pack deployment that does the actual sync work. Use OSS data-providers for inventory and configuration; use the enterprise connector pack for actual scheduled ingestion.

Create a Local Filesystem Provider

Terminal window
curl -X POST "$MEMORYLAYER_URL/v1/data-providers" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Engineering Docs (local)",
    "provider_type": "local",
    "description": "Local docs directory mounted into the server",
    "enabled": true,
    "connection_args": {
      "root_path": "/data/docs/engineering",
      "glob": "**/*.md",
      "recursive": true
    },
    "schedule": "0 2 * * *",
    "metadata": {"team": "engineering"}
  }'

Fields:

FieldRequiredDescription
nameYesProvider name (must not be empty)
provider_typeNoConnector type (defaults to local in OSS)
descriptionNoHuman-readable description
enabledNoWhether the provider is active (default true)
connection_argsNoNon-sensitive connection parameters
encrypted_argsNoSensitive credentials (stored encrypted, never returned in API responses)
scheduleNoCron expression for automatic sync
metadataNoArbitrary key-value metadata

List Providers

Terminal window
curl "$MEMORYLAYER_URL/v1/data-providers?limit=50&offset=0" \
  -H "Authorization: Bearer $API_KEY"

Get a Provider

Terminal window
curl "$MEMORYLAYER_URL/v1/data-providers/$PROVIDER_ID" \
  -H "Authorization: Bearer $API_KEY"

The response never includes encrypted_args — only non-sensitive connection_args are returned.

Update a Provider

Only provided fields are updated; others remain unchanged.

Terminal window
curl -X PUT "$MEMORYLAYER_URL/v1/data-providers/$PROVIDER_ID" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "schedule": "0 */6 * * *",
    "connection_args": {
      "root_path": "/data/docs/engineering",
      "glob": "**/*.{md,txt}",
      "recursive": true
    }
  }'

Delete a Provider

Terminal window
curl -X DELETE "$MEMORYLAYER_URL/v1/data-providers/$PROVIDER_ID" \
  -H "Authorization: Bearer $API_KEY"

Deleting a provider does not delete previously ingested documents or memories.

Sync Scheduling

The schedule field accepts standard cron expressions:

ExpressionMeaning
0 2 * * *Daily at 2:00 AM
0 */6 * * *Every 6 hours
0 0 * * 1Weekly on Monday at midnight
*/30 * * * *Every 30 minutes

Set schedule to null to disable automatic sync. The last_sync_at field on the provider record shows the timestamp of the most recent successful sync.

You can also trigger a one-off sync at any time:

Terminal window
curl -X POST "$MEMORYLAYER_URL/v1/data-providers/$PROVIDER_ID/sync" \
  -H "Authorization: Bearer $API_KEY"

OSS behaviour: in the open-source server the local connector’s sync() is a no-op — the endpoint returns success with an empty synced_documents list and the schedule cron is stored but never fired. The enterprise connector pack and Forge are what turn these calls into real ingestion runs. The patterns above are still the right way to configure providers in OSS so the same records work unchanged once a sync-capable deployment is in place.