Skip to content
memorylayer.ai Documentation

Enterprise Data Connectors

The enterprise connector pack extends the open-source Data Providers API with cloud-storage, source-control, and team-chat sources. The REST surface is identical — create, list, update, delete, and trigger sync calls work the same. Only the available provider_type values change.

Connector Types

TypeDescriptionTypical Connection Args
local_fsFilesystem watch backed by the connectors VFS catalog (watches for changes, deduplicates blobs, supports include/exclude globs)root_path, watch, include_globs, exclude_globs
s3Amazon S3 bucketbucket, prefix, region, endpoint_url
githubGitHub repository: code, issues, PRs, releases, wikisrepo, branch, include_issues, include_prs, token
gdriveGoogle Drive folderfolder_id, oauth_refresh_token
dropboxDropbox folderfolder_path, oauth_refresh_token
slackSlack workspace — channels and DMsworkspace_id, channels, include_dms, oauth_token
teamsMicrosoft Teams — channels and chatstenant_id, team_id, channels, oauth_token
discordDiscord guild text channelsguild_id, channels, bot_token
web_scraperURL list or sitemap crawler with optional link-followingurls, depth, follow_links, same_domain_only
manual_uploadCatch-all bucket for client-uploaded blobsmime_filters, max_size_bytes

Connection-arg shapes are documented per-connector inside the memorylayer-data-connectors package. The OAuth-bearing connectors (gdrive, dropbox, slack, teams) use refresh tokens stored in encrypted_args rather than long-lived access tokens.

Example: GitHub Repository

Terminal window
curl -X POST "$MEMORYLAYER_URL/v1/data-providers" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "scitrera/memorylayer",
    "provider_type": "github",
    "connection_args": {
      "repo": "scitrera/memorylayer",
      "branch": "main",
      "include_issues": true,
      "include_prs": true
    },
    "encrypted_args": {
      "token": "ghp_..."
    },
    "schedule": "0 */6 * * *"
  }'

Example: Slack Workspace

Terminal window
curl -X POST "$MEMORYLAYER_URL/v1/data-providers" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Eng Slack",
    "provider_type": "slack",
    "connection_args": {
      "workspace_id": "T01234567",
      "channels": ["eng-platform", "eng-releases"],
      "include_dms": false
    },
    "encrypted_args": {
      "oauth_token": "xoxp-..."
    },
    "schedule": "*/30 * * * *"
  }'

Example: S3 Bucket

Terminal window
curl -X POST "$MEMORYLAYER_URL/v1/data-providers" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Research Papers S3",
    "provider_type": "s3",
    "connection_args": {
      "bucket": "research-papers",
      "prefix": "published/",
      "region": "us-east-1"
    },
    "encrypted_args": {
      "aws_access_key_id": "AKIA...",
      "aws_secret_access_key": "..."
    },
    "schedule": "0 2 * * *"
  }'

Deployment Model

The connectors run inside a separate data_connectors server (typically reachable via Aether). The core MemoryLayer server proxies create / list / sync requests to it; sync workers stream content through a VFS catalog and into the documents pipeline. Operators do not need to expose the connectors process directly to end users — only the core server’s /v1/data-providers endpoint is consumed by clients.

Contact memorylayer.ai for access to the enterprise connector pack and deployment templates.