Embedding worker / DLQ

When products don’t become searchable in the AI — when the embedding bar is stuck at 0% or partially complete — the issue is almost always in one of four places: the BullMQ worker process, Redis, the AI provider key, or the dead-letter queue. Walk this diagnostic from cheapest check to most involved.

Step-by-step

1. 1

   Is the worker running?

   Open Settings → Status → Worker. The heartbeat should be within the last 60 seconds. If it’s stale:

   • Self-hosted: the worker process (npm run worker) isn’t running, or has crashed. Check logs.
   • SaaS: very rare. Check the public status page; we’ll have an ongoing incident if our workers are degraded.
2. 2

   Is Redis reachable?

   The same Status panel shows Redis latency. If it’s elevated or erroring:

   • Self-hosted: check the Redis instance is reachable from the worker container. The local Docker compose uses port 6380 (not 6379) — a common mistake.
   • SaaS: confirm at /status.
3. 3

   Is an AI provider configured?

   Under Admin → Providers at least one provider needs to have embeddings in its capability list and a healthy status. Embeddings are pinned to 1,536 dimensions; supported provider/model combos are:

   • OpenAI — text-embedding-3-small (default). Cheap and fast.
   • Azure OpenAI — same model via an Azure deployment.

   Anthropic and Gemini are configured for chat but not embeddings.
4. 4

   Is the API key valid?

   Click Test connection on the provider row. If the test fails:

   • 401 — wrong key, or the key was rotated.
   • 429 quota — OpenAI org quota hit, or hourly token cap reached. Raise quota or fail over to Azure.
   • 404 model_not_found — your Azure deployment name is wrong.
5. 5

   Are jobs piling up?

   Open Settings → Queues. Look at the embedding and document-embedding queues. A backlog of waiting jobs is normal during initial sync, but if the count is rising without active workers, the worker isn’t consuming. Restart the worker.
6. 6

   Is the DLQ populated?

   When a job fails 3 times with backoff (5 attempts for some critical queues), it lands in the dead-letter queue. Check Settings → Queues → Dead-letter.

   • Click on a dead job to see the last error.
   • Common causes: malformed product (description longer than 8k tokens — extremely rare), transient provider error during an incident.
   • Click Replay to re-enqueue. Bulk replay available with a checkbox + “Replay selected”.

Manual replay (advanced)

For automated replay, the merchant API exposes a DLQ replay endpoint:

curl -X POST https://api.zubbyai.com/api/v1/merchant/{storeId}/queues/embedding/dlq/replay \
  -H "Authorization: Bearer sk_live_..." \
  -H "X-Store-ID: {storeId}" \
  -d '{ "max": 500 }'

# Response: { "replayed": 487, "skipped": 13 }

Re-embed everything

If you’ve made bulk changes to product copy and want every product re-embedded, click Re-embed all under Catalog → Sync. The worker streams through the catalog; each product is enqueued once. Re-embedding is idempotent — running it twice doesn’t double-charge.

Telemetry signals to trust

• queues.embedding.depth — Prometheus metric exposed by the worker. A flat-line above zero with no consumers is the smoking gun.
• queues.embedding.failures_total — failures per minute. A spike correlates with provider incidents.
• queues.dlq.depth — total dead-letter depth. Anything above 0 is worth a glance.

Adjacent

• Troubleshooting overview
• How the AI works
• Security & privacy