Vector Databases Introduction

Vector databases are specialized data storage systems optimized for handling high-dimensional vector embeddings. They are widely used in semantic search, recommendation engines, anomaly detection, and natural language processing (NLP). 

Vector databases are especially efficient in agentic workflows. Unlike traditional databases that rely on structured or relational queries, vector databases are designed to store and retrieve data based on vector similarity. 

By storing embeddings, which are mathematical representations of your content, vector databases enable semantic search. This allows automated workflows, or “agents,” to handle user queries with minimal manual intervention. For example, if someone asks “How do I configure the Oracle NetSuite connector?”, the system can locate the most relevant information even if the exact phrase isn’t present in the documentation.

When paired with low-code workflows, vector databases make it possible to ingest large document collections, embed them for search, connect a chat interface such as Slack, and use AI models to generate context-aware responses.

Currently, Cyclr has the following vector database Connectors:

• Pinecone

Why use vector databases with Cyclr? #

Vector databases can be useful in integration workflows and automations in multiple ways. Key use cases include:

• Knowledge Agents: Respond to natural language queries using embedded company content.
• Semantic Search: Replace keyword-based search with meaning-based retrieval.
• Intelligent Classification: Organize and tag documents, images or any other data by similarity.
• Retrieval-Augmented Generation (RAG) Workflows: Use Cyclr workflows to retrieve relevant content from a vector database and pass it into a language model for more accurate responses.

How do they work? #

At a high level, vector databases support similarity search through four core stages:

• Ingestion: Content such as text, images, or other data types is converted into high-dimensional vectors using embedding models. These models translate human-readable input into a numerical format that encodes semantic relationships.
• Indexing: Once vectors are generated, in most cases, the vector database organizes them using Approximate Nearest Neighbor (ANN) indexing algorithms. Common examples include HNSW (Hierarchical Navigable Small World), FAISS (Facebook AI Similarity Search – an open-source library developed by Meta AI), and IVF (Inverted File indexes). These indexes allow for efficient and scalable retrieval across large datasets.
• Querying: When a user or system submits a query, it is also converted into a vector using the same embedding model used during ingestion. This ensures that both stored and incoming data exist in the same representational space.
• Retrieval: The database compares the query vector to those in the index using distance metrics such as cosine similarity or Euclidean distance. It returns the most similar vectors, along with any associated metadata, based on their proximity in vector space.

High-Level Workflow

The process of setting up an integration using vector databases generally follows these steps:

1. Ingestion: Input data is converted into embeddings.
2. Indexing: Embeddings are stored in a vector database.
3. Querying:
   • A user query is converted into a vector embedding using the same model.
   • The query vector is compared against stored vectors.
4. Retrieval: The top matches are passed into a language model, which generates a contextual response.

Agentic Workflows #

Agentic workflows combine data ingestion, vector search, and AI-driven responses into automated processes. This term is not a formal industry category, but it describes a growing design pattern for building intelligent, low-intervention workflows.

A typical query-and-response workflow proceeds as follows:

1. A user sends a query via Slack, Teams, or another trigger.
2. The workflow generates an embedding for the query.
3. The vector database retrieves the most relevant results.
4. The retrieved context is passed into a language model.
5. The response is delivered back through the original channel.
   

For example, a channel message in Slack could trigger the Cycle. The message would be embedded, searched against Pinecone for the top three matches, and the results compiled. A language model would then generate a short response, which Cyclr posts back into the Slack channel.

The same approach can extend to Microsoft Teams for internal knowledge sharing, or platforms like Zendesk and Intercom for customer support.

Watch: Building a RAG Knowledge Agent with Slack, Google Drive, and Pinecone (Video Walkthrough – Episode 1)