Introduction to Vector Databases
Submitted By: Submitted To:
Malik Saad Ahmed Mr. Syed Saad
1
Introduction to Vector Databases
1 Introduction
Vector databases are a type of NoSQL databases which are used to store vector em-
beddings of text. They are primarily used in Natural language processing to store vector
representations of text in a database for various applications such as
• Retrieval Augmented Generation
• Semantic Search
• Similarity Search Applications
2 Examples of Vector Databases
There are various vector databases which are widely used in real world such as
• Pinecone
• Qdrant
• Weaviate
2.1 Pinecone
Pinecone is a proprietary, cloud-native vector database widely used to store and search
vector embeddings. Unlike traditional databases, Pinecone is specifically designed for simila-
rity search and recommendation systems, providing a highly efficient and scalable solution for
handling high-dimensional vector data.
Pinecone offers a user-friendly API that simplifies integration into applications. It sup-
ports multiple programming languages, including Python, JavaScript, and Node.js. What
differentiates Pinecone from other vector databases is that it is fully managed—developers
don’t need to worry about infrastructure, scaling, or maintenance. The Pinecone service is
preconfigured, adding helpful abstractions that make development faster and more reliable.
2.2 Qdrant
It is another open source cloud hosted database built in Rust. Qdrant is highly scalable
and performant: it can handle billions of vectors efficiently and with minimal latency. Its
advanced vector indexing, search, and retrieval capabilities make it ideal for applications that
require fast and accurate search results.
It supports vertical and horizontal scaling, advanced compression techniques, highly
flexible deployment options including cloud-native, hybrid cloud, and private cloud solutions
and powerful security features. What differentiates it from Pinecone is that it can be run
locally, on-prem, in the cloud, or as a managed service, and it even offers a hybrid cloud option
for enterprises.
2
2.3 Weaviate
Weaviate is a vector database that stores both objects and vectors, allowing for combining
vector search. It is modular, cloud-native, and real-time, making it easy to scale machine
learning models. Weaviate comes with optional modules for text, image, and other media
types, which can be chosen based on your task and data. You can also use more than one
module, depending on the variety of data.
3 Vector Database Migration
Migrating between vector database solutions, such as moving data from Pinecone to
Milvus, can be moderately challenging due to differences in APIs, data structures, and feature
support.
While the task of transfering embediings is straighforward, we have to make sure that
the configurations of both databases are same for the transfer to be successful. For example,
pinecone is self managed and it hides the architecture of its infrastructure while Milvus requires
manual management.
3.1 Tools for Database Migration
There are various tools and techniques for vector database migration like
• VTS built on Apache SeaTurnnel
• Zilliz Migration Service
3.2 Steps Involved in Database Migration
The steps involved in migrating from one database to another involves
1. Identify the source and target database
2. Determine the data that must be transfered like metadata, vectors, schemas
3. We have to compare compatibility between both databases. This includes dimensions,
data types and constraints
4. Extract embeddings and save in portable formats like .json or .npy.
5. Import the data using API, SDKs, or ETL tools.
