Cockroach Labs Blog - engineering

The Database at 550 Kilometers: What Orbital Computing Means for Distributed Databases

I. The hum of modern data centers

In Ashburn, Virginia, a row of servers draws 40 megawatts from the grid and exhales it as heat. Chilled water circulates through copper pipes at 2,800 liters per minute to carry that heat away. Outside, cooling towers the height of apartment buildings return it to the atmosphere. Ashburn now consumes more electricity than some nations. The machines inside process queries. They store application state.

This is a data center and there are thousands like it. They account for 4.4 percent of the electricity consumed in the United States, a figure that will double before the end of the decade. Concerns about their environmental impact and energy costs led New York to pass a one-year ban on the construction of new hyperscaler data centers in July 2026. 14 other states have also introduced bills to restrict new data center construction.

II. Why are companies planning data centers in space?

In January 2026, SpaceX filed an application with the Federal Communications Commission for one million satellites. These are not communication satellites, but data centers. At the World Economic Forum in Davos, Elon Musk stated that the lowest-cost location for AI computation would be in space within two to three years.

Separately, a company called Starcloud filed for 88,000 satellites of its own. Starcloud had already placed a single Nvidia H100 GPU into low Earth orbit and run a Gemini inference workload on it. The chip functioned and the workload completed.

The combined investment required to deploy these constellations exceeds one trillion dollars. The economics rest on three substitutions:

Solar power replaces grid electricity, and in orbit it is five to seven times more productive than on the surface. No atmosphere absorbs it. No clouds block it.
Radiator panels replace cooling water. Heat leaves the satellite by emission into the cosmic microwave background at 2.7 Kelvin, the residual temperature of the universe itself. The reservoir is free. The panels are not. Vacuum is also an excellent insulator, and a 700-watt chip must shed every joule through radiation alone, into a sky that contains the Earth and, half the time, the Sun.
Lastly, open orbit replaces land and zoning. While several U.S. states are currently considering legislation to pause terrestrial data center construction, there are no neighbors in orbit to object. Well, none that we know of!

Until now, most discussion of orbital infrastructure has focused on compute: how to power GPUs, cool them, and connect them across space. However, every stateful application also depends on a database. If space-based computing becomes practical, distributed databases will face an entirely new operating environment: one where latency, topology, failure domains, and even jurisdiction are constantly in motion. This article explores what those challenges might look like, and why some of the architectural principles developed for terrestrial distributed systems may prove unexpectedly relevant beyond Earth.

III. What happens when databases move to space?

Moving databases into space introduces a different set of engineering challenges than moving compute. While most of the orbital computing conversation is about GPUs, inference, training runs, and floating-point operations per second per dollar per kilowatt, every stateful application ultimately depends on a distributed database to persist state, coordinate work, and maintain consistency. These are important quantities, and the engineers working on them are solving genuinely difficult problems in thermal management, power delivery, and radiation hardening.

Compute is stateless, however. A GPU performs a matrix multiplication, returns a result, and retains nothing. It has no memory of the previous calculation and no obligation to the next one. Every application that persists data, coordinates work, or maintains a record of what has happened requires a database to store that information. A database must survive node failure, tolerate network partitions, and maintain consistency across replicas that may disagree about the order of events.

One million satellites will run computations. Some of those computations will need to remember their results. Where does that memory live and how will it be managed?

That question shifts the conversation from orbital compute to orbital data infrastructure. Solving one without the other would leave many real-world applications incomplete.

IV. How do distributed databases already solve part of the problem?

CockroachDB was designed for a scenario that, at the time of its creation, was entirely terrestrial: a database distributed across multiple data centers, in multiple geographic regions around the world, that survives the loss of any node, any rack, or any entire data center without human intervention and without losing a single committed transaction.

One of the mechanisms that drives this behavior is a consensus protocol called Raft. Every write is proposed to a group of replicas. A majority must agree before the write is committed. If a replica fails to respond, whether because of a network partition, a hardware fault, or atmospheric reentry, the remaining replicas continue without it. No operator is paged. The system just handles it.

On the ground, this architecture protects against power outages, fiber cuts, and the occasional backhoe. The nodes sit in racks, the racks sit in buildings, and the buildings sit on foundations that don’tt move. In orbit, the foundations move at 7.66 kilometers per second and instead of backhoes or water leaks, there are meteor showers.

V. How would a distributed database work across a satellite constellation?

Consider a cluster of 300 CockroachDB nodes distributed across a constellation in low Earth orbit at 550 kilometers altitude. Each node is contained in a satellite, and each satellite communicates with its neighbors via optical laser links. The constellation is arranged in orbital planes, each plane containing several dozen satellites that trace the same ground track, offset by time. (An orbital plane is a flat surface that passes through the center of the Earth: The orbital plane contains the satellite's orbit, every orbit traces an ellipse, and that ellipse is contained within that single flat surface. It’s like sliding a cardboard sheet through the center of an orange.)

A CockroachDB cluster on the ground uses a concept called localities to describe its topology. A node might declare itself as --locality=region=us-east,zone=us-east-1a. The database uses this information to place data near the users who access it and to ensure that replicas are spread across failure domains. If us-east goes down, an application can use the data copies in us-west and eu-central.

An orbital node might declare itself as --locality=shell=leo-550,plane=47,position=12. The database would distribute replicas across planes, ensuring that no single orbital plane is a single point of failure. A micrometeorite strike that disables three satellites in Plane 47 does not affect replicas in Planes 23 and 68. A "shell" is a spherical surface at a given altitude, a soap bubble at a certain altitude above the Earth. Orbital planes intersect the Earth and the shells. The satellite travels across the shell where it intersects with the orbital plane.

This much is straightforward. CockroachDB already knows how to do this. The localities are strings. The replication logic is the same whether the string says us-east or plane-47.

The problems begin when the strings start to move.

State of Resilience 2025: Confronting Outages, Downtime, and Enterprise Readiness — a 2025 report on operational resilience across 1,000 global enterprises and how distributed SQL combats costly downtime.

VI. Should data live in orbit or on Earth?

Whether data should live in orbit or on Earth depends on a tradeoff between autonomy and economics. Understanding that tradeoff starts with an important difference between compute and storage.

A GPU is an extraordinary value per kilogram. An Nvidia H100 weighs approximately 3 kilograms and, in a terrestrial data center, generates revenue measured in thousands of dollars per hour. At the current Falcon 9 launch cost of roughly $3,300 per kilogram, the launch expense is a rounding error against the card's lifetime earnings. The economics of space-based compute work because compute is dense in value per unit mass. Storage is not.

An enterprise NVMe SSD stores four terabytes in about 200 grams. The launch cost for the drive itself is trivial: less than a dollar per terabyte to put it in orbit. But the drive requires a satellite. The satellite requires a chassis, solar panels, thermal radiators, a laser communication terminal, and a station-keeping propulsion system. The total mass per node is measured in hundreds of kilograms. At $3,300 per kilogram, each satellite costs several hundred thousand dollars to launch before it stores a single byte.

And storage, unlike compute, does not earn its keep through activity. A GPU performs work. It multiplies matrices, runs inference, returns results. Its value is measured per hour. A disk holds state. It persists a row that was written once and may be read four times over the next year, or four million times. Its value is measured per year of durability, not per cycle of computation.

This creates a question that the compute-focused orbital industry has not yet confronted: Is it cheaper to store data in orbit or to transmit it to the ground?

The answer has implications beyond infrastructure costs. It affects how autonomous an orbital platform can become, how resilient it remains during network disruptions, and which categories of applications, from scientific research to defense to AI inference, can operate without depending on continuous connectivity to Earth.

The argument for orbital storage is autonomy. A database that keeps its state on the satellites can operate independently of ground stations. If the laser link to the surface is interrupted by weather, orbital mechanics, or geopolitics, the constellation continues serving queries from local replicas. The data is co-located with the compute, for low latency low and high availability.

The argument for ground storage is economics. Terrestrial storage costs approximately $20 per terabyte per year. Space-based storage, once satellite and launch costs are amortized, costs orders of magnitude more. A hybrid architecture that runs compute in orbit but persists durable state to ground stations would minimize the mass in orbit and the cost of the constellation.

However, a hybrid architecture reintroduces the very dependency that orbital computing was meant to eliminate. If the database is on the ground, every orbital query must wait for a round trip through the atmosphere. At 550 kilometers, that is 3.67 milliseconds each way. That may be acceptable for a single query. For a write that must be acknowledged before the next computation proceeds, it is a bottleneck that compounds with every dependent operation.

The likely answer is the same one that distributed databases have always given: It depends on the workload. Hot data that is frequently accessed and latency-sensitive, lives near the compute. Cold data that is archival, compliance-driven and infrequently read sinks to the cheapest storage available. On the ground, that means SSDs for hot data and object storage for cold. In orbit, it means satellite storage for hot data and ground stations for cold.

VII. How do latency and consensus change in orbit?

A signal traveling from a satellite at 550 kilometers altitude to a ground station directly below requires 1.83 milliseconds. The round trip takes 3.67 milliseconds. This is fast enough for most database operations. A Raft consensus round among three replicas on the same orbital plane, separated by a few hundred kilometers of laser link, might complete in under 5 milliseconds.

But not all replicas will be on the same plane. The entire purpose of distributing replicas across failure domains is that they are physically separated. A satellite in Plane 12 communicating with a satellite in Plane 47 on the opposite side of the constellation may need to route through five or six intermediate satellites, each adding its processing and propagation delay. A consensus round across the constellation could take 30 to 80 milliseconds.

On the ground, this would be unremarkable. A CockroachDB cluster spanning US-East and EU-West routinely completes consensus in 70 to 90 milliseconds. The database is designed for it.

In orbit, the latency is not constant: Since the satellites are moving, the routing topology changes with every orbit. A consensus round that takes 30 milliseconds at time T may take 65 milliseconds at time T plus twelve minutes, when the orbital geometry has shifted and the optimal laser path now transits through a different set of relay nodes.

CockroachDB handles this. Its leaseholder mechanism assigns a single node as the coordinator for each range of data. Reads are served locally from the leaseholder. Writes go through Raft consensus but require only a majority quorum. If one replica is temporarily further away due to orbital mechanics, the other two can still commit the write.

However, optimal leaseholder placement on the ground is a function of user geography: Place the leaseholder near the users who read the data most frequently. In orbit, "near" is not a fixed concept. The leaseholder that is closest to the ground station in Singapore right now will be over the Indian Ocean in twelve minutes and over East Africa in twenty-four.

More broadly, orbital computing transforms what "local" means. On Earth, locality is primarily geographic. In orbit, locality becomes temporal as well, because the optimal location for serving data changes continuously as satellites move. Database architecture must adapt to both dimensions.

VIII. Does relativity matter for an orbital database?

Surprisingly, relativity is not the primary engineering challenge for an orbital database. A satellite at 550 kilometers altitude moves at 7.66 kilometers per second through a gravitational field weaker than the one on the surface below. The reader familiar with GPS will expect this to matter: GPS satellites carry atomic clocks that drift 38 microseconds per day due to relativistic effects, and the system would be useless without corrections. Surely a database in orbit must contend with the same physics.

At 550 kilometers, general relativity causes the onboard clock to run approximately five microseconds per day fast, because gravity is weaker and time passes more quickly further from the mass. Special relativity causes it to run approximately 28 microseconds per day slow, because the satellite is moving at orbital velocity and moving clocks run slow. The net drift is roughly 23 microseconds per day. The orbital clock falls behind the ground clock by about 0.27 nanoseconds every second.

CockroachDB does not use atomic clocks. It uses hybrid logical clocks, a mechanism that combines a physical timestamp with a logical counter to establish a consistent ordering of events across nodes. The system is designed to tolerate clock skew. Its default maximum offset is 500 milliseconds. The relativistic drift across the duration of a database transaction, which completes in milliseconds, is on the order of picoseconds. That’s smaller than the tolerance by a factor of roughly one trillion.

The physics that the public associates with space, the curvature of spacetime, the dilation of clocks, the warping of simultaneity, is irrelevant to the engineering of an orbital database. The physics that matters is far more mundane: the propagation delay of light across a laser link, the changing topology of a constellation as it rotates, and the three-millisecond penalty for every message that must traverse the atmosphere. Einstein is not the problem. Newton is the problem.

In other words, the limiting factor isn’t exotic physics. It’s the practical realities of moving data across a changing network fast enough to maintain consistency.

IX. How should databases adapt to moving satellite networks?

Databases operating across satellite constellations must continuously adapt to a network whose topology is always changing. The challenge of optimal leaseholder placement in an orbital constellation is about geometry, not databases. A constellation of satellites forms a mesh, which is embedded on a set of orbital shells. Each shell is a sphere at a fixed altitude, and the satellites trace geodesics on that sphere, which is to say, great circles. The mesh is not static; it rotates, and the relative distances between nodes on different orbital planes change continuously as the planes precess.

The latency between any two nodes is a function of the geodesic distance on the mesh at a given moment, modified by the routing topology of the laser links. The optimal leaseholder placement for a given range of data is the node that minimizes the expected latency to the set of clients accessing that range.

On the ground, this optimization is performed periodically and produces a static assignment: leaseholder for range R goes to node N in region X. In orbit, the optimal assignment drifts continuously. The leaseholder that minimizes latency at minute zero is suboptimal by minute six and possibly the worst choice by minute forty-seven.

A simple approximation may suffice for early deployments: Precompute a periodic leaseholder schedule based on the known orbital elements. The constellation repeats its geometry every orbital period, roughly 95 minutes for a 550-kilometer orbit. The leaseholder placement repeats with it. The database could carry a timetable, like a train schedule, rotating leaseholders through a predetermined sequence.

This would not be optimal. However, it would be correct, and correctness – at 550 kilometers – matters more than optimality.

X. How does data sovereignty work in orbit?

Data sovereignty becomes significantly more complex in orbit because satellites routinely cross national jurisdictions. A satellite in low Earth orbit at 550 kilometers altitude completes one orbit in approximately 95 minutes. In that time, it crosses over dozens of national jurisdictions. The ground track of a single orbit might pass over Canada, the Atlantic Ocean, Portugal, Algeria, Niger, Chad, the Central African Republic, the Democratic Republic of Congo, Zambia, Mozambique, the Indian Ocean, and Australia before the satellite has completed half its circuit.

Data sovereignty laws are defined by terrestrial borders. The European Union's General Data Protection Regulation applies to the personal data of EU residents, regardless of where the data is physically stored. But "where the data is physically stored" has always meant a building with an address. A rack with a serial number. A jurisdiction with a court.

A satellite over Portugal holds a copy of a row written by a user in Frankfurt. Ninety-five minutes later, that satellite and that row are over the Pacific. Does this constitute a cross-border data transfer? That’s a question that no existing regulation was written to answer. However those questions are ultimately resolved, they will influence far more than database design. They’ll shape where regulated industries can deploy orbital applications, how global services satisfy regional compliance requirements, and which governance models emerge for computing beyond terrestrial infrastructure.

CockroachDB's zone configurations allow operators to pin data to specific localities. On the ground, this means constraining replicas to specific regions: EU data stays in EU data centers. In orbit, the "EU data center" is an EU data center for approximately eight minutes per orbit. Then it’s an Atlantic data center, then an African data center...

The database can be configured to replicate data only to satellites that are currently over approved jurisdictions. But "currently" changes every few minutes, and re-replicating data at orbital velocity would generate more network traffic than the laser links could carry.

The more practical approach: define sovereignty by the satellite's registration, not its position. A satellite registered in the EU and operated by an EU entity could be considered EU territory for data purposes, much as a ship flying a flag is subject to the laws of its flag state regardless of its location at sea.

XI. What happens when an orbital database node disappears?

An orbital database must distinguish between nodes that are temporarily unreachable and those that are permanently lost. On the ground, when a CockroachDB node fails, an operator investigates. The node might be restarted, its disk replaced, its data rebalanced to surviving nodes.

CockroachDB's decommission process is designed for planned removals. An operator marks a node as decommissioning, and the database migrates its data to other nodes before shutting it down cleanly. This process assumes the node is reachable and cooperative.

But there is a subtlety. On the ground, a failed node might come back. The operator reboots it, and it rejoins the cluster with its data intact. The database reconciles the returning node's state with the current state and continues. In orbit, a node that goes silent might return after a brief communication blackout caused by orbital geometry, or it might never return because it has reentered the atmosphere.

The database can’t distinguish between these cases in real time. The Raft protocol doesn’t include a field for "this node is burning up over the South Pacific." The liveness timeout must be tuned carefully: too short, and the database will replicate data unnecessarily every time a satellite passes behind the Earth relative to its peers; too long, and the cluster operates with reduced redundancy while waiting for a node that will never return.

Hardware obsolescence introduces another dimension. For example, Nvidia releases new GPU architectures faster than satellites can be manufactured and launched. A constellation deployed in 2028 may contain hardware that is two generations behind by 2030. Rolling upgrades, the standard method for updating CockroachDB, require taking nodes offline and replacing the binary. In orbit, "replacing the binary" is possible through software updates. Replacing the hardware is not. The node's CPU, memory, and storage are fixed for the lifetime of the satellite, which may be five to ten years.

CockroachDB has always been hardware-agnostic. It runs on whatever the operating system provides. In orbit, this property becomes essential: The database must continue to operate on aging hardware while newer satellites join the constellation with faster processors and larger storage. The cluster must absorb heterogeneous nodes gracefully, routing more work to the faster nodes and less to the older ones.

This is, again, something CockroachDB already does. It just hasn’t done it at orbital timescales.

XII. What would orbital databases still need?

Existing distributed databases could operate in orbit today, but they would require new capabilities to become truly orbital-aware. CockroachDB would function in an orbital constellation today. It would get there and survive. It would not be optimized for the environment, however.

Many of these capabilities would have value beyond orbital computing. As infrastructure becomes increasingly dynamic, whether because of mobile edge deployments, autonomous systems, or future space-based platforms, the ability for distributed databases to adapt automatically to changing topology, latency, and failure domains becomes broadly applicable.

The features that a space-based database deployment would require don’t yet exist, including:

Dynamic localities. The current locality system assumes that a node's position in the topology is fixed. A node in us-east stays in us-east. An orbital node's effective locality changes continuously. The database would need a locality system that accepts position updates and adjusts replica placement accordingly, or a higher-level abstraction that treats orbital shells and planes as stable localities while acknowledging that their relationship to the ground shifts with time.

Orbital-aware leaseholder policies. Leaseholder placement today is driven by access patterns observed over time. In orbit, the access patterns are predictable from physics. The database should accept an orbital model and pre-compute leaseholder schedules, rather than discovering the optimal placement reactively through latency observations.

Autonomous decommission. The current decommission workflow requires an operator to initiate the process. An orbital node that is clearly failing should be able to trigger its own decommission and migrate its data to surviving nodes without waiting for humans or agents to issue a command from the ground.

Jurisdiction-aware replication. Zone configurations would need a temporal dimension: “This data may reside on this node only while the node is over an approved jurisdiction.” That’s likely impractical at fine granularity, but the framework for expressing the constraint should exist, even if the initial implementation uses flag-state semantics rather than real-time position.

History suggests that infrastructure evolves by solving tomorrow's problems before they become mainstream. Distributed databases were built to tolerate failures across regions and continents long before anyone imagined constellations of orbital data centers. If computing continues to expand beyond Earth's surface, the next generation of distributed systems may prove that some of today's most resilient architectures were already pointing in that direction.

These are engineering problems and they are solvable. Orbital engineering is dominated by considerations of weight. Data doesn't have physical weight, but it does have operational weight. It must be replicated. It must be consistent. It must survive.

CockroachDB: The Definitive Guide, 2nd Edition (O'Reilly) — a practical deep dive into multi-region deployment, survival goals, and resilience for readers who want to go from thought experiment to production.

This post draws on reporting by Eric Berger in Ars Technica, March 2026.

Isaac Wong is EVP of Research & Development at Cockroach Labs, where he leads the engineering organization behind CockroachDB to shape the company's long-term technical vision. He oversees the teams driving the database's core architecture, reliability, and continued innovation.

Vehicle Search with SQL and Vector Embeddings

"I'll know it when I see it" is the classic car buyer's line, but it's also the one thing old-school search bars totally fail to deliver.

Traditional keyword search fails when a customer wants a car that "looks like this photo from Fast and Furious." This is where vector search comes in, transforming unstructured data into mathematical vectors to find semantically similar items.

In this blog I’ll walk you through a fun little demo I cooked up called "Cockroach Cars." I use a combination of Python and SQL to find cars currently available for purchase. We'll check out how to use CockroachDB as a spot to stash vectors, make embeddings, peek at the vector space, and even do cool image-to-image similarity searches—all right inside the database just using SQL.

1. Visual Search: Finding That "Fast and Furious" Look

Time to talk about the main attraction: finding a car just from a picture! This is the cool part. There is no need to sift through lists, hoping the seller provided the right keywords in their description, like "vintage muscle" or "street racer." Just paste the photo, the app turns it into a vector, and then asks CockroachDB to find the most similar cars.

The Process

Input: The user copies an image to their clipboard.
Embedding: In the program, a vector embedding is generated for that image using the loaded CLIP model.
SQL Query: We run a K-Nearest Neighbors (KNN) search directly in CockroachDB using the <-> (L2 Distance/Euclidean) operator.
Provide filters like the number of items returned within a price range.

The user copies an image of their ideal "Fast and Furious" vehicle to the clipboard, along with the quantity of cars found in the search results and the desired price range.

Check out the SQL query that makes this semantic magic happen to find the ride you actually want.

The query filters by price range (a standard scalar filter) and sorts by vector similarity (distance) simultaneously.

The Results

In our example search, we used an input image of a classic muscle car. The search returned:

1969 Pontiac Firebird ($46k)
1969 Chevrolet Camaro ($335k)
1966 Chevrolet Sport ($41k)
1968 Pontiac GTO ($45k)
1964 Oldsmobile Jetstar 88 Holiday Coupe ($33k)
1967 Pontiac Firebird ($29k)

The results are remarkably consistent. Despite different makes and models, all returned vehicles share the distinctive visual characteristics of late-60s American muscle cars (boxy hoods, circular headlights, chrome bumpers).

2. Under the Hood

To bring "Cockroach Cars" to life, we need a setup that can handle both standard relational data and high-dimensional vector operations without getting bogged down. A stack that is developer-friendly, and capable of executing complex similarity searches right inside the database. Here is the specific toolkit I used to build this engine.

The Tech Stack

CockroachDB Database: With distributed SQL and vector search, all in one.
Language: Python (Jupyter Notebook environment)
Python Libraries: sentence-transformers (CLIP model), scikit-learn (PCA/KMeans), plotly (3D visualization)
Database Driver: psycopg (PostgreSQL adapter)

I’m using the CLIP (Contrastive Language-Image Pre-training) model (the clip-ViT-B-32 version from sentence-transformers). CLIP is perfect for this because it puts both images and text in the same "vector space," which lets you search with either text-to-image or image-to-image.

    # Load the model
    from sentence_transformers import SentenceTransformer
    model = SentenceTransformer('clip-ViT-B-32')

Then connect to a CockroachDB cluster using a connection string formatted for psycopg.

The Data

I collected the vehicle image dataset from various internet sources, including Kaggle and GitHub. To create the corresponding synthetic metadata, I developed a Python script leveraging the Faker package. The complete process involved three steps:

Loading the images into a staging table
Generating both the image embeddings and the metadata
Inserting this information into the for_sale_inventory table

This final table holds the vehicle metadata alongside their respective image embeddings.

3. Data Exploration

Before diving into vectors, let’s do some standard SQL analysis to get a feel for the dataset. We're working with 4,597 vehicles and 19 columns with images, embeddings, and metadata.

A quick look at the data reveals columns like vin, make, model, price_in_usd, registration_date, fuel_type and image_embedding.

result = %sql SELECT vin, make, model, price_in_usd, registration_date, fuel_type, image_embedding FROM for_sale_inventory;                                     
df = result.DataFrame()        # Convert to Pandas DataFrame
df.head(5)                     # Preview 5 data rows

Upon execution of the code above, the query successfully retrieved 4,597 records from the inventory table, which are now stored in our DataFrame (see below).

Visualizing Inventory Distribution

I use matplotlib and seaborn to whip up a dashboard for visualizing the inventory from the Pandas dataframe (result) created in the sample code above.

Price Distribution: A histogram shows the price spread, likely right-skewed with a long tail of luxury vehicles.
Fuel Type: A pie chart reveals a diverse mix of Electric (21.9%), Hybrid (22.1%), Petrol (33.2%), and Diesel (22.8%) vehicles.
Inventory Cloud: Word clouds for "Make" and "Model" highlight dominant brands like Volkswagen, Acura, and Pontiac.

4. Visualizing the Vector Space

CockroachDB's C-SPANN vector indexing is super smart! It uses a hierarchical k-means tree (see figure below) to organize vector data for very efficient similarity searches. Basically, it grabs your categorized data (like, all the Toyota models), groups them using k-means (think of it like sorting them into handy 3D clusters), and then makes a map of all that complex high-dimensional info. This organized tree lets the database zoom right to the closest matches without having to look through everything.

Image Caption: Hierarchical K-means Tree

For more information see:

Introducing Distributed Vector Indexing to CockroachDB by David Bressler
Real-Time Indexing for Billions of Vectors: How we built fast, fresh vector indexing at scale in CockroachDB by Andy Kimball

RELATED: Check out this video by Technical Evangelist Rob Reid, who explains why having vectors in the same database as your relational data is a modern-day super power.

Okay, so those car image vectors are 512-dimension arrays of numbers, which, let's be honest, don't make much sense to us humans. To help us see how the images relate to each other, we shrink those super-high-dimensional vectors down to a simple 3-dimensions. In this example we only do this dimensionality reduction for the bottom, or 'leaf,' nodes, though.

The Pipeline

Fetch Embeddings: We retrieve the raw vector strings from CockroachDB.
Parse & Standardize: Convert strings to float32 arrays and normalize them using StandardScaler.
Dimensionality Reduction (PCA): We use Principal Component Analysis (PCA) to reduce the 512 dimensions down to just 3 components. This retains the most significant variance in the data.
Clustering (K-Means): We group the vehicles into 50 clusters (representing different car makes) to color-code the visualization.

The 3D Plot

Using plotly, we plot the vehicles in 3D space.

In the interactive plot provided, visually similar vehicles are naturally grouped together. The red diamonds, highlighted on the left side of the figure above, represent the centroids, such as the coordinates for Toyota. The right side of the figure displays the vector coordinates (a car image) corresponding to a selected embedding.

Conclusion

So, the "Cockroach Cars" walkthrough shows how awesome it is when you blend vector search right into standard SQL. You ditch the need for separate embedding services or vector databases, which means you can build smart apps that filter data and find similar stuff semantically with just one quick, easy query. We can query metadata (WHERE price > 50000) and vectors (WHERE image_embedding <-> query) in a single, ACID-compliant SQL statement.

This doesn't just make the whole setup and your data pipeline simpler; you also get all that legendary CockroachDB resilience, easy scaling and multi-region capabilities. So, even the craziest "fast and furious" searches stay efficient, reliable, and available everywhere.

Learn more about how top AI Innovators are achieving success with CockroachDB

Alejandro Infanzon is a Solutions Architect at Cockroach Labs.

Value Separation in Pebble: Storage Engine Optimization

Motivation

At its foundation, CockroachDB depends on a key-value storage engine called Pebble. In CockroachDB v25.4, the storage team introduced value separation within Pebble: an optimization that improves compaction efficiency for many workloads. Value separation increases the throughput of the core of CockroachDB, the storage engine, by up to ~50%, depending on the workload, through algorithmic improvements. It reduces redundant I/O, reducing cost-to-serve at scale. We’ll look at how Pebble represents data today, what’s changed, and how that translates to database efficiency.

Background

The higher layers of CockroachDB write key-value (KV) pairs to Pebble, and the storage engine persists them durably while maintaining organization to efficiently service reads. Internally, Pebble implements a log-structured merge-tree (LSM): a data structure that organizes KV pairs into key-sorted runs. Asynchronous compactions continuously rewrite data in the background, sorting KVs to keep up with incoming writes.

During these compactions, a selection of input SSTables (SSTs) are read, merge-sorted by key, and written to new SSTs in sorted order. Compactions reduce the cost of reads by reducing the number of read I/O operations performed per logical read operation (called read amplification). But compactions come with tradeoffs. Namely, compactions consume CPU, read bandwidth, and substantial write bandwidth.

Workloads with large values (e.g. wide rows or JSONB documents) especially incur the overhead of compactions since these operations must read and rewrite substantial data. The sorting that compactions perform is a function of keys only; not values. Based on this observation, significant research (notably the WiscKey paper) has explored the concept of value separation, storing values out-of-band, physically separated from the corresponding keys, allowing compactions to sort keys without repeatedly rewriting values. This physical separation of keys and values provides additional flexibility in the tradeoff between read performance, space amplification, and write amplification.

In this blog post, we’ll examine the tradeoffs involved in value separation, the unique design of Pebble’s implementation, and the empirical benefits we’ve observed.

Impact

Cockroach Labs runs regular benchmarks of various workloads against nightly builds of CockroachDB to track the performance of the database over time and catch regressions early. The below graph plots the throughput of a simple workload upserting key-value pairs with small keys (BIGINT primary key) and 4 KiB values. Value separation was enabled in this benchmark at the end of June, resulting in a ~47% increase in throughput on the same hardware.

Image Caption: kv0/values=4096 benchmark throughput over time (2025)

This benchmark demonstrates this feature’s significant impact on efficiency and cost-to-serve. Workloads with smaller values see less pronounced impact. Various heuristics determine which values are physically separated and allow even small-valued workloads to benefit.

Prior art

Value separation is a frequently examined topic in log-structured merge tree storage systems. Projects like WiscKey (Lu et al., FAST 2016) and LavaStore (Wang et al., VLDB 2024) demonstrated that significant reductions in write amplification are possible through storing values out-of-band. However, separation of values comes with its own tradeoffs, and none of these published implementations were appropriate for CockroachDB’s workloads.

Pebble sits at the core of CockroachDB’s storage layer where correctness, performance, and a reasonable cost-to-serve all have to coexist. Much of the complexity in external value separation lies within the heuristics of when to compact values to ensure reads remain performant, space amplification is minimal, while still realizing the write-amplification advantages.

Mechanics

Pebble’s implementation of value separation introduces a new type of file to the storage engine called a blob file. A blob file stores values that have been separated, independent from the corresponding key. Physically, a blob file consists of a series of value blocks, containing values, an index describing where the value blocks begin and end, and a footer of metadata.

Both index blocks and value blocks are encoded in Pebble’s columnar-block format (similar to PAX) allowing constant time lookup of a block by its block ID and a value by its value ID. When a key’s value is separated into a blob file, the SSTable KV encodes a concise value handle:

    // Handle describes the location of a value stored within a blob file.
    type Handle struct {
      BlobFileID base.BlobFileID
      ValueLen   uint32
      // BlockID identifies the block within the blob file containing the value.
      BlockID BlockID
      // ValueID identifies the value within the block identified by BlockID.
      ValueID BlockValueID
    }

A value handle compactly describes where a reader should look to find the value. When a compaction encounters a separated value, most of the time it simply copies the value handle as-is to the new output SSTable. For large values, the value handle is orders of magnitude smaller than the value itself, and compactions that copy just the handle conserve read and write bandwidth.

When a reader needs to access a value, it uses the handle to load the identified file, block and finally value. This indirection introduces additional read I/O, so separating a value is a tradeoff. Infrequently retrieved values or large values are better candidates for separation. Iterators cache blob files and their blocks over the lifetime of an iterator to avoid duplicate lookups.

Tradeoffs

Locality

When CockroachDB rewrites blob files during flushes or compactions, the process initially establishes a simple relationship: each new blob file corresponds directly to a single output SSTable. However over time, as the LSM evolves through subsequent compactions, these once clean one-to-one relationships begin to fragment. A side effect of this is that the values referenced by SSTs become increasingly scattered across more and more blob files. This loss of locality slows scans, as caching of recently accessed blob file blocks becomes less effective.

In order to mitigate this effect, pebble maintains a per-SSTable property called the blob reference depth. The reference depth is the maximum number of blob files in the working set of a scan across an SST. For example an SST in which keys within the range [a,f) reference one blob file and keys within the range [f,z) reference a distinct blob file has a reference depth of 1. When a compaction’s output SSTables would have a large blob reference depth due to the increasing interleaving of blob file references, Pebble instead writes new blob files. Writing new blob files copies the referenced values to new files and results in output SSTs with a reference depth of 1 again. Bounding the reference depth bounds the overhead incurred by reads.

Space Amplification

The increasing fragmentation of references to blob files has another negative effect: space amplification. Compactions may eliminate keys that reference blob values. This occurs when a key is overwritten by a newer version or deleted by a tombstone. Additionally, when the reference depth heuristic triggers a compaction to write new blob files, the compaction eliminates references to many values in blob files at once. Over time, these effects accumulate. Each blob file increasingly consists of a mix of live data: values still referenced by SSTables in the current LSM version and dead data: values that have been logically deleted or superseded.

This gradual divergence between total blob file size and the volume of live data stored within it leads to blob file space amplification. Managing and mitigating this amplification efficiently is key to keeping storage utilization under control and maintaining predictable performance at scale. Notably, in workloads with high value homogeneity, this amplification is less pronounced because the resulting blob files and SSTables tend to compress more effectively.

Blob File Rewrites

To keep blob file space amplification in check, Pebble introduces a new compaction type: blob file rewrite compactions. Over time, as keys are overwritten or deleted, blob files accumulate a mix of live and dead values. These rewrite compactions reclaim that space by copying only still-referenced values into new blob files.

Each SSTable that points to external blob values carries a small blob-reference index block containing a run-length encoded bitmap per referenced blob file recording which values within a blob file are still live:

When Pebble’s heuristics decide to rewrite a blob file, Pebble collects all the SSTs that reference that file and unions their bitmaps together, OR-ing them into a single view of which values remain in use. Values not referenced by any SSTable are dropped during the rewrite.

Blob file rewrite compactions move and reorganize values that still have outstanding references, but they do not rewrite the referencing SSTables. Doing so would be expensive and reduce the write-bandwidth savings of value separation. Instead, Pebble’s value handles are designed to continue to identify the correct value across blob file rewrites through additional indirection:

The handle’s BlobFileID is a stable identifier for a logical blob file. The LSM maintains a mapping from BlobFileID to a physical blob file.
The handle’s BlockID is an index into the original blob file’s value blocks. A blob file that is produced by a blob-file rewrite compaction includes a special column within the index block remapping the original blob file’s BlockID to the index of the physical block in the rewritten blob file.
The handle’s ValueID is an index into the original blob file’s value block. In order for the value index to remain correct within the rewritten blob file, we cannot remove values altogether, but we can replace them with empty values (effectively storing a single offset , typically 2 bytes per absent value). As an additional optimization, when re-mapping the above BlockID, we also include a value ID offset allowing us to avoid storing empty values for the first n empty values within the block.

Blob file rewrites allow Pebble to reclaim disk space storing garbage. They prevent unreferenced data from lingering indefinitely, keeping space amplification low while preserving the performance and efficiency benefits of separating values from keys.

CockroachDB separation heuristics

Workloads with large values reliably benefit from value separation and by default CockroachDB in v25.4+ will separate values 256 bytes and larger automatically. CockroachDB can realize benefits more broadly too. Some regions of the storage engine’s keyspace are rarely read and more latency tolerant (for example, the Raft log which is typically held in-memory until fully replicated). CockroachDB stores these values out-of-band too, even if they’re smaller than 256 bytes, thereby avoiding pulling them into compactions unnecessarily, reducing write amplification without affecting read-path latency where it matters.

CockroachDB uses multi-version concurrency control (MVCC) to negotiate concurrent transaction commits. MVCC preserves obsolete versions of rows for transaction isolation and powering AS OF SYSTEM TIME (AOST) queries. In practice, most queries execute at current timestamps and only read the most recent version of a row. CockroachDB eagerly separates the values corresponding to MVCC garbage, improving the locality for reads at recent timestamps and reducing the impact of compactions through value separation. However, some nuances higher in the system currently prevent CockroachDB from fully realizing this benefit.

Notably, CockroachDB's SQL optimizer issues a significant number of AS OF SYSTEM TIME queries to maintain accurate statistics, and these queries intentionally scan across MVCC history. As a result, these scans and their retrievals of separated values can dominate read bandwidth, offsetting much of the theoretical benefit of separating historical values. Balancing these competing behaviors – reducing unnecessary I/O while still supporting efficient AOST scans – will guide future work.

The concept of blob files could also allow us to explore the idea of enabling the automatic and cost-effective placement of data onto appropriate storage tiers. Less frequently accessed data could be moved to inexpensive storage tiers by classifying it based on data age, while maintaining frequently accessed data on fast, expensive storage.

Try CockroachDB Today

Spin up your first CockroachDB Cloud cluster in minutes. Start with $400 in free credits. Or get a free 30-day trial of CockroachDB Enterprise on self-hosted environments.

Real-Time Indexing for Billions of Vectors: How we built fast, fresh vector indexing at scale in CockroachDB

In a past life, I worked on an app that let users upload and share photos with friends and family. I’m amazed at how far technology has progressed since that time. It feels like magic that AI can “look” at a set of photos and “know” that they were taken at a child’s first birthday party or on a hike in the mountains. Natural language queries like “show me photos from my trip to the Statue of Liberty” or even “find that photo where I’m about to collide on the soccer field with another player” are no longer science fiction.

Our photo-sharing startup never reached millions of users, but I suspect that many of you reading this are working on systems that have. At that scale, it’s surprisingly easy to find yourself managing billions, or even tens of billions of user-generated items. If it’s a photo app, most users will have hundreds or thousands of photos. Power users or organizations might have tens or hundreds of thousands. If it’s not photos, it might be documents, notes, videos, or audio. The type of content varies, but the math is the same: millions of users, each contributing hundreds or thousands of items, and you’re quickly operating at billions-scale.

Even with just a few hundred items, users expect fast, accurate search. If they upload something, they want to find it immediately. If they search, they want results in the blink of an eye. Increasingly, basic keyword search isn’t enough. In the age of ChatGPT, users expect semantic search, with results based on the meaning of the content, not just filenames, metadata, keywords, or tags.

Some solutions to this problem assume that the entire dataset fits into memory on a single machine. Or, at most, they rely on a fast local SSD. Many of them don’t expect your data to be distributed across regions, or to be constantly changing, or to be part of a transactional system where consistency and freshness actually matter. They often come with significant limitations, like requiring writes to be batched, returning stale results, or needing specialized hardware to perform well.

CockroachDB was built with a different set of assumptions. As a distributed database, it expects data to live across multiple machines, which may span availability zones or even regions. It’s designed to scale linearly, so that adding more machines leads to proportionally higher throughput. And as a transactional SQL database, it prioritizes returning fresh data and supporting real-time updates. All of that has to be resilient to machine, disk, and network failures.

Read on to learn how we combined recent academic research with practical engineering to solve the semantic search problem at massive scale, with fresh, real-time results, by leveraging CockroachDB’s unique distributed architecture.

Embedding Meaning into Vectors

To start, it’s important to understand how systems can make sense of photos or search documents by meaning. Companies like OpenAI offer embedding models that convert an image, document, or other media into a long list of floating-point numbers – a vector – that captures its meaning. If two photos or documents are similar, say two beach photos, they’ll be mapped to vectors that are near each other in high-dimensional space.

Embedding meaning into vectors reduces complex problems like image recognition and semantic search into a simpler one: finding nearby vectors. These models are built on the same deep learning techniques that power systems like ChatGPT – large neural networks trained to capture meaning and context across many kinds of data.

This even works across media types. Multimodal models embed text and images into the same vector space. So the word “beach” and an actual beach photo end up in the same region. When a user types “beach,” we can embed that query into a vector and search for nearby photo vectors. The closest matches are very likely to be related to the beach.

How Meaning is Indexed

Embedding vectors often have hundreds or thousands of dimensions that allow them to represent complex meaning. But that also makes them hard to search. Think about it: should beach photos come before or after food photos? What about photos of food at the beach? There’s no natural ordering for multi-dimensional vectors, the way there is for numbers or strings. That means traditional indexes don’t apply.

Instead of scanning for exact matches, semantic queries need to find vectors that are nearby in multi-dimensional space. At a small scale, brute-force search is often good enough – you can scan the dataset, compute distances, and return the closest matches. But as the number of vectors grows into the tens of thousands or beyond, that approach quickly becomes too slow to be practical.

Vector indexes address this by efficiently finding approximate nearest neighbors. These indexes trade a small amount of accuracy for a large gain in performance. While they don’t guarantee that the exact nearest vectors will be returned, the results are close enough to be useful, and the performance benefits make real-time search possible at scale.

Adapting Vector Indexing Algorithms for Distributed SQL

Even with a good vector indexing algorithm, plugging it into a distributed SQL database like CockroachDB isn’t straightforward. To support elastic scale, fault tolerance, and multi-region availability, any indexing algorithm needs to follow a set of architectural rules:

No central coordinator. Any node in the cluster should be able to serve reads and writes. The index can’t rely on a single leader to coordinate queries or updates.
No large in-memory structures. Index state must live in persistent storage. We can’t assume every node has gigabytes of RAM available for caching vectors, and we want to avoid long warm-up times spent building large in-memory structures. This is especially important for Serverless deployments.
Minimal network hops. Cross-node round-trips are expensive. Indexes that require sequential traversal across nodes can accumulate latency quickly and make performance unpredictable.
Sharding-compatible layout. Index data must map naturally to CockroachDB’s key-value ranges so it can be split, merged, and rebalanced like any other data.
No hot spots. As vector inserts and queries scale up, the index must avoid concentrating traffic on a single node or range. Load should be balanced across the cluster.
Incremental updates. The index must handle inserts and deletes in real time, without blocking queries, requiring large rebuilds, or hurting search quality.

These constraints ruled out many common approaches. We needed something that fit cleanly into CockroachDB’s execution model and harnessed the power of its distributed architecture. That’s where C-SPANN comes in.

Introducing C-SPANN

C-SPANN, short for CockroachDB SPANN, is a vector indexing algorithm that incorporates ideas from Microsoft’s SPANN and SPFresh papers, as well as Google’s ScaNN project.

At the core of C-SPANN is a hierarchical K-means tree. Vectors are grouped into partitions based on similarity, with each partition containing anywhere from dozens to hundreds of vectors. Each partition has a centroid, which is the average of the vectors it contains, representing their “center of mass”. Those centroids are recursively clustered into higher-level partitions, forming a tree that efficiently narrows the search space.

Each partition is stored as a self-contained unit in CockroachDB’s key-value layer, making the index naturally sharding-compatible. Partition data is laid out as a contiguous set of key-value rows within a CockroachDB range. As partitions are added, removed, or grow in size, the underlying ranges can be automatically split, merged, and rebalanced by the database, just like any other table data.

At query time, the search starts at the root of the tree. We compare the query vector to the centroids at that level, then descend into the partitions with the closest matches. This process repeats at each level until we reach the leaves, where we scan a small number of candidate vectors. Partitions at each level can be processed in parallel, helping to reduce latency. And because vectors within a partition are packed together and are similar by design, we can take advantage of SIMD CPU instructions to efficiently scan blocks of vectors.

Because the tree fanout is typically around 100, the structure remains wide and shallow. This keeps the number of levels (and therefore the number of network round-trips) small and predictable, even at large scale. An index with 1 million vectors requires just 3 levels; even one with 10 billion vectors needs only 5 levels. To reduce round-trips even further, the root partition can be cached in memory.

C-SPANN also avoids central coordination. Any node can serve queries or handle inserts and updates. The index structure lives in persistent storage, so there’s no need for large in-memory vector caches or custom data structures that must be rebuilt at startup. Instead, partition rows are cached automatically by the storage layer’s block cache, just like any other table data. This allows searches to avoid repeated disk reads, without requiring extra RAM or specialized vector caching logic.

Check out this demo by technical evangelist, Rob Reid, to see vector indexing in action:

Maintaining a Healthy Index

As new vectors are inserted into the index, they naturally scatter across partitions, which are themselves distributed across the cluster. There’s no single range or node that absorbs a disproportionate share of the write traffic, which helps to prevent hot spots from forming. But over time, some partitions will grow too large and need to be split.

Splits happen automatically in the background to reduce impact on foreground transactions. When a split is triggered, the vectors in the original partition are divided into two roughly equal groups using a balanced variant of the K-means algorithm. Each group becomes a new, more tightly clustered partition with its own centroid. The tree is updated to reflect this change, and future inserts are routed to the new partitions based on proximity to these new centroids. Here’s an example where partition 4 is replaced by partitions 5 and 6 at the leaf level of the tree:

It’s also worth noting that partition splits are distinct from CockroachDB’s range splits, though the two work together to ensure scalability and consistent performance. A partition is a logical unit within the index that groups similar vectors. A range is a physical unit of storage in the key-value layer. Splitting a partition improves search efficiency by maintaining tight clustering of vectors. Splitting a range helps balance data storage and access across the cluster. Together, these mechanisms reduce hot spots and help spread both query and insert load more evenly. When nodes are added to the system, ranges containing index partitions are automatically distributed across the new nodes, allowing the total workload to scale out with the cluster at near-linear rates.

There’s one wrinkle worth noting: some vectors may no longer be in the “right” partition after a split. A vector in the splitting partition might be closer to a nearby partition’s centroid than to either of the new centroids. Likewise, a vector in a nearby partition might now be closer to one of the new centroids. In both cases, vectors need to be relocated to the partition with the closest centroid. To see how this can happen, consider these red and blue clusters (centroids are marked with X):

After the blue cluster is split, one of its vectors is reassigned to the red cluster because it’s now closer to the red centroid than to either of the new blue centroids. Similarly, one of the red vectors is reassigned to the righthand blue cluster for the same reason. Relocating vectors based on updated proximity is introduced in the SPFresh paper (as part of ensuring “nearest partition assignment”) and plays a key role in maintaining high clustering accuracy after splits.

While splits ensure that partitions don’t grow too large, merges ensure they don’t shrink too small. If vectors are deleted or moved such that a partition falls below the minimum size, a background process merges it away. Its vectors are reassigned to nearby partitions, and the original partition is removed from the tree.

Taken together, splits, merges, and partition reassignments are highly effective at preserving index accuracy, even after many cycles of vector inserts, updates, and deletes. In fact, the approach works so well that there's not a lot to gain from rebuilding the index after adding new data. You can start with an empty table, insert millions of vectors, and still get high accuracy. The index adapts rapidly and dynamically as the data evolves, keeping itself balanced and efficient over time.

Reducing Index Size by 94%

Full-precision vectors are expensive. OpenAI embeddings, for example, use 1,536 dimensions with 2-byte floats, which comes out to about 3 KB per vector. Multiply that by millions of vectors, and the index size adds up quickly. Storage is one cost, but often the greater expense is the CPU and memory required to load and scan full vectors during index searches.

To reduce this overhead, C-SPANN uses a technique called quantization to compress the vectors stored in the index. Instead of storing full vectors, it stores compact binary representations that approximate the originals. During search, distances are computed using these quantized forms, which are both smaller and faster to scan.

While many quantization algorithms exist, we use one called RaBitQ, which reduces each vector dimension to a single bit. It stores those bits along with a few precomputed values per vector, achieving roughly a 94% reduction in size for common cases. In the OpenAI embedding example, that shrinks a vector from about 3 KB to only around 200 bytes.

This approach integrates naturally with the K-means tree: each vector is quantized relative to the centroid of the partition it belongs to, allowing for tighter grouping and better accuracy. Because quantization is local to each partition, splits and merges only require re-quantizing the vectors within the affected partition. This enables the index to evolve incrementally and locally, without centralized coordination or global retraining.

While I won’t dive into every detail, I want to show you how beautiful and simple the core RaBitQ algorithm is. Each data vector is first “mixed” with a random orthogonal transform, which spreads any data skew more evenly across dimensions while still preserving angles and distances. It’s then mean-centered with respect to the partition centroid and normalized to unit length. Finally, each dimension is converted to a bit: zero if the value is less than zero, one otherwise.

The result is a string of bits that captures the essence of the original vector in highly compressed form. These bits are stored alongside the dot product between the quantized and original vectors, as well as the exact distance of the original vector from the centroid. Remarkably, that’s enough to estimate distances with reasonable accuracy. To make distance comparisons fast as well as compact, RaBitQ uses a different quantization method for the query vector that assigns 4 bits per dimension and is optimized for SIMD instructions.

Since quantization is lossy, these distance estimates are only approximate. To correct for this, C-SPANN includes a reranking step. We scan quantized vectors to build a candidate set, then fetch the original full vectors from the table to re-compute exact distances. By over-fetching candidate vectors, we can compensate for quantization error. RaBitQ provides error bounds that help determine how many extra vectors are needed to find the true nearest neighbors with high probability.

The result is the best of both worlds: fast, compact scans with accurate results.

An Index for Every User

I’ve explained how C-SPANN can cluster vast numbers of vectors and keep the index fresh with real-time, incremental updates. But there’s a further twist to the story. In most real-world applications, those vectors belong to someone, whether it be a user, a customer, a tenant, or some other owner. And most queries are scoped to just that one owner. In fact, including vectors from other owners could be a security issue.

CockroachDB vector indexes handle this cleanly by supporting prefix columns, which allow the index to be partitioned by ownership (or anything else). Here’s a simple example:

CREATE TABLE photos (
  id UUID PRIMARY KEY,
  user_id UUID,
  embedding VECTOR(1536),
  VECTOR INDEX (user_id, embedding)
);

In this case, the vector index is partitioned by the leading user_id column. That means photo embeddings are indexed and searched per user. Here’s a query that finds the 10 closest photos for a given user, using pgvector compatible syntax:

SELECT id
FROM photos
WHERE user_id = $1
ORDER BY embedding <-> $2
LIMIT 10

Even if the index contains billions of photos, this query will only search the subset that belongs to one user. Performance for inserts and searches is proportional to the number of vectors owned by that user, not the total number of vectors in the system. Contention between users is minimized, since queries don’t touch the same index partitions or rows.

Behind the scenes, the index maintains a separate K-means tree for each distinct user. From the system’s perspective, there isn’t much difference between 1 billion vectors arranged in a single tree or the same number spread across a million smaller trees. Vectors are still assigned to partitions and packed into ranges in the CockroachDB key-value layer. Those ranges are automatically split, merged, and distributed across nodes, just like any other data, enabling near-linear scaling as usage grows.

Users in Every Region

Prefix columns become even more powerful when used with CockroachDB’s multi-region features. For example, you can use a REGIONAL BY ROW table to store each user's data in their home region, which reduces latency and helps meet data domiciling requirements:

This statement automatically adds a crdb_region column to the table, which is included in the vector index alongside user_id and embedding. This ensures that both table and index rows are co-located in the region specified by each row’s crdb_region value. Photos for a user in Europe will be stored in Europe, with fast, local access from that region. Photos for a user in the US will be stored in the US, with equally low-latency access there. The combination of crdb_region and user_id as prefix columns partitions the index by both location and ownership, making it efficient, secure, and locality-aware by default.

Future Work

We released a preview of vector indexing in our 25.2 release. It includes all of the core functionality described above, but several optimizations are still underway to improve accuracy and performance. Merge operations and partition reassignments are not yet fully implemented. Root partition caching and expanded use of SIMD instructions will help speed up both searches and inserts. We're also working to minimize contention between foreground queries and background operations like splits and merges.

On the functionality side, we plan to expand support for IMPORT, ALTER INDEX and additional distance metrics. The current implementation supports Euclidean distance, and we’re actively working to add support for cosine and inner product as well. And while the current implementation can leverage the vector index when filtering on prefix columns, we’re aiming to support a wider range of WHERE clause filter patterns in future releases.

If you're working with a large number of vectors, need to support high QPS, fresh results, geo-locality, high-availability, or anything else discussed here, we'd love to hear from you. Your use case could help shape our future roadmap.

Try CockroachDB Today

Spin up your first CockroachDB Cloud cluster in minutes. Start with $400 in free credits. Or get a free 30-day trial of CockroachDB Enterprise on self-hosted environments.

Tutorial: Augment your AI use case with RAG on CockroachDB

With the rise of AI across industries, it’s becoming more important to address the limitations of LLMs. One increasingly popular method is Retrieval Augmented Generation (RAG), which allows you to provide external knowledge sources as context to Large Language Models (LLMs).

In this blog post, we dive into the significance and benefits of RAG applications first. Then we explore the advantages of leveraging CockroachDB, a highly scalable and resilient distributed SQL database, as the foundation for building efficient and robust RAG applications. To illustrate the practical implementation, I’ve created a knowledge base chatbot. Readers will experience a clear walkthrough of the application's operation via an interactive demo, gaining insights into how CockroachDB vector search seamlessly integrates with LLM APIs to ensure real-time, contextualized responses. Finally, we'll summarize key takeaways and highlight why CockroachDB provides an ideal backend for modern RAG-powered applications.

What is Retrieval Augmented Generation (RAG)? How does RAG improve LLMs?

Large Language Models (LLMs) bring tremendous knowledge and creativity, enabling groundbreaking capabilities in generating human-like text and automating numerous tasks. However, despite being trained on massive datasets, these models often exhibit limitations when dealing with specific domain knowledge or real-time personalized transactional information. As seen online, LLMs can generate inaccuracies or completely fabricated information—a phenomenon known as “hallucinations.” Hallucinations occur when an LLM generates plausible-sounding but factually incorrect or entirely fabricated statements.

Retrieval Augmented Generation (RAG) addresses this critical challenge by introducing grounded data, meaning responses are directly supported by accurate and relevant information. Instead of solely relying on the model’s pre-trained knowledge, RAG first queries a specialized database containing domain-specific knowledge to identify the most relevant and update-to-date information. This retrieved context is then fed back into the LLM, enabling the generation of precise, trustworthy answers based on verified data rather than ungrounded predictions.

Retrieval Augmented Generation (RAG) Use Cases

There is a lot of potential for RAG in a variety of industries. Here are just a few:

For example businesses utilize RAG applications to enhance knowledge base queries, allowing employees to rapidly access relevant, domain-specific insights, which boosts productivity and supports informed decision-making.
RAG facilitates real-time information retrieval within enterprise applications, enabling immediate access to critical, up-to-date information, which is essential for operations like risk assessment, financial analytics, and strategic planning.
In customer support scenarios, RAG empowers chatbots to deliver accurate, context-aware answers, substantially enhancing user satisfaction and reducing response times by grounding the LLM in a database of customer support documentation.
For content generation tasks, such as marketing copy or technical documentation, RAG leverages verified sources to ensure the content is factual, consistent, and aligns closely with organizational guidelines.

Why Build a RAG Application on CockroachDB

Unified Storage of Source Data, Metadata, and Vector Data

CockroachDB enables the storage of source data, metadata, and vector embeddings within the same database, significantly simplifying data management, lowering cost, and improving performance for RAG applications. This unified storage approach allows vectors and their corresponding source data to be updated simultaneously, providing immediate, real-time access to the latest information without the delays associated with multi-database architectures.

When vector data and source data reside in separate systems it requires additional pipelines and frequent synchronization efforts. Such systems often suffer from latency and inconsistency, leading to outdated vector indexes, which increases the risk of retrieving and serving stale information. CockroachDB addresses this fundamental limitation by streamlining the data management processes, thus delivering accurate, timely responses in RAG-enabled applications.

Scalability, Reliability, and Simplicity of Operations

CockroachDB is purpose-built to run business-critical applications at scale, providing robust scalability and unmatched resilience, which makes it ideal for powering RAG systems that handle intensive workloads and large data volumes. Trusted by numerous enterprises large and small, CockroachDB's distributed architecture ensures continuous availability by automatically self-healing from hardware failures or network issues, substantially reducing maintenance overhead and operational complexity.

Measure what matters

Traditional benchmarks only test when everything’s perfect. But you need to know what happens when everything fails. CockroachDB's new benchmark, "Performance under Adversity," tests real-world scenarios: network partitions, regional outages, disk stalls, and so much more.

CockroachDB is PostgreSQL-wire compatible, and our VECTOR implementation is compatible with the pgvector extension, making it easier for engineering teams to integrate with existing systems. But by leveraging CockroachDB for vector storage and retrieval, developers avoid the scalability limitations associated with pgvector/Postgres. With CockroachDB’s vector capabilities, you can scale your RAG applications while maintaining consistent performance under increasing load.

Additionally, CockroachDB’s consistency model ensures outdated vectors will not be queried, further enhancing the accuracy of retrieval results. CockroachDB’s Helm chart & Kubernetes Operator seamlessly handles large-scale deployments, making it an ideal backend for powering efficient and reliable vector search functionalities.

Advanced Access Control and Security

As powerful Large Language Models (LLMs) become central to enterprise applications, security and access control become paramount considerations. CockroachDB addresses these concerns comprehensively: Role-based access controls (RBAC) and Row-Level Security, empowering organizations to provide fine-grained user permissions. This ensures that users access only the data they're authorized to view, preventing unauthorized disclosures and enhancing overall data governance. Additionally, CockroachDB has native geo-data placement capabilities to enable data governance and improve performance by keeping data close to the customers that need it. Such precise control mechanisms set CockroachDB apart in securely supporting knowledge bases integrated with advanced generative AI capabilities.

How to build a knowledge base chatbot with RAG on CockroachDB

Building a knowledge based application using RAG on CockroachDB can significantly streamline how enterprises leverage their private knowledge bases, greatly improving productivity across various internal workflows. This chatbot leverages a general-purpose Large Language Model (LLM), significantly reducing the cost and complexity associated with training a domain-specific model, while still maintaining accurate, relevant, and timely responses.

Data flow

As illustrated in the above flowchart, in this knowledge base chatbot scenario, a user initiates interaction by asking a question. This query is first processed by an embedding model, which vectorizes the question, transforming it into a numerical representation. The vectorized query is then passed to CockroachDB, which houses both the vector embeddings and the source data. Using similarity search capabilities, CockroachDB quickly identifies and retrieves the most relevant content sources.

Once relevant sources are retrieved, they serve as grounded context alongside the original user query. The grounding step ensures the LLM generates context-aware and facts-based responses. This combination reduces the risk of hallucination.

Application architecture

The Retrieval Augmented Generation (RAG) application architecture depicted in the diagram consists of three primary components: a frontend chatbot UI, a FastAPI-based REST API service calling Cockroach Cloud, and an LLM API. The user interacts with the chatbot through the frontend interface, submitting questions or queries. These user inputs are then sent to the REST API service, which orchestrates data processing by communicating with Cockroach Cloud to perform similarity searches and retrieve relevant, grounded information. Next the REST API leverages the LLM API to vectorize the user's query, use contextual prompts to produce accurate, contextually enriched responses. This design enables efficient data management, real-time retrieval, and seamless integration between the database and the generative capabilities of LLMs, providing users with timely, accurate, and reliable information.

Check out a full demo of the chatbot at our YouTube channel:

RAG and CockroachDB: Better Together

RAG enhances Large Language Models (LLMs) by grounding their responses in verified, domain-specific data, thus improving accuracy and reliability, which is often required in many applications, including customer support, content generation, knowledge base queries, and real-time information retrieval.

CockroachDB is an ideal foundation for RAG applications, given its ability to store source data, metadata, and vector data all in the same place. By bringing vector capabilities into a distributed SQL database, you can easily implement AI use cases, simplify database management, and gain the scalability, resilience, and familiarity of a proven distributed SQL database.

To learn more about how you can leverage CockroachDB for your AI applications, contact us! We’d love to hear from you.

Try CockroachDB Today

Spin up your first CockroachDB Cloud cluster in minutes. Start with $400 in free credits. Or get a free 30-day trial of CockroachDB Enterprise on self-hosted environments.

Ideal isn’t real: Stress testing CockroachDB’s resilience

In today’s always-on, global applications, raw speed is no longer enough. Modern enterprises demand not only blistering transaction rates but also unwavering stability when the unexpected strikes; whether that’s a network hiccup, a failing disk, or an entire data center going dark. With organizations averaging 86 outages a year, failures are the norm today. But traditional benchmarks like TPC-C only tell you how fast a database runs in perfect conditions, or sunny days. They leave out the more important story of what happens when things go wrong.

As announced by Cockroach Labs, our new database benchmarking methodology, “Performance under Adversity” captures these complex, operational realities of modern enterprise systems: where everything fails all the time. The benchmark is a single, continuous test that weaves seven increasingly severe failure scenarios into a standard OLTP workload. By measuring throughput, latency, and failure rates through rolling upgrades, disk stalls, network partitions, node restarts, and even full zone or regional outages, our benchmark reveals how well CockroachDB, a distributed SQL database, maintains service continuity when it matters most.

In this blog post, we’ll dive into what makes CockroachDB uniquely resilient under pressure. We will examine its baseline performance, its behavior under internal operational stress, and its ability to self-heal through grey failures and large-scale outages. We explain exactly how CockroachDB stays online and keeps data safe even beyond perfect, sunny days and clear operational skies.

Measure what matters

The Seven Phases of Adversity

Before exploring how CockroachDB tackles the operational realities of modern enterprise systems, let’s first outline the behaviors you should expect from a resilient database as it endures each phase of real-world adversity.

What Keeps CockroachDB Resilient and Stable Under Pressure?

To understand how CockroachDB handles the pressure, let us look at the results of the benchmark in the interactive dashboard and then analyze the result of each phase.

Baseline Performance: Normal conditions only running TPC-C workload with no internal or external stressors (a perfect, “sunny day”)

It is important to note that tpmC measures transactions per minute for TPC-C and each transaction is ~14 queries. So the corresponding SQL queries per seconds (QPS) is tpmC multiplied by a factor of 14/60. The largest transaction tested is the "New Order" transaction, which consists of up to 14 SQL statements involving 8 database tables to complete an order within a transaction. This workload is both read- and write-intensive, designed with high execution frequency and strict response-time requirements to simulate real-world online database activity.

Internal Operational Stress: In this phase, routine maintenance tasks are executed, including backups, change feeds, online schema changes, and rolling upgrades.

During backups, change-data capture (CDC), and index builds (online schema change), the database must scan or rewrite tables. These operations consume significant CPU and I/O bandwidth.

Likewise, during rolling upgrades, for each node being upgraded, CockroachDB drains SQL connections on the node, transfers data leases from the node to the other nodes and then restarts the node with the upgraded binary. Any application connected to the node will experience a SQL connection drop when the node restarts; and the application may try to reconnect, generating a burst of new connections. Upon rejoining, the restarted node receives snapshots of data that changed while it was down from other nodes. The lease transfers, connection burst, and snapshot streaming to restarted nodes generates heavy CPU and IOPS load.

Without careful throttling and scheduling, large background scans, large connection bursts, and snapshot streaming will inevitably siphon CPU, I/O, and network capacity from your primary workload. In turn driving up latency, eroding throughput, and risking overall cluster health.

As can be seen on the dashboard during this phase on CockroachDB, tpmC remained unaffected during internal operational stress with minimal latency impact, and CockroachDB maintained stability.

CockroachDB avoids the aforementioned pitfalls with its built-in admission control layer, which dynamically prioritizes client-facing traffic over background operations. Not only that, admission control throttles access to a resource (CPU or IO) before the resource becomes overloaded, rather than after. The result is rock-solid stability, consistently low latencies, and minimal throughput impact even when backups, CDC streams, index builds, and rolling upgrades run concurrently.

^{Image: 15-node CockroachDB cluster under internal operational stress}

Next, let us understand how CockroachDB handles external stressors that are not controlled by the database administrator.

^{Image: 15-node CockroachDB cluster withstanding external stressors (disk stalls, network partitions, node restarts, and facility and region outage)}

Disk Stalls: Transient disk stalls are a frequent reality in cloud environments whether on AWS EBS, Azure SSD, or GCP PD occurring as often as 20–30 times per hour in a single cluster. During a stall, write-ahead log (WAL) writes block until the disk recovers, often leading to transaction timeouts, latency spikes, and elusive “grey” failures.

As can be seen on the dashboard, CockroachDB handles disk stalls with minimal latency impact and stable throughput.

CockroachDB turns disk stalls into a non-event by leveraging multiple disks per node: if one disk stalls, WAL writes automatically fail over to an alternate disk within 100ms. The result is seamless resilience, with steady latency without blips and no noticeable drop in performance or throughput.

Network Failures: Network failures can result in network partitions, which isolate one or more nodes from the rest of the cluster. These issues can arise from hardware or software failures, traffic congestion, or misconfiguration and can last anywhere from seconds to minutes (or, in rare cases, even longer). When partitions occur, they can result in read-errors, write-errors, latency spikes, and potentially stale or inconsistent data. Such gray failures are notoriously hard to diagnose because they’re often transient and shifting.

As evident on the dashboard, which shows a minor latency spike for a brief duration and minimal impact on throughput, CockroachDB is partition-tolerant.

We ran these tests using CockroachDB v25.1. It survives network partitions with epoch leases, which fence out stale masters and prevent a split‐brain scenario. During a partition, only the majority side (with more than half of ranges) can renew or re‐elect leases and the minority side leases stales out. Any future operations by stale leaseholders are rejected and the surviving leader continues serving traffic seamlessly, preserving consistency and availability, and minimizing impact on throughput. Once the network recovers from partition, all the nodes start participating in quorum.

Starting with v25.2, CockroachDB survives network partitions thanks to its leader-leases architecture: every Raft leader also holds the range’s lease, ensuring a single, unequivocal authority for both reads and writes. By unifying leadership and leaseholding in a single role, it eliminates single points of failure and split-brain risks. So even when node communication is severed, the surviving leader continues serving traffic seamlessly, preserving consistency and availability; and minimizing impact on throughput.

Node Restarts: Similar to rolling upgrades, a node restart also triggers lease transfer from the restarted node to other nodes across the clusters. If the node takes longer than 5 minutes to restart, it also triggers range rebalancing. Additionally any applications connected to the node will experience SQL connection drops, and will try to reconnect generating a burst of new connections. These are random and ungraceful reboots, and the resulting lease transfers, connection burst, and snapshot streaming generate heavy storage engine CPU load, IOPS and network traffic. In turn impacting latency and throughput by taking away resources from foreground traffic. However CockrochDB’s Admission Control handles this by prioritizing foreground traffic over rebalancing activities while maintaining stability, and resulting in only minor latency increases and negligible throughput impacts.

In the benchmarking run where three nodes were restarted randomly, the dashboard shows a brief, minor latency spike and negligible impact on throughput of the CockroachDB cluster.

Zone Outages: Zone outages can occur when the data center hosting the zone experiences power or network outage. When this occurs, data replicas in the impacted zone need to move over to the remaining zones to ensure full cluster availability. This data movement is done by streaming range data among the nodes in the remaining zones, thus balancing the range distribution in the cluster.

Range rebalancing across multiple nodes generates heavy storage engine CPU load, IOPS and network traffic. Unchecked, the range rebalancing can overload these resources and starve foreground queries of resources resulting in negative impact on latency and throughput and in extreme cases destabilizing the cluster altogether. However, CockroachDB, as shown on the dashboard, only experienced a very brief, minor latency spike and virtually no change in throughput during the zone outage.

Overall availability and stability of the CockroachDB cluster throughout the zone outage is thanks to the automatic redirection of traffic from failed zones to available zones and especially how CockroachDB’s Admission Control handles foreground traffic and background range rebalancing traffic.

Regional Outages: Enterprises with presence in multiple geographies require that users in all locations have consistent experiences with latency and throughput. In addition, these enterprises must comply with data domiciling regulations. For such organizations, CockroachDB can operate in multiple regions by placing data closer to users in different geographic locations.

Similar to zone outages, regional outages trigger range rebalancing across the nodes of the remaining regions to maintain full cluster availability. As explained earlier, range rebalancing across multiple nodes can negatively impact foreground latency and throughput, and cluster stability. However this is not an issue in CockroachDB, as shown in the dashboard.

Again, similar to zone outage, CockroachDB’s admission control automatically redirects traffic to available regions and dynamically prioritizes client-facing traffic over background operations. Along with properly throttling access to resources before they become overloaded, results in stability with minimal latency and throughput impact.

One final observation to make is the consistency in CPU and disk usage throughout the entire benchmarking run.

You can check out a full demo of the interactive dashboard on our YouTube channel:

Resilience Is Essential, Not Optional

As you’ve seen, the new database benchmarking methodology, “Performance under Adversity,” pushes CockroachDB through seven increasingly severe failure scenarios and in every case CockroachDB delivers steady throughput, low latency, and zero downtime. That resilience comes from engineered features like back-pressured range rebalancing, admission control that favors client traffic, co-located leaseholders, and multi-disk WAL failover.

After all, it’s not enough to know how fast your database can go when the sun is shining and conditions are perfect. You need to know that your database will keep churning at scale, under load, and in the face of every curveball the real world can throw. With CockroachDB and our new benchmark, you get both performance and peace of mind even beyond the sunny days.

Explore the interactive dashboard to review phase-by-phase metrics, run your own tests, or spin up a free CockroachDB cluster today.

Try CockroachDB Today

Spin up your first CockroachDB Cloud cluster in minutes. Start with $400 in free credits. Or get a free 30-day trial of CockroachDB Enterprise on self-hosted environments.

Disclaimer: Results are from specific test setups. Try CockroachDB yourself to see how it fits your needs.

CockroachDB 25.2: Celebrating a decade of innovation with enhanced performance, vector indexing, and more

A few months ago Cockroach Labs turned 10, and this release celebrates 10 years of continuous distributed SQL innovation. We’re introducing even more performance and scalability improvements and AI vector indexing, along with many other new features and enhancements to both self-hosted and cloud offerings in the areas of change data capture (CDC), security, observability, and migrations.

CockroachDB v25.2 is the most performant, scalable, and secure version of CockroachDB yet. My favorite new feature highlights of this release include:

Performance improvements: 50% increased throughput and more

We’ve achieved 50% increased throughput as compared to 24.3 as measured on average across nine workloads. These performance improvements are the result of over 100 changes, both large and small, including new features like generic query plans (now in GA).

Buffered writes (now in Preview). Buffered writes is an under-the-covers way to improve performance by bundling up database writes more efficiently to increase throughput and decrease latency, while also reducing hardware requirements. Buffered writes performance improvements are not counted towards the 50% improvement discussed earlier because it’s still a preview feature and the default setting for buffered writes is ‘off’.

Buffered writes introduces a new step in the transaction flow, which temporarily stores transaction writes on the client side (gateway) until the transaction commits. This approach minimizes the number of roundtrips to the leaseholders, reduces pipeline stalls, and allows for the passive use of the 1-phase commit fast-path. By deferring writes until commit time, the system can reduce redundant writes and serve read-your-writes locally, resulting in performance gains that in our tests range from 15% (Sysbench oltp_read_write) to 40% (customer-specific testing) improvements.

Database restore enhancements. Improvements to the database restore capabilities have led to an increase in up to 4x the speed when restoring from backup for some restore use cases.

Vector indexing is in Preview for AI applications at scale

In 24.2 we released pgvector support for the vector datatype, which was a good way for customers to begin experimenting with advanced LLM applications. When we introduced pgvector support, we did so for vectors with thousands of dimensions. Now customers are moving their testing along to where performance is important.

To support performant search and discovery we are now introducing vector indexing in Preview based on a protocol we’re calling Cockroach-SPANN (or C-SPANN). C-SPANN is based on Microsoft’s SPANN and SPFRESH research papers which, along with RaBitQ, deliver an indexing method that is small, fully distributed, and easy to update without degrading index quality. We’re very excited about this release as it directly addresses scalability concerns of vector applications for customers with support for indexing billions of vectors. With vector indexing, we now make it even more compelling to combine the strengths of a vector database and an operational database onto a single, horizontally scalable and performant solution.

Introducing row-level security

Row-level security (released as GA) is a critical feature for large organizations in sectors such as healthcare and finance, where sensitive customer information must be meticulously protected. These organizations often have a diverse array of database users, including database administrators and application users, each requiring different levels of data access. CockroachDB provides robust mechanisms for managing roles and privileges to restrict access to specific tables, and now we are adding the capability to enforce row-level security. This means that both access to individual tables can be controlled and there is built-in functionality to restrict access to specific rows within those tables based on the user or role.

Row-level security ensures sensitive data is protected and access to the data is tightly controlled. With row-level security customers also now have better tenant isolation to protect their customer information within a single table without relying upon application level constraints.

Enhanced observability: We now enable organizations to gain more insights into their workloads to improve overall performance. Capabilities include generating work-load level index recommendations to improve performance, finer-grained SQL metrics, and comprehensive hotspot logs.

New CockroachDB Cloud regions and plan switching without downtime

New CockroachDB Cloud regions: There are two new GCP regions and two new AWS regions to support customer deployments with even richer region support to help keep your data as close to your client applications as possible. For the full list of regions supported, check out the docs.

CockroachDB Cloud plan switching: With plan switching, customers deploying CockroachDB as a managed service will now have complete flexibility in the plans they choose, and switching between them as their needs evolve. Plan switching enables customers to easily scale workloads up or down without added downtime based on application scale and security needs. This helps to avoid plan lock-in, and lets customers select their plan tiers based on evolving needs and lessons learned. Plan switching currently supports Basic to Standard (and the reverse), with Standard to Advanced (and the reverse) coming out soon.

To see the full list of features and improvements in v25.2 and our Cloud offering head over to the “what’s new” landing page.

Try CockroachDB Today

Spin up your first CockroachDB Cloud cluster in minutes. Start with $400 in free credits. Or get a free 30-day trial of CockroachDB Enterprise on self-hosted environments.

CockroachDB Redefines Database Performance with Real-World Testing

“Performance under Adversity” pressure testing proves most resilient database

For decades, database benchmarking has been stuck in the past. Industry-standard benchmarks like TPC-C – first approved 33 years ago, in 1992 – were designed to measure peak performance in pristine lab environments, isolated from the chaos of the real world. For example, they assume single datacenter deployments, ignoring the complexity of global applications and the global distribution of data. While they might include durability tests, they often do not explicitly measure performance during failures or maintenance, nor do they measure recovery time after failures. What they do show you is how a database behaves when nothing goes wrong. For example, once the steady-state run is disrupted (for example by a power pull) the test is considered over.

But nothing has ever stayed perfect in production.

Why we must re-evaluate database performance now

Modern applications are globally distributed, always-on, and increasingly complex. There are real world operations such as rolling upgrades, schema changes, and backups. There might be inconsistent storage access or complex network issues. While in the cloud, something fails all the time. Downtime, planned or unplanned, is no longer just a nuisance — it’s a multi-million dollar problem. Just ask CrowdStrike, FAA, Lloyds Banking Group, and the PlayStation Network, all of whom have suffered recent, very public outages.

In 2011, McKinsey predicted the era of big data, and since then we’ve seen an explosive growth in the amount of data we generate and how quickly that data is shared between both users and machines. The increasing use of AI and applications such as virtual agents, coupled with the increased complexity of interconnected systems only drives this growth higher. As a result, organizations are more vulnerable to disruptions such as cyberattacks, natural disasters, and outages from equipment failures.

Our “State of Resilience 2025” Report surveyed 1,000 senior technology leaders, and found that on average, organizations experience 86 outages each year, putting revenue and brand reputation at risk regularly. Even more telling: 79% of companies admit they aren’t ready to comply with emerging regulations surrounding operational resilience, like DORA and NIS2. Of the 95% of executives who acknowledged their operational vulnerability, almost half have yet to take action.

Today, organizations often rely on chaos engineering tools such as Chaos Monkey and Chaos Mesh to ensure applications are resilient in the face of changing, real-world conditions. However, most chaos engineering tools only target a single instance and do not simulate more complex failures such as loss of an entire availability zone (AZ) or region. They also only focus on a single point of failure and do not include impact to database performance. Knowing how to coordinate the use of these tools on a given infrastructure is critical to assess the impact of resilience on sustained performance.

So what if there was a database benchmark that could measure these resilience factors along with its impact to sustained performance?

Measuring what matters: Resilience, recovery, and sustained "Performance under Adversity"

For the last 10 years we have built CockroachDB with the core belief that performance isn’t about speed – it’s about survival. In fact, we believe resilience is the new performance metric.

That’s why we’ve released a novel methodology for benchmarking modern databases that tests the impact of real-world conditions faced by application and infrastructure teams every day. The shift from measuring peak performance allows us to measure how infrastructure performs under pressure.

Then, we show how CockroachDB responds to these conditions, preventing downtime, delivering business value and fast, consistent performance, and allowing teams to meet their overall TCO goals.

The metrics behind real-world testing

Database benchmarking for the real world interactive dashboard

Our new benchmarking methodology goes far beyond traditional metrics. It’s designed to answer the hard questions, like how your system behaves during rolling upgrades, node failures, or full data center outages.

The benchmark includes seven escalating phases of adversity in a single continuous run, each one introducing progressively harsher failure conditions and operational stress. The metrics provide insight on impact to resilience and sustained performance.

Baseline Performance: Measure steady-state throughput under normal conditions.
Internal Operational Stress: Simulate resource-intensive operations such as change data capture, full backups, schema changes, and rolling upgrades.
Disk Stalls: Randomly inject I/O freezes to evaluate storage resilience.
Network Failures: Simulate partial and full network failures preventing one partition from communicating with nodes in another partition.
Node Restarts: Unpredictably reboot database nodes (1 at a time) to test recovery time and impact to performance.
Zone Outages: Take down an entire Availability Zone.
Regional Outages: Take down an entire Region.

Through all these phases, we measure throughput in transactions per minute (tpmC), latency (90th and 95 percentile), and recovery time to baseline.

The benchmarking dashboard: See resilience in action

These results aren’t just published. We’ve built a live, interactive dashboard that lets you:

Explore performance trends: View graphs of throughput and latency over time.
Replay failure scenarios: Dive into node restarts, flaky storage, and regional outages.
See resilience in action: Track CockroachDB’s self-healing in real time.

Now you can see how CockroachDB actually performs under the conditions your team cares about most.

Future-proof at scale and with lower TCO

By combining resilience and performance into a single database benchmark and testing methodology, organizations can future-proof their database infrastructure at scale and without risking their operational data. Using CockroachDB, teams can deploy applications with confidence, because we are sharing the receipts that enable us to deliver consistent performance and state-of-the-art resilience. In turn this means lower TCO for organizations because any downtime negatively affects performance, time to market, and cost. That doesn’t include the broader impacts of loss in customer trust and satisfaction, and damage to brand and reputation.

Reproducing the benchmark on CockroachDB

Like any robust benchmark, reproducibility is critical. We’ve made it easy to run the tests yourself. Our benchmarking methodology is fully documented and open. For our testing, we used version 25.1 of CockroachDB, distributed across three geographically separate regions, and ran automated tests to simulate failures like network partitions, disk stalls, and node restarts.

For architects, operators, and data leaders, this changes the game. You’re no longer left guessing how your database will behave during chaos. With “Performance under Adversity,” you get proof.

Proof that your apps will stay online. Proof that you can scale without sharding. Proof that you can survive and thrive under pressure.

Join us in setting a new standard for database performance.

Disclaimer: Results are from specific test setups. Try the dashboard yourself to see how it fits your needs.

Measure what matters

How to Deploy CockroachDB on Red Hat OpenShift Virtualization

Rising VMware licensing costs and the uncertainty following the Broadcom acquisition are prompting many enterprises to explore alternatives for their virtualization platforms. Simultaneously, more organizations than ever want to adopt cloud-native architectures and operational models, even as they grapple with strict on-premises and hybrid requirements.

Red Hat OpenShift Virtualization stands out as a compelling alternative to VMware. It unifies traditional virtual machines (VMs) with container-based workloads under the same Kubernetes-driven platform—delivering cost efficiency, consistent operations, and open standards. Meanwhile, CockroachDB offers a distributed SQL database that is highly scalable, resilient, and truly cloud-native. Together, they provide a forward-looking solution to modernize infrastructure while still protecting mission-critical data and workflows.

This blog presents an in-depth technical blueprint for deploying CockroachDB on OpenShift Virtualization, referencing real-world validation tests, and context for our recently announced partnership with Red Hat. While most relevant to organizations in highly regulated industries and those seeking immediate VMware alternatives, the suggestions apply broadly to any enterprise requiring a cloud-native, future-proof virtualization strategy.

Why Consider OpenShift Virtualization as a VMware Alternative?

Since Broadcom’s acquisition of VMware, VMware has introduced new licensing models and cost structures that can be less than ideal, especially for environments still anchored on-premises. In addition, many enterprises running mission-critical workloads on VMware worry about renewed vendor lock-in, especially as they navigate modernization to containers and microservices. As an alternative to VMware, Red Hat’s OpenShift Virtualization offers a number of key advantages:

Unified Platform: Rather than managing virtualization separately from container orchestration, OpenShift Virtualization merges kernel-based virtual machines (KVMs) and Kubernetes containers into a single control plane, simplifying infrastructure and operational overhead.
Cost and Operational Efficiency: By consolidating onto a single platform, enterprises often reduce infrastructure sprawl, license expenses, and administrative complexity.
Hybrid Cloud-Readiness: Built on Red Hat OpenShift, clusters can span data centers, private clouds, and public clouds—facilitating a measured, step-wise migration away from purely on-premises environments.
Compliance and Control: For highly regulated industries (finance, healthcare, government), the ability to keep workloads on-premises under a standardized platform—while selectively extending to the public cloud—delivers both agility and compliance.
Future-Proof Architecture: With virtualization and containers managed together, organizations can steadily refactor or containerize VMs over time without abrupt migrations, mitigating risk during modernization.

Why CockroachDB for VMware-to-OpenShift Migrations?

CockroachDB is a cloud-native, distributed SQL database well-suited for hybrid and multi-cloud deployments. Its architecture is inherently designed to:

Scale Horizontally: New nodes can be added to the cluster as your needs grow—no complex re-sharding or forklift upgrades.
Resilience & Fault Tolerance: Built-in replication using the Raft consensus protocol ensures automatic failover with minimal operator intervention.
Strong Consistency: Retains ACID transactional guarantees across distributed nodes, vital for mission-critical systems.
Familiar SQL Interface: Postgres-compatible queries minimize developer retraining and reduce adoption friction.
Cloud-Native Capabilities: Automated rebalancing, rolling upgrades, and integration with orchestration tools like Kubernetes.

Benefits in an OpenShift Virtualization Context

Shared-Nothing Architecture: Each CockroachDB node runs independently with its own CPU, memory, and storage, aligning seamlessly with the VM-based approach in OpenShift Virtualization.
Operational Consistency: Whether on-premises or in the cloud, CockroachDB’s operational model stays consistent—fitting well into a hybrid environment managed by OpenShift.
Reduced Complexity in Migrations: CockroachDB’s built-in backup and restore features ease data transfer from VMware-based environments to OpenShift Virtualization VMs.

Validating CockroachDB on OpenShift Virtualization

While testing CockroachDB, I validated performance, availability, and deployment simplicity on OpenShift Virtualization. My own evaluations covered:

Seamless Installation: CockroachDB installs cleanly on Linux VMs provisioned by OpenShift Virtualization.
High Availability: Node failure tests confirmed automatic failover and rebalancing without downtime for read/write workloads.
Performance at Scale: Even under heavier TPC-C workloads (with 1,750 warehouses in testing), CockroachDB handled throughput without performance bottlenecks.
Easy Observability: Standard metrics endpoints integrate smoothly with Prometheus/Grafana or other enterprise monitoring tools.

These tests gave me confidence that CockroachDB is ready for deployments in OpenShift Virtualization environments, even for mission-critical apps.

CockroachDB on OpenShift Virtualization: Overall Architecture

OpenShift Virtualization extends the Red Hat OpenShift platform, allowing you to run both containers and virtual machines under a unified Kubernetes-based management plane. In the illustration below, we highlight not only the CockroachDB VMs but also the worker nodes, the KVM hypervisor layer, and essential components of the OpenShift ecosystem that orchestrate and manage these workloads.

Key Components

Red Hat OpenShift Platform

Provides the Kubernetes-based control plane for container orchestration, networking, and security.
Includes OpenShift Virtualization, enabling you to run KVM-based virtual machines alongside containers.

Worker Nodes (RHEL CoreOS)

Each worker node runs a KVM hypervisor and the virtualization components.
Labeled for scheduling to ensure each CockroachDB VM resides on a different physical node.

Dedicated CockroachDB VMs

Each VM (e.g., crdb-node-1, crdb-node-2, crdb-node-3) hosts a CockroachDB node.
Deployed on RHEL/CentOS or another supported OS. Leverages local or persistent storage volumes to maintain the shared-nothing architecture.

Networking Services

Headless Service (crdb-internal): Enables node discovery via stable DNS entries within the cluster.
LoadBalancer Service (crdb-loadbalancer): Facilitates external SQL traffic (port 26257) and UI/metrics (port 8080) to all CockroachDB nodes.

Storage Layer

Local SSDs or a supported Persistent Volume (PV) solution.
Must provide POSIX compliance and sufficient throughput.

OpenShift API / Management Plane

Coordinates all scheduling, networking, and lifecycle actions across the cluster.
Exposes a single interface for both container-based workloads and VMs.

By encapsulating the virtualization layer within OpenShift, enterprises unify both containerized and traditional VM workloads under consistent tooling. CockroachDB’s distributed nature fits neatly into this framework, ensuring strong fault tolerance and near-linear scalability even as you transition away from traditional VMware-based environments.

Architectural Summary CockroachDB on OpenShift Virtualization

Shared-Nothing Architecture in Action

CockroachDB’s shared-nothing model is fully realized in this deployment model:

Each VM is pinned to its own worker node (using labels and scheduling constraints).
CPU, memory, and SSD/local storage remain dedicated to each VM, eliminating noisy-neighbor issues commonly found in resource-shared deployments.
Network communications rely on standard Kubernetes services for node discovery and client ingress.

Core Components of CockroachDB

CockroachDB consists of a number of key components that allow us to provide the always-on availability, operational resilience, and performance at scale that our customers have come to know and depend on. A few are particularly important to call out when leveraging CockroachDB with OpenShift Virtualization

SQL Engine: Provides a powerful SQL interface compatible with PostgreSQL.
Transactional Key-Value Store: Uses the Raft protocol for replication and consensus-driven writes.
Data Distribution & Replication: Dynamically balances data ranges across nodes for optimal performance and fault tolerance.

Deployment Considerations

1. Network Configuration

Ports
- Port 26257 for SQL and inter-node traffic.
- Port 8080 for the DB Console and metrics.
Firewall: Ensure these ports are open on each VM to allow node-to-node communications.
Service Types
- Headless (no ClusterIP) for internal node discovery.
- LoadBalancer for external access by apps or DBAs.

2. Storage Requirements

Local SSD or Fast Persistent Storage: CockroachDB depends heavily on low-latency local storage for transaction logs and data files.
POSIX Compliance: Storage must be POSIX-compliant to ensure ACID guarantees under crash recovery, strongly recommend using SSDs.

3. High Availability & Resilience

Node Affinity: Label worker nodes to ensure each CockroachDB VM is scheduled separately, avoiding single-point-of-failure scenarios.
Automatic Failover: If a VM or node fails, CockroachDB automatically re-replicates data to remaining nodes.

4. Load Balancing

Round-Robin LB: The Kubernetes LoadBalancer typically distributes incoming SQL queries across all healthy nodes.
Internal Connectivity: The headless service ensures the CockroachDB processes use stable DNS names for Raft replication.

Summary of the Technical Deployment Steps

Below is a quick recap of the recommended approach:

Node Preparation
- Label three OpenShift worker nodes (e.g., node-role.kubernetes.io/crdb-node-1=true) so that each CockroachDB VM can be pinned to a distinct node.
- Create a dedicated namespace cockroachdb for resources.
Networking Setup
- Headless Service (crdb-internal): No clusterIP, used for internal DNS resolution among the CockroachDB nodes.
- LoadBalancer Service (crdb-loadbalancer): Exposes external connections to port 26257 (SQL) and 8080 (UI).
VM Provisioning
- Create three VMs via the OpenShift Virtualization console (e.g., crdb-node-1, crdb-node-2, crdb-node-3) using the CentOS 9 image.
- Allocate resources per CockroachDB production best practices (refer to the production checklist).
- Update firewall rules (if necessary) to open ports 26257 and 8080.
- Adjust SELinux settings if needed to ensure the CockroachDB processes can run and bind to required ports.
CockroachDB Installation
- Download CockroachDB binaries on each VM.
- Place the cockroach binary in /usr/local/bin and ensure the PATH is updated.
Cluster Initialization
- Start each node with --insecure or TLS flags as desired, specifying --join to the internal FQDNs for the other nodes.
- On one node, run cockroach init to bootstrap the cluster.
- Validate using cockroach sql --execute="SHOW DATABASES;" to confirm a healthy cluster.
External Connectivity
- Retrieve the LoadBalancer external IP (oc get svc crdb-loadbalancer -n cockroachdb).
- Connect from outside using cockroach sql --host=<LoadBalancer-IP>:26257.
Verification and Testing
- Run basic DDL, DML, and transaction operations to confirm cluster stability.
- Confirm high availability by stopping one node’s CockroachDB process; the cluster should remain writable.

Reliability and Performance Observations

1. Test Results at a Glance

No Installation Barriers: The CockroachDB nodes came online without errors.
Connectivity: Internal and external clients could connect via the configured services.
Basic SQL: DDL, DML, and transactional statements worked as expected with ACID guarantees.
High Availability: Node shutdown or partition tests showed zero downtime for read/write workloads; automatic rejoin upon node recovery.
Performance: With TPC-C–style workloads at moderate scale (1,750 warehouses), the cluster maintained stable throughput and latency even as I scaled the cluster by adding nodes and also introducing node failures/network partitions.

2. Monitoring

CockroachDB provides a DB Console (port 8080) for real-time cluster stats and integrates seamlessly with Prometheus for metrics collection. During testing, no issues arose in scraping or visualizing performance data.

3. Rolling Upgrades

Upgrading CockroachDB nodes individually can be done without cluster downtime. This is critical for production environments that demand continuous availability.

Migration from VMware to OpenShift Virtualization

1. Key Considerations

Risk Management: Minimal disruption is paramount for mission-critical apps. CockroachDB’s automated data replication and zero-downtime approach to adding/removing nodes reduce risk during migrations.
Phased Approach: Migrate certain workloads or lines-of-business first, then gradually decommission VMware clusters.
Backup & Restore: CockroachDB’s built-in backup tools simplify data transfers from existing databases.
Kubernetes familiarity: Teams familiar with VMware need training in Kubernetes-based virtualization concepts. Early pilots help build internal knowledge.

2. Practical Steps

Set Up OpenShift Virtualization: Create an OpenShift environment with worker nodes sized appropriately for the new VM-based cluster.
Network & Storage Alignment: Ensure that your target environment can provide the same or better performance characteristics than VMware (especially regarding storage IOPS and network throughput).
Deploy CockroachDB: Follow the steps outlined above or the official deployment guide.
Data Migration: Use CockroachDB’s IMPORT, BACKUP/RESTORE, or third-party ETL tools to seed data from legacy systems.
Validation & Cutover: Conduct acceptance tests against the new CockroachDB cluster. Transition application connections to the newly deployed environment once validated.

Key Takeaways

Cost-Effective and Future-Proof: Moving away from VMware to OpenShift Virtualization provides significant cost and operational advantages.
Cloud-Native Database: CockroachDB complements this modernization by offering a highly resilient, horizontally scalable SQL platform that fits both on-premises and across multiple clouds.
Validated Architecture: My testing was able to give me confidence and confirm that CockroachDB runs reliably on OpenShift Virtualization with minimal operational friction and feels just as similar to deploying CockroachDB on cloud VMs.
Streamlined Migration: Enterprises can migrate mission-critical workloads at their own pace, leveraging proven backup/restore flows and phased cutovers.

As new complexities and uncertainties arise, organizations seeking flexibility, performance, and cost savings will find a robust alternative in Red Hat OpenShift Virtualization, powered by CockroachDB’s distributed SQL capabilities. This approach not only addresses immediate licensing and lock-in concerns, but also paves the way for hybrid and multi-cloud strategies that are increasingly essential in modern IT landscapes.

Try CockroachDB Today

Spin up your first CockroachDB Cloud cluster in minutes. Start with $400 in free credits. Or get a free 30-day trial of CockroachDB Enterprise on self-hosted environments.

MOLT Verify: Ensuring Data Integrity in Database Migrations

Database migrations are complex, and ensuring the migrated data is 100% correct is critical. After all, a migration isn’t over until you feel confident and comfortable cutting over to the new infrastructure. To streamline migrations, Cockroach Labs created the MOLT (Migrate Off Legacy Technology) toolkit, which includes three tools that improve different parts of the user journey.

Ultimately, we built MOLT Verify to give engineers confidence when moving data from legacy databases to CockroachDB. In this deep dive, we’ll explore what MOLT Verify is, how it works under the hood, and how to use it effectively for seamless migrations. We’ll cover its role in the migration process, the architecture and integration with databases, how it verifies data consistency, and best practices for engineers.

To learn more about the rest of the MOLT tools, check out our blog on MOLT Convert (previously MOLT SCT), and our colleagues’ post on MOLT Fetch.

What is MOLT Verify?

MOLT Verify is a data validation tool in Cockroach Labs’ migration toolkit. Its primary role is to compare a source database (e.g. PostgreSQL or MySQL) against a CockroachDB target to ensure that everything matches after migration. MOLT Verify automatically compares the source and target databases at multiple levels—schema and row-level data validation. This prevents the risk of silent corruption, missing records, or mismatches, and offers an automated, scalable, and repeatable way to validate database migrations in minutes instead of days.

The tool performs a series of checks so you feel confident that your new CockroachDB cluster faithfully mirrors the original database:

Table verification: It confirms that each table in the source has a corresponding table in the target with the same schema definition (table name, number of columns, etc.)
Column definition verification: For each table, it validates that column names, data types, constraints, nullability, and other attributes match between source and target. This catches any schema drift or type conversion issues from the migration.
Row data verification: It compares the actual data in each table, row-by-row, to ensure every value in the source appears identically in CockroachDB.

Supported databases

MOLT Verify currently supports verifying data from PostgreSQL, Oracle, and MySQL sources (as well as CockroachDB itself) against a CockroachDB cluster – covering the most common migration paths. It’s provided as a standalone CLI tool (with downloadable binaries or a Docker image) that you run during migration.

Key Benefits of MOLT Verify

MOLT Verify acts as a safety net in the migration workflow, dramatically reduce the risk of data corruption or loss when migrating off legacy systems. After you’ve loaded data into CockroachDB (whether via bulk import or a live replication tool), running MOLT Verify lets you catch any missing records, mismatched values, or schema inconsistencies before you cut over your application.

In a typical migration plan, you would use MOLT Verify as a final validation step: for example, after all data is replicated to CockroachDB, run MOLT Verify to validate consistency, then proceed with the cutover once everything matches.

In particular, we recognized that large-scale migrations are uniquely challenging with regard to validation due to the sheer volume of data, and manual validation is virtually impossible in such scenarios. MOLT Verify, with its automated verification layer, checks every record in every row in every table, ensuring that the migration is accurate before production cutover.

The benefits of MOLT Verify include:

Cutover without fear of losing or corrupting data
Faster automated data validation checks
Validation beyond row counts: schema consistency, data integrity, row-level discrepancies
Designed for large-scale distributed migrations, validate billions of records systematically and efficiently

Architecture: How MOLT Verify Integrates with CockroachDB and Source Databases

Under the hood, MOLT Verify is built as a standalone tool that connects to both the source and target databases and orchestrates a comparison between them. This design means it can work with heterogeneous systems and will impose minimal load on the source and target clusters, beyond standard SQL reads. Ultimately, how you choose to handle concurrency will affect the load placed on your clusters.

Key points about MOLT Verify’s architecture and integration:

Standard Database Connections: MOLT Verify uses native connection protocols for each database. For example, it connects to CockroachDB using the PostgreSQL wire protocol (since Cockroach speaks Postgres-compatible SQL), and connects to MySQL through the MySQL protocol. This means it doesn’t require any special plugins on the databases; if you can connect via a SQL client, MOLT Verify can connect too.
Multi-Database Support via Abstraction: The tool is designed with an abstraction layer to handle different database flavors. It knows how to retrieve metadata and data from each supported source. It treats the source as the “truth” and CockroachDB as the destination to verify.
Schema Mapping and Assumptions: Because different databases have different ways to organize schemas, MOLT Verify makes some assumptions to simplify the comparison. For example, when comparing MySQL to CockroachDB, it assumes you’re comparing one MySQL database schema to a CockroachDB database’s public schema. MySQL’s notion of a database is analogous to a schema or namespace in Postgres/Cockroach. So if you have multiple schemas or databases to migrate, you would run MOLT Verify separately for each logical schema. This one-to-one mapping ensures the tool knows which tables to line up between source and target.
External Binary with CockroachDB Integration: Cockroach Labs distributes MOLT Verify as part of the MOLT CLI. The CLI package includes a molt command (and a helper binary called replicator used for live replication). You can download it for Linux, Mac, or Windows, or run it via a Docker container.
Parallel and Distributed Processing: Architecturally, MOLT Verify employs a multi-threaded design to handle large schemas efficiently. This is critical for integration with CockroachDB, which is a distributed database – MOLT Verify can generate multiple concurrent read requests that CockroachDB will handle on its distributed nodes, thereby speeding up the scan of large tables. Similarly, for a source like Postgres or MySQL, multiple threads can pull data from different tables (or different parts of the same table) concurrently, thus reducing overall runtime.

In summary, the design allows MOLT Verify to slot into migration workflows without requiring special infrastructure – if you can run the binary and connect to your databases, you can use MOLT Verify.

Verification Mechanism: How MOLT Verify Ensures Data Consistency

MOLT Verify’s core responsibility is to ensure data consistency and correctness during migration. It does so by performing a thorough comparison at multiple levels of the database, flagging any discrepancies. Let’s break down the verification mechanism in detail.

Schema and Table Verification

The first thing MOLT Verify checks is that the schema has been migrated correctly. It retrieves the list of tables from the source and the target, and verifies that every expected table exists on CockroachDB with the same schema structure. This includes comparing table names and primary keys, and then drilling down into columns. For each table, it checks all columns to ensure the column names, data types, and attributes (NULL/defaults, constraints, etc.) match between source and target. If, for example, a column in MySQL was an INT but the target schema accidentally used a STRING, or if a NOT NULL constraint was dropped, MOLT Verify will catch those discrepancies.

Row Data Verification

After confirming the schemas align, MOLT Verify moves on to the data itself. This is the most intensive part: it compares the contents of each table row-by-row between the source and CockroachDB. Under the hood, MOLT Verify will typically perform an ordered scan of the table on each database (often using the primary key order to have a consistent traversal) and then compare the rows as it goes. Essentially, it performs a synchronized read from source and target and checks that for every primary key (or unique identifier) present in the source, the target has a row with identical values.

MOLT Verify will report summary counts of matched vs mismatched data for each table. For example, after running, you might see an output (in JSON or log format) per table indicating how many rows were successfully verified and how many differences were found. An example summary line looks like this:

{"table_schema":"public",
"table_name":"common_table",
"num_truth_rows":6,
"num_success":3,
"num_missing":2,
"num_mismatch":1,
"num_extraneous":2, 
... 
"message":"finished row verification"}

In this example, MOLT Verify is reporting that out of 6 rows in the source (“truth”) table, 3 rows matched exactly, 2 rows were missing on the target, 1 row had mismatched data, and 2 extraneous rows were found on the target. Similar statistics are output for each table.

This row-by-row comparison mechanism is essentially performing a massive diff between the two databases. Importantly, MOLT Verify does this in a streaming and memory-efficient way. It processes a chunk of rows at a time (by default 20,000 rows per batch) rather than loading an entire table into memory. It reads a batch from the source and the corresponding batch from the target, compares them, then moves on.

Handling Live Data Changes

In a migration scenario, you might be running MOLT Verify while changes are still trickling into the source (and being replicated to the target). If the source database is still “live,” it’s possible a row could change between the time you read it from the source and the time you read it from the target (or between batches). MOLT Verify accounts for this with the --live flag. When you run with the --live flag, the tool will automatically retry verification on any rows that didn’t match before declaring them as errors. This is useful during continuous replication or minimal-downtime migrations – it prevents false alarms due to timing lags. If after a retry the row still doesn’t match, then it’s a true mismatch. The output even tracks a num_live_retry count to show how many mismatches were fixed by a later retry (so you know how much data was in flux).

There is also a --continuous mode, which tells MOLT Verify to run in a loop continuously – essentially, once it finishes verifying all tables, it will start over until you stop it. This could be used to constantly monitor data consistency during a long-running migration, giving you up-to-date information on whether the target is in sync.

Get all the details in our documentation.

Check out MOLT Verify in action as technical evangelist, Rob Reid validates the migration from his PostgreSQL database to CockroachDB:

Best Practices: Tips for Using MOLT Verify Effectively

To get the most out of MOLT Verify and ensure a smooth migration, consider the following best practices and recommendations:

Run Verification at Key Points in the Migration

Don’t wait until the final switch to run MOLT Verify for the first time. It’s wise to use it in earlier stages as well. For example, after an initial test data load, run MOLT Verify on a subset of data to catch any glaring schema or data issues. During iterative testing (the “validation” phase of migration), use it on smaller samples. By the time you do the full migration, you’ll be familiar with the tool and have confidence that it will likely report zero issues.

Always run a final verification on the full dataset right before cutover (or as part of the cutover) – this is your final gate to ensure consistency. If using continuous replication, keep MOLT Verify running with --continuous so you have up-to-date info, and then do one last run in non-continuous mode when replication is stopped.

Minimize Changes for the Final Verification

While MOLT Verify can work with live data, you will get the cleanest results if the source is static or nearly static during the verification. Try to schedule the verification during a read-only window on the source if possible. Many teams will stop application writes, let the replication catch up, then run MOLT Verify on a static snapshot of data. This avoids the complexity of transient mismatches. If you must verify while changes are happening, definitely use --live to filter out false mismatches, and possibly run multiple passes (--continuous) until the mismatches go to zero.

Tune Concurrency and Batching for Your Environment:

You can adjust --concurrency, --concurrency-per-table, and --row-batch-size to optimize runtime. The --concurrency flag toggles how many tables to verify at a time, and then --concurrency-per-table allows you to adjust how many threads should run per table. Start with defaults, and if you find verification is taking too long, consider increasing concurrency. If you do raise concurrency, monitor the source database – you don’t want to accidentally overload production with read queries. It might be useful to test on a staging environment to gauge how much concurrency the source can handle.

If the network is a bottleneck, see if larger batches improve throughput. Conversely, if you run into memory constraints on the MOLT Verify side (or if either database starts to see issues like long transaction times), you might lower the batch size to reduce the footprint. The goal is to make verification as fast as possible without jeopardizing the stability of your systems.

MOLT Verify also offers --schema-filter and --table-filter options (accepting regex patterns) to limit what it checks. This is useful for phased migrations. For instance, if you are migrating in waves, you could verify one schema at a time by filtering. Or if one giant table dominates the runtime, you might isolate it in a separate run.

RELATED

Check out our tutorials on how to migrate from PostgreSQL or MySQL to CockroachDB.

MOLT to CockroachDB with Confidence

Ensuring data integrity during migrations is crucial for any organization. MOLT Verify provides an automated, comprehensive validation process that gives you the confidence to complete your migration swiftly and without the fear of data corruption. Remove guesswork from your migrations with MOLT Verify, validating every record systematically before production.

To recap best practices: use the tool early and often, verify in as controlled an environment as possible, pay attention to its output, and address any problems it uncovers. Migrations are like a high-stakes endeavor, but with tools like MOLT Verify, you have a stethoscope to check the health of your data before declaring the operation a success.

Get started with CockroachDB Cloud today. We’re offering $400 of free credits to help kickstart your CockroachDB journey, or get in touch today to learn more.

Cockroach Labs Blog - engineering

The Database at 550 Kilometers: What Orbital Computing Means for Distributed Databases

I. The hum of modern data centers

II. Why are companies planning data centers in space?

III. What happens when databases move to space?

IV. How do distributed databases already solve part of the problem?

V. How would a distributed database work across a satellite constellation?

Related

VI. Should data live in orbit or on Earth?

VII. How do latency and consensus change in orbit?

VIII. Does relativity matter for an orbital database?

IX. How should databases adapt to moving satellite networks?

X. How does data sovereignty work in orbit?

XI. What happens when an orbital database node disappears?

XII. What would orbital databases still need?

Related

Vehicle Search with SQL and Vector Embeddings

1. Visual Search: Finding That "Fast and Furious" Look

The Process

The Results

2. Under the Hood

The Tech Stack

The Data

3. Data Exploration

Visualizing Inventory Distribution

4. Visualizing the Vector Space

The Pipeline

The 3D Plot

Conclusion

Value Separation in Pebble: Storage Engine Optimization

Motivation

Background

Impact

Prior art

Mechanics

Tradeoffs

Locality

Space Amplification

Blob File Rewrites

CockroachDB separation heuristics

Try CockroachDB Today

Real-Time Indexing for Billions of Vectors: How we built fast, fresh vector indexing at scale in CockroachDB

Embedding Meaning into Vectors

How Meaning is Indexed

Adapting Vector Indexing Algorithms for Distributed SQL

Introducing C-SPANN

Maintaining a Healthy Index

Reducing Index Size by 94%

An Index for Every User

Users in Every Region

Future Work

Try CockroachDB Today

Tutorial: Augment your AI use case with RAG on CockroachDB

What is Retrieval Augmented Generation (RAG)? How does RAG improve LLMs?

Retrieval Augmented Generation (RAG) Use Cases

Why Build a RAG Application on CockroachDB

Unified Storage of Source Data, Metadata, and Vector Data

Scalability, Reliability, and Simplicity of Operations

Measure what matters

Advanced Access Control and Security

How to build a knowledge base chatbot with RAG on CockroachDB

Data flow

Application architecture

RAG and CockroachDB: Better Together

Try CockroachDB Today

Ideal isn’t real: Stress testing CockroachDB’s resilience

Measure what matters

The Seven Phases of Adversity

What Keeps CockroachDB Resilient and Stable Under Pressure?

Resilience Is Essential, Not Optional

Try CockroachDB Today

CockroachDB 25.2: Celebrating a decade of innovation with enhanced performance, vector indexing, and more

Performance improvements: 50% increased throughput and more

Vector indexing is in Preview for AI applications at scale

Introducing row-level security

New CockroachDB Cloud regions and plan switching without downtime

Try CockroachDB Today

CockroachDB Redefines Database Performance with Real-World Testing

“Performance under Adversity” pressure testing proves most resilient database

Why we must re-evaluate database performance now