Spotify's RAP Index Lets Data Lakes Answer Queries in Milliseconds

For years, the data world has been split into two camps. On one side, you have your data lake—cheap, flexible, and built for big analytical scans. On the other, operational databases that can fetch a single record in milliseconds but cost a fortune to scale and maintain. Spotify, which manages exabytes of data in Google Cloud Storage and petabytes in Bigtable, felt that split acutely. Every time an engineer wanted to look up a specific user's listening history in real time, the data had to be copied from the lake into a faster store. That duplication is expensive, slow, and frankly, a pain.

So Spotify built something new. They call it Random Access Parquet (RAP). It's an external index layer that sits on top of Apache Parquet files, giving you low-latency point queries directly on your data lake. No copying. No separate operational database. Just the same Parquet files you already have, now able to answer queries like "what's this user's top track today?" in milliseconds.

The trick is an index that maps query keys—like a user ID—to specific Parquet files and even row positions within those files. Instead of scanning thousands of files to find one record, the query engine first checks the index, then does a targeted range read from object storage. It's like having a card catalog for your library, rather than walking through every aisle to find a single book.

Why Point Queries Are So Hard in Data Lakes

Data lakes are great for analytics. Trino, BigQuery, Spark—they all love scanning huge datasets and aggregating results. But they're not built for the kind of queries you'd send to a database like Bigtable or Cassandra. Those engines optimize for throughput, not latency. A point query—"get me row 42"—involves query planning, metadata traversal, and file discovery, even if the underlying storage can deliver data in milliseconds.

Spotify notes that cloud object storage like Google Cloud Storage now offers millisecond access times, but the overhead of planning and finding the right files can still add hundreds of milliseconds or more. For an online service that needs to respond in under 100ms, that's a dealbreaker. Hence the need for an index that eliminates the guesswork.

How RAP Works: An Index That Never Touches Your Data

RAP's index is built incrementally as new data arrives in Apache Iceberg tables. When a new batch of data is written, an index builder generates an append-only index fragment. It never modifies the immutable Parquet files. This means your analytical pipelines, machine learning jobs, notebooks, and AI agents can all keep using the same dataset without any changes, while online applications get the low-latency access they need.

The index maps keys to file paths and row groups. When a query comes in, the system looks up the key, finds the exact location, and issues a range read to fetch just those bytes. No full scans, no reading thousands of files, no redundant copies.

Storage Layout Tweaks That Make It Fast

Indexes alone aren't enough. Spotify also uses several storage layout optimizations to reduce the number of files touched and the amount of data read. For example, they sort data by query key, which means records with the same key are stored together. That reduces the number of files a query needs to access. They also interleave value columns so that multiple attributes can be fetched in a single continuous read. And they use covering indexes, which allow some queries to be answered entirely from the index without even touching the Parquet files.

These techniques come at a small cost: slightly larger files and indexes. But the payoff is huge. Some point queries now require only a single range read of a few kilobytes. That's a far cry from scanning gigabytes of data.

Secondary Indexes and Multi-Dimensional Queries

RAP also supports secondary indexes. Say you run a marketplace and want to find all orders for a given buyer ID or seller ID. You don't want to rewrite your Parquet files to support each new query pattern. With RAP, you can add secondary indexes at the service layer, without changing your data pipelines. These indexes can be hash-based for exact-match queries or sorted for range queries.

Spotify also mentions using storage layout techniques like Z-ordering and Hilbert curves to improve data locality for secondary query dimensions. These are fancy ways of saying you can organize data so that related records are physically close, reducing the number of reads needed.

The Bigger Picture: Open Data Lakes for Online Workloads

RAP is part of a broader industry push to make data lakes handle more than just batch analytics. Google Cloud recently introduced an Apache Iceberg-based lakehouse architecture for AI applications, aiming to reduce data duplication while enabling operational access. But Spotify's approach is different: instead of changing the storage format, they add a dedicated external index layer that's optimized for point queries, while keeping compatibility with existing Parquet and Iceberg tables.

Data engineers have taken notice. Andrew Lamb sees RAP as a great example of extending open data formats to support interactive workloads. Vikas Singh, in a LinkedIn discussion, points out that as cloud object storage gets faster, the bottleneck shifts to query planning and metadata access—and RAP addresses exactly that by precomputing the index.

Why This Matters for Free Living

Now, you might be wondering: what does this have to do with free living? At its core, free living is about cutting out the unnecessary—the redundant systems, the wasted effort, the duplicated data. It's about making do with what you have and finding clever ways to make it work harder for you. Spotify's RAP is a perfect example of that philosophy in action. Instead of maintaining two separate storage systems—one for analytics, one for operations—they found a way to make one system serve both purposes.

This isn't just a tech story. It's a mindset. Whether you're managing your personal finances, your closet, or your digital life, the principle is the same: before you buy more, before you build more, ask if you can make what you already have work better. Sometimes the answer is a clever index, not a whole new database.

What's Next for RAP and Data Lakes

Spotify hasn't open-sourced RAP yet, but the architecture is described in enough detail that other companies could implement something similar. As data lakes continue to grow—and as AI applications demand faster access to training data and inference-time lookups—expect to see more tools that blur the line between analytical and operational workloads.

The future might not be about choosing between a lake and a database. It might be about having your lake and querying it too, without the overhead of copying data around. That's the promise of RAP, and it's a step toward a simpler, leaner data ecosystem.

For now, if you're building data systems, keep an eye on this space. And if you're just trying to live a little freer, remember: sometimes the best solution is not more stuff, but smarter use of what you've got.