Working with the data files

Why parquet and not CSV

A release holds up to about 248 million rows. At that size the file format stops being a detail and starts deciding whether the work is pleasant or miserable. Four things make the difference:

It stores columns, not rows

Ask for domain and rating and the reader touches only those two columns on disk. The score and the two link counts are never read. In a CSV every byte of every row has to be read and parsed to get at any field.

It compresses far better

Values of one type sit next to each other, which compresses much harder than mixed rows of text. We use ZSTD. The same data as CSV would be several times larger to download and to keep.

It carries its own types

A rating arrives as a 16 bit integer and a score as a double, because the file says so. Nothing has to guess, and none of the usual CSV hazards apply: no quoting rules, no locale-dependent decimal separators, no column silently read as text.

It can skip work

The file is split into row groups, each recording the minimum and maximum value it holds. Our rows are sorted by domain, so a reader looking for one domain can skip almost the entire file rather than scanning it. This is why a targeted query against a 25 GB file returns in moments.

Parquet is also a genuinely open standard with mature readers in every language that matters, so choosing it does not tie you to us or to any one vendor.

Open one in thirty seconds

If you install one thing, install DuckDB. It is a single binary, needs no server, and reads parquet natively.

# macOS
brew install duckdb

# Windows
winget install DuckDB.cli

# Linux, or anywhere
curl https://install.duckdb.org | sh

Then, in the folder holding your download:

duckdb -c "SELECT domain, rating FROM '2025-12.parquet' WHERE domain = 'example.com'"

That is the whole setup. No import, no schema to declare, no database to create. Run duckdb -ui instead and you get a notebook interface in your browser.

Prefer to look before you buy? The free sample file is a CSV with the same columns, so you can look at real rows before you handle a parquet file at all.

Desktop apps that read parquet

If you would rather click than type, any of these will open a release. All are third-party tools we have no connection with.

DuckDB

A single binary, no server, no install ceremony. duckdb -ui opens a notebook in the browser. If you only ever learn one tool for this, learn this one.

DBeaver

Free, cross platform database GUI. Create a DuckDB connection, then point it at a file with read_parquet('...'). Good when you want a grid, a query editor and export buttons.

Tad

A desktop viewer built specifically for parquet and CSV. Opens a file, gives you a pivotable grid, and does not ask you to write SQL.

VisiData

Terminal spreadsheet. vd 2025-12.parquet and you are browsing, sorting and filtering without leaving the shell.

ParquetViewer

A small Windows viewer for a quick look at contents and schema.

DataGrip

Commercial IDE for databases; reads parquet through a DuckDB connection like DBeaver does.

Excel on Windows can import parquet through Power Query (Data, then Get Data, then From File, then From Parquet), but a full release has far more rows than a worksheet holds. Filter it down with one of the tools above first, or open the free sample, which is a CSV.

Code

Every example below reads the file directly. None of them need an import step or a database.

Columns: domain (string), rating (int16), score (double), inbound_count (int64), outbound_count (int64).

-- No import step. Query the file where it sits.
SELECT domain, rating
FROM   read_parquet('2025-12.parquet')
WHERE  domain IN ('example.com', 'bbc.co.uk');

-- Everything above a threshold, biggest first.
SELECT domain, rating, inbound_count
FROM   read_parquet('2025-12.parquet')
WHERE  rating >= 60
ORDER  BY rating DESC
LIMIT  100;

-- Score a list of your own domains by joining a CSV against the release.
SELECT mine.domain, ratings.rating
FROM   read_csv('my-domains.csv', header = true) AS mine
LEFT   JOIN read_parquet('2025-12.parquet') AS ratings USING (domain);

-- Compare two months to see what moved.
SELECT older.domain, older.rating AS was, newer.rating AS now
FROM   read_parquet('2025-09.parquet') AS older
JOIN   read_parquet('2025-12.parquet') AS newer USING (domain)
WHERE  abs(newer.rating - older.rating) >= 10;

# DuckDB: SQL, no import step, lowest memory.
import duckdb

rows = duckdb.sql("""
    SELECT domain, rating
    FROM   read_parquet('2025-12.parquet')
    WHERE  rating >= 60
""").df()

# Polars: lazy, so the filter and column choice push down into the file.
import polars as pl

frame = (
    pl.scan_parquet("2025-12.parquet")
      .filter(pl.col("rating") >= 60)
      .select("domain", "rating")
      .collect()
)

# pandas: simplest, but reads everything you ask for into memory.
# Always pass columns, or you pull all five for no reason.
import pandas as pd

frame = pd.read_parquet("2025-12.parquet", columns=["domain", "rating"])

# PyArrow, if you want the row groups and statistics yourself.
import pyarrow.parquet as pq

table = pq.read_table(
    "2025-12.parquet",
    columns=["domain", "rating"],
    filters=[("rating", ">=", 60)],
)

library(arrow)
library(dplyr)

ratings <- open_dataset("2025-12.parquet")

ratings |>
  filter(rating >= 60) |>
  select(domain, rating) |>
  arrange(desc(rating)) |>
  collect()

# Or with DuckDB, if you prefer SQL:
library(duckdb)

connection <- dbConnect(duckdb())
dbGetQuery(connection, "
  SELECT domain, rating FROM read_parquet('2025-12.parquet') WHERE domain = 'example.com'
")

require "duckdb"

database = DuckDB::Database.open
connection = database.connect

result = connection.query(<<~SQL)
  SELECT domain, rating
  FROM   read_parquet('2025-12.parquet')
  WHERE  rating >= 60
  ORDER  BY rating DESC
  LIMIT  10
SQL

result.each { |domain, rating| puts "#{domain}: #{rating}" }

// Node: npm install @duckdb/node-api
import { DuckDBInstance } from "@duckdb/node-api"

const instance = await DuckDBInstance.create()
const connection = await instance.connect()

const reader = await connection.runAndReadAll(`
  SELECT domain, rating
  FROM   read_parquet('2025-12.parquet')
  WHERE  rating >= 60
  LIMIT  10
`)

console.table(reader.getRowObjects())

package main

import (
    "fmt"
    "github.com/parquet-go/parquet-go"
)

type Rating struct {
    Domain        string `parquet:"domain"`
    Rating        int16  `parquet:"rating"`
    Score         float64 `parquet:"score"`
    InboundCount  int64  `parquet:"inbound_count"`
    OutboundCount int64  `parquet:"outbound_count"`
}

func main() {
    rows, err := parquet.ReadFile[Rating]("2025-12.parquet")
    if err != nil {
        panic(err)
    }

    for _, row := range rows {
        if row.Rating >= 60 {
            fmt.Printf("%s: %d\n", row.Domain, row.Rating)
        }
    }
}

// Spark
Dataset<Row> ratings = spark.read().parquet("2025-12.parquet");

ratings.select("domain", "rating")
       .filter("rating >= 60")
       .orderBy(functions.col("rating").desc())
       .show(20);

// A whole directory of releases at once, with the month as a column.
Dataset<Row> history = spark.read().parquet("releases/*.parquet");

use polars::prelude::*;

fn main() -> PolarsResult<()> {
    let frame = LazyFrame::scan_parquet("2025-12.parquet", Default::default())?
        .filter(col("rating").gt_eq(lit(60)))
        .select([col("domain"), col("rating")])
        .collect()?;

    println!("{frame}");
    Ok(())
}

// dotnet add package Parquet.Net
using Parquet;
using Parquet.Schema;

using Stream stream = File.OpenRead("2025-12.parquet");
using ParquetReader reader = await ParquetReader.CreateAsync(stream);

DataField domain = (DataField)reader.Schema.DataFields[0];
DataField rating = (DataField)reader.Schema.DataFields[1];

for (int group = 0; group < reader.RowGroupCount; group++)
{
    using ParquetRowGroupReader rows = reader.OpenRowGroupReader(group);
    var domains = await rows.ReadColumnAsync(domain);
    var ratings = await rows.ReadColumnAsync(rating);
    // ...
}

Loading into your own database

Most warehouses ingest parquet directly, which is the fastest route by a wide margin.

-- ClickHouse
INSERT INTO ratings FROM INFILE '2025-12.parquet' FORMAT Parquet;

-- BigQuery
bq load --source_format=PARQUET dataset.ratings 2025-12.parquet

-- Snowflake, after staging the file
COPY INTO ratings FROM @my_stage/2025-12.parquet FILE_FORMAT = (TYPE = PARQUET);

-- DuckDB, if you want a local database rather than querying the file each time
CREATE TABLE ratings AS SELECT * FROM read_parquet('2025-12.parquet');

PostgreSQL has no native parquet reader, so convert first. Have DuckDB write the CSV and let COPY stream it in, which is exactly how this site loads a release:

duckdb -c "COPY (SELECT domain, rating FROM read_parquet('2025-12.parquet'))
           TO 'ratings.csv' (FORMAT csv, HEADER false)"

psql -c "CREATE TABLE ratings (domain text COLLATE "C" PRIMARY KEY, rating smallint NOT NULL)"
psql -c "py ratings (domain, rating) FROM 'ratings.csv' WITH (FORMAT csv)"

Add the primary key after the load rather than before. The rows arrive already sorted by domain, so the index builds quickly, and the copy itself is far faster without it.

Working with a file this big

• • Name your columns. Asking for two columns instead of five is the single biggest saving available, and it costs one argument.
• • Filter in the query, not after it. A WHERE clause or a lazy frame lets the reader skip row groups. Loading everything and filtering in memory does the opposite.
• • Stream the download to disk. Do not buffer a 25 GB response in memory. Every HTTP client has a streaming mode.
• • Check what you already have. The API returns an etag and size_bytes for every file. Compare before re-downloading a release you already hold.
• • Keep the files. The back catalogue is what makes month-on-month comparison possible, and a join across two releases is a single query.