Creating ENSRainbow Files

ENSRainbow provides two methods for creating .ensrainbow files from different data sources. This guide helps you choose the right method and provides step-by-step instructions.

Prerequisites

Before creating .ensrainbow files, ensure you have:

1. ENSNode repository cloned:

   git clone https://github.com/namehash/ensnode.git
   cd ensnode

2. Dependencies installed:

   pnpm install

3. Working directory: Navigate to the ENSRainbow directory:

   cd apps/ensrainbow

All commands in this guide assume you’re in the apps/ensrainbow directory unless otherwise specified.

Overview

A .ensrainbow file is ENSRainbow’s binary format for storing label-to-labelhash mappings. It uses Protocol Buffers for efficient serialization and supports streaming for large datasets.

For detailed information about the file format structure, see the Data Model documentation.

Choosing Your Conversion Method

Method	Input Format	Use Case	Command
SQL Conversion	Gzipped SQL dump (ens_names.sql.gz)	Converting legacy ENS Subgraph data	pnpm run convert
CSV Conversion	CSV file (1 or 2 columns)	Custom datasets, test data, external sources	pnpm run convert-csv

When to Use SQL Conversion

• Converting existing ENS Subgraph rainbow tables
• Working with legacy ens_names.sql.gz files
• Migrating from previous ENS data formats

When to Use CSV Conversion

• Creating test datasets
• Converting data from external sources
• Working with custom label collections
• Building incremental label sets

Method 1: Converting from SQL Dumps

The convert command processes gzipped SQL dump files from the ENS Subgraph.

Command Syntax

pnpm run convert \
  --input-file <path/to/ens_names.sql.gz> \
  --output-file <output.ensrainbow> \
  --label-set-id <label-set-id> \
  --label-set-version <version-number>

Required Parameters

• --input-file: Path to the gzipped SQL dump file
• --label-set-id: Identifier for the label set (e.g., subgraph, discovery-a)
• --label-set-version: Version number for the label set (non-negative integer)

Optional Parameters

• --output-file: Output file path (defaults to rainbow-records.ensrainbow)

Example: Converting ENS Subgraph Data

# Convert main ENS Subgraph data
pnpm run convert \
  --input-file ens_names.sql.gz \
  --output-file subgraph_0.ensrainbow \
  --label-set-id subgraph \
  --label-set-version 0

Example: Converting Test Data

# Convert ens-test-env data
pnpm run convert \
  --input-file test/fixtures/ens_test_env_names.sql.gz \
  --output-file ens-test-env_0.ensrainbow \
  --label-set-id ens-test-env \
  --label-set-version 0

How It Works

1. Streams the gzipped SQL file to avoid memory issues
2. Parses SQL COPY statements to extract label/labelhash pairs
3. Validates each record and skips invalid entries
4. Writes protobuf messages with length-delimited encoding
5. Creates a header message followed by individual record messages

Method 2: Converting from CSV Files

The convert-csv command processes CSV files with flexible column formats.

Command Syntax

pnpm run convert-csv \
  --input-file <path/to/data.csv> \
  --output-file <output.ensrainbow> \
  --label-set-id <label-set-id> \
  --label-set-version <version-number> \
  [--progress-interval <number>] \
  [--existing-db-path <path/to/existing/database>]

Required Parameters

• --input-file: Path to the CSV file
• --label-set-id: Identifier for the label set
• --label-set-version: Version number for the label set

Optional Parameters

• --output-file: Output file path (defaults to rainbow-records.ensrainbow)
• --progress-interval: Progress logging frequency (default: 10000 records)
• --existing-db-path: Path to existing ENSRainbow database to filter out existing labels

CSV Format Support

The CSV converter supports two formats:

Single Column Format (Label Only)

ethereum
vitalik
ens

The converter automatically computes labelhashes using the labelhash() function.

Two Column Format (Label + Labelhash)

ethereum,0x541111248b45b7a8dc3f5579f630e74cb01456ea6ac067d3f4d793245a255155
vitalik,0xaf2caa1c2ca1d027f1ac823b529d0a67cd144264b2789fa2ea4d63a67c7103cc
ens,0x5cee339e13375638553bdf5a6e36ba80fb9f6a4f0783680884d92b558aa471da

The converter validates that provided labelhashes match the computed hash for each label.

Label Filtering

The CSV converter includes built-in filtering capabilities to prevent duplicate labels:

Filtering Existing Labels

Use --existing-db-path to filter out labels that already exist in an existing ENSRainbow database:

pnpm run convert-csv \
  --input-file new-labels.csv \
  --output-file incremental_1.ensrainbow \
  --label-set-id my-dataset \
  --label-set-version 1 \
  --existing-db-path data-my-dataset

This will:

• Check each label against the existing database
• Skip labels that already exist (avoiding duplicates)
• Only write new labels to the output file
• Log filtering statistics in the conversion summary

Filtering Duplicate Labels Within CSV

The converter automatically filters duplicate labels within the same CSV file, keeping only the first occurrence of each label.

Filtering Statistics

The conversion process logs detailed statistics:

=== Conversion Summary ===
Total lines processed: 1000
Valid records: 850
Filtered existing labels: 100
Filtered duplicates: 50
Duration: 150ms

Example: Creating Test Dataset

# Create test dataset from CSV
pnpm run convert-csv \
  --input-file test-labels.csv \
  --output-file test-dataset_0.ensrainbow \
  --label-set-id test-dataset \
  --label-set-version 0

Example: Creating Discovery Dataset

# Create discovery dataset (initially empty)
echo "" > empty.csv
pnpm run convert-csv \
  --input-file empty.csv \
  --output-file discovery-a_0.ensrainbow \
  --label-set-id discovery-a \
  --label-set-version 0

How It Works

1. Detects CSV format automatically (1 or 2 columns)
2. Streams CSV parsing using fast-csv for memory efficiency
3. Validates column count and data format
4. Computes or validates labelhashes as needed
5. Filters existing labels if --existing-db-path is provided
6. Filters duplicate labels within the same CSV file
7. Writes protobuf messages with the same format as SQL conversion

Common Workflows

Workflow 1: Migrating from ENS Subgraph

# 1. Convert SQL dump to .ensrainbow
pnpm run convert \
  --input-file ens_names.sql.gz \
  --output-file subgraph_0.ensrainbow \
  --label-set-id subgraph \
  --label-set-version 0

# 2. Ingest into LevelDB
pnpm run ingest-ensrainbow \
  --input-file subgraph_0.ensrainbow \
  --data-dir data-subgraph

# 3. Validate the database
pnpm run validate --data-dir data-subgraph

# 4. Start the API server
pnpm run serve --data-dir data-subgraph --port 3223

Workflow 2: Creating Test Environment

# 1. Convert test data
pnpm run convert \
  --input-file test/fixtures/ens_test_env_names.sql.gz \
  --output-file ens-test-env_0.ensrainbow \
  --label-set-id ens-test-env \
  --label-set-version 0

# 2. Ingest test data
pnpm run ingest-ensrainbow \
  --input-file ens-test-env_0.ensrainbow \
  --data-dir data-test-env

# 3. Run with test data
pnpm run serve --data-dir data-test-env --port 3223

Workflow 3: Building Custom Dataset

# 1. Create CSV with your labels
echo "mylabel1
mylabel2
mylabel3" > custom-labels.csv

# 2. Convert to .ensrainbow
pnpm run convert-csv \
  --input-file custom-labels.csv \
  --output-file custom_0.ensrainbow \
  --label-set-id custom \
  --label-set-version 0

# 3. Ingest and serve
pnpm run ingest-ensrainbow \
  --input-file custom_0.ensrainbow \
  --data-dir data-custom

pnpm run serve --data-dir data-custom --port 3223

Workflow 4: Creating Incremental Updates

# 1. Create initial dataset
pnpm run convert-csv \
  --input-file initial-labels.csv \
  --output-file my-dataset_0.ensrainbow \
  --label-set-id my-dataset \
  --label-set-version 0

# 2. Ingest initial data
pnpm run ingest-ensrainbow \
  --input-file my-dataset_0.ensrainbow \
  --data-dir data-my-dataset

# 3. Create incremental update (filtering existing labels)
pnpm run convert-csv \
  --input-file new-labels.csv \
  --output-file my-dataset_1.ensrainbow \
  --label-set-id my-dataset \
  --label-set-version 1 \
  --existing-db-path data-my-dataset

# 4. Ingest incremental update
pnpm run ingest-ensrainbow \
  --input-file my-dataset_1.ensrainbow \
  --data-dir data-my-dataset

# 5. Serve updated data
pnpm run serve --data-dir data-my-dataset --port 3223

Workflow 5: Using Custom Label Set Server

# 1. Configure custom label set server
export ENSRAINBOW_LABELSET_SERVER_URL="https://my-label-set-server.com"

# 2. Download from custom server
# The script downloads to labelsets/ subdirectory
./scripts/download-ensrainbow-files.sh my-dataset 0

# 3. Ingest and serve
# Files are downloaded to labelsets/ by the script
pnpm run ingest-ensrainbow \
  --input-file labelsets/my-dataset_0.ensrainbow \
  --data-dir data-my-dataset

pnpm run serve --data-dir data-my-dataset --port 3223

Script Output Locations

ENSRainbow download scripts save files to specific subdirectories:

• .ensrainbow files: labelsets/
• Database archives: databases/{schema_version}/
• Checksums and licenses: Same directory as the downloaded file

File Naming Conventions

Follow the naming convention: {label-set-id}_{label-set-version}.ensrainbow

Examples:

• subgraph_0.ensrainbow - Main ENS data, version 0
• subgraph_1.ensrainbow - Main ENS data, version 1 (incremental update)
• discovery-a_0.ensrainbow - Discovery dataset, version 0
• ens-test-env_0.ensrainbow - Test environment data, version 0

Next Steps

After creating your .ensrainbow file:

1. Ingest the data into a ENSRainbow database
2. Validate the database to ensure integrity
3. Start the API server to serve the data

For complete CLI reference information, see the CLI Reference documentation.

Creating and Publishing Custom .ensrainbow Files

If you want to create, publish, and distribute your own .ensrainbow files, follow these steps:

1. Create Your Dataset

First, prepare your data in either SQL or CSV (recommended) format, then convert it using the appropriate method:

# For CSV data
pnpm run convert-csv \
  --input-file my-labels.csv \
  --output-file my-dataset_0.ensrainbow \
  --label-set-id my-dataset \
  --label-set-version 0

# For CSV data with filtering (if you have an existing database)
pnpm run convert-csv \
  --input-file my-labels.csv \
  --output-file my-dataset_1.ensrainbow \
  --label-set-id my-dataset \
  --label-set-version 1 \
  --existing-db-path data-my-dataset

# For SQL data
pnpm run convert \
  --input-file my-data.sql.gz \
  --output-file my-dataset_0.ensrainbow \
  --label-set-id my-dataset \
  --label-set-version 0

2. Validate Your File

Test your .ensrainbow file by ingesting it locally:

# Ingest your custom dataset
pnpm run ingest-ensrainbow \
  --input-file my-dataset_0.ensrainbow \
  --data-dir data-my-dataset

# Validate the database
pnpm run validate --data-dir data-my-dataset

# Test the API
pnpm run serve --data-dir data-my-dataset --port 3223

3. Publish Your File

Option A: Direct File Sharing

• Upload your .ensrainbow file to a web server or cloud storage
• Provide a direct download URL
• Share checksums for integrity verification

Option B: Package as Database Archive

For better performance, package your data as a pre-built database:

# Ingest your .ensrainbow file
pnpm run ingest-ensrainbow \
  --input-file my-dataset_0.ensrainbow \
  --data-dir data-my-dataset

# Package the database
tar -czvf my-dataset_0.tgz ./data-my-dataset

# Calculate checksum
sha256sum my-dataset_0.tgz > my-dataset_0.tgz.sha256sum

4. Document Your Label Set

Create documentation for your custom label set including:

• Label Set ID: The identifier users will specify
• Description: What labels are included and their source
• Version: Current version number
• Download URLs: Where to get the files
• Checksums: For integrity verification
• Usage Examples: How to use your dataset

Example Documentation Format

## Custom Label Set: my-dataset

**Label Set ID**: `my-dataset`
**Current Version**: `0`
**Description**: Custom ENS labels from [source description]

### Download
- Database Archive: `https://example.com/my-dataset_0.tgz`
- Checksum: `https://example.com/my-dataset_0.tgz.sha256sum`

### Usage
```bash
# Using with Docker
docker run -d \
  -e DB_SCHEMA_VERSION="3" \
  -e LABEL_SET_ID="my-dataset" \
  -e LABEL_SET_VERSION="0" \
  -p 3223:3223 \
  ghcr.io/namehash/ensnode/ensrainbow:latest

Setting Up Your Own Label Set Server

A Label Set Server is a storage and hosting service for .ensrainbow files and prebuilt database archives. It’s not the ENSRainbow API server itself, but rather a way to distribute your custom datasets for others to download and use.

1. Choose Your Hosting Platform

You can host your label set files on any web server or cloud storage service:

• AWS S3: Industry standard with versioning
• Cloudflare R2: Cost-effective alternative to S3
• Simple HTTP server: For internal/private use

2. Organize Your Files

Structure your label set files following ENSRainbow conventions:

my-label-set-server/
├── labelsets/
│   ├── my-dataset_0.ensrainbow
│   ├── my-dataset_0.ensrainbow.sha256sum
│   ├── my-dataset_1.ensrainbow
│   └── my-dataset_1.ensrainbow.sha256sum
└── databases/
    ├── 3/  # Schema version
    │   ├── my-dataset_0.tgz
    │   ├── my-dataset_0.tgz.sha256sum
    │   ├── my-dataset_1.tgz
    │   └── my-dataset_1.tgz.sha256sum
    └── 4/  # Future schema version

3. Use Existing Download Scripts

ENSRainbow provides ready-to-use download scripts that users can configure to download from your label set server:

Download .ensrainbow Files

# Configure your label set server URL
export ENSRAINBOW_LABELSET_SERVER_URL="https://my-label-set-server.com"

# Download .ensrainbow file using the existing script
./scripts/download-ensrainbow-files.sh my-dataset 0

Download Prebuilt Database Archives

# Configure your label set server URL
export ENSRAINBOW_LABELSET_SERVER_URL="https://my-label-set-server.com"

# Download prebuilt database using the existing script
./scripts/download-prebuilt-database.sh 3 my-dataset 0

Script Features

The existing scripts automatically handle:

• Checksum verification for data integrity
• Resume downloads if files already exist and are valid
• License file downloads (optional)
• Progress reporting for large files
• Error handling with cleanup of partial downloads

4. Document Your Label Set Server

Create a README or documentation page for your label set server:

# My Label Set Server

This server hosts custom ENS label sets for ENSRainbow.

## Available Label Sets

### my-dataset
- **Description**: Custom ENS labels from [source]
- **Versions**: 0, 1
- **Schema Versions**: 3
- **Base URL**: `https://my-label-set-server.com`

### another-dataset
- **Description**: Additional labels from [source]
- **Versions**: 0
- **Schema Versions**: 3
- **Base URL**: `https://my-label-set-server.com`

Usage

Users should have the ENSNode repository cloned and be in the apps/ensrainbow directory.

Option 1: Download .ensrainbow Files

# Configure your label set server
export ENSRAINBOW_LABELSET_SERVER_URL="https://my-label-set-server.com"

# Download .ensrainbow file
./scripts/download-ensrainbow-files.sh my-dataset 0

# Ingest into ENSRainbow
pnpm run ingest-ensrainbow \
  --input-file labelsets/my-dataset_0.ensrainbow \
  --data-dir data-my-dataset

# Start ENSRainbow server
pnpm run serve --data-dir data-my-dataset --port 3223

Option 2: Download Prebuilt Databases (Faster)

# Configure your label set server
export ENSRAINBOW_LABELSET_SERVER_URL="https://my-label-set-server.com"

# Download prebuilt database
./scripts/download-prebuilt-database.sh 3 my-dataset 0

# Extract database
tar -xzf databases/3/my-dataset_0.tgz -C data-my-dataset --strip-components=1

# Start ENSRainbow server
pnpm run serve --data-dir data-my-dataset --port 3223

5. Version Management

Implement proper versioning for your label sets:

# When releasing a new version
LABEL_SET_ID="my-dataset"
NEW_VERSION="1"

# Create new .ensrainbow file
pnpm run convert-csv \
  --input-file updated-labels.csv \
  --output-file ${LABEL_SET_ID}_${NEW_VERSION}.ensrainbow \
  --label-set-id ${LABEL_SET_ID} \
  --label-set-version ${NEW_VERSION}

# Create prebuilt database
pnpm run ingest-ensrainbow \
  --input-file ${LABEL_SET_ID}_${NEW_VERSION}.ensrainbow \
  --data-dir data-${LABEL_SET_ID}-${NEW_VERSION}

tar -czvf ${LABEL_SET_ID}_${NEW_VERSION}.tgz ./data-${LABEL_SET_ID}-${NEW_VERSION}

# Calculate checksums
sha256sum ${LABEL_SET_ID}_${NEW_VERSION}.ensrainbow > ${LABEL_SET_ID}_${NEW_VERSION}.ensrainbow.sha256sum
sha256sum ${LABEL_SET_ID}_${NEW_VERSION}.tgz > ${LABEL_SET_ID}_${NEW_VERSION}.tgz.sha256sum

# Upload to your label set server
# (implementation depends on your hosting platform)

6. Testing Your Label Set Server

Before publishing, test that your label set server works correctly:

# Set your test server URL
export ENSRAINBOW_LABELSET_SERVER_URL="https://my-label-set-server.com"

# Test downloading .ensrainbow file
./scripts/download-ensrainbow-files.sh my-dataset 0

# Verify checksum was validated
# The script will fail if checksums don't match

# Test downloading prebuilt database
./scripts/download-prebuilt-database.sh 3 my-dataset 0

# Verify the database works
pnpm run ingest-ensrainbow \
  --input-file labelsets/my-dataset_0.ensrainbow \
  --data-dir test-data

pnpm run validate --data-dir test-data

Running Your Own ENSRainbow Server

If you want to run your own ENSRainbow API server (separate from the label set server), see the Local Development guide for instructions on setting up and running ENSRainbow locally or in production.

Related Documentation

• Data Model - Understanding the .ensrainbow file format
• Label Sets & Versioning - Managing label set versions
• CLI Reference - Complete command documentation
• Local Development - Setting up your development environment