Sparkscan

Overview

Sparkscan is a full-stack block explorer and data platform for the Spark network - a Bitcoin Layer 2 built on statechains and virtual UTXOs. It provides real-time transaction tracking, address analytics, token pages, and historical balance charts across the entire network.

The system is made up of roughly ten services spanning five languages, connected through a streaming data pipeline on RisingWave. I handled most of the development.

Architecture

Data pipeline

The core of Sparkscan is a CDC-based streaming pipeline. The Spark node writes to a PostgreSQL database on AWS RDS. RisingWave ingests that data via postgres-cdc connectors, creating live replicas of the source tables.

From there, a dbt project running on RisingWave defines a layered model graph:

• Imports pull raw tables from the CDC source
• Pre/staging layers clean and normalize the data
• Intermediate models join and enrich (e.g. resolving token metadata, computing address stats)
• Fact models produce analytics like hourly balance snapshots and live net worth
• Aggregate models roll up balances and stats per address
• Sinks push the final materialized views back into a PostgreSQL database that the API reads from

All models are materialized views in RisingWave, so they update incrementally as new data arrives - there’s no batch processing.

Real-time updates

A Python service called sparkscan-updates subscribes to RisingWave materialized views using the risingwave-py subscription API. When rows change, it transforms the data and fans it out to publishers:

• Centrifugo for WebSocket delivery to the frontend
• Cloudflare Durable Objects for edge-local state
• Webhooks for external consumers

Handlers cover transactions, balances, token transfers, lightning invoices, and more. Each handler runs in its own thread, subscribing to a specific materialized view.

API layer

Two services serve the API:

• sparkscan-gw - a Go gateway built on chi, handling API key management, rate limiting, metering, and billing via Schematic. It uses ent as its ORM and Redis for auth and rate-limit state.
• sparkscan-elysia - a Bun/Elysia REST API that serves the actual chain data: addresses, tokens, transactions, historical stats. It reads from the PostgreSQL sink database populated by the RisingWave pipeline.

API documentation is generated from OpenAPI specs and served via Zudoku.

Frontend

The explorer itself is a Next.js app with Clerk authentication, i18n via next-intl, and Convex for real-time features. Kubb auto-generates typed API client hooks from the gateway’s OpenAPI spec. UI is built with shadcn/ui, Radix primitives, and Recharts for data visualization.

Observability and internal tooling

Eye of Sauron is an internal dashboard built with Next.js, ClickHouse, and MapLibre/Three.js. It provides a 3D globe visualization of network activity and geospatial analytics.

A separate Rust + TypeScript Cloudflare Workers monorepo collects data from the Spark network and inserts it into ClickHouse for the internal dashboard.

Batch pipelines

Apache Airflow runs scheduled jobs for data that doesn’t come through the CDC stream: BTC price fetches from CoinGecko, pool snapshots, token icon syncs, and net worth calculations. These DAGs write to Cloudflare R2 and the pipeline databases.

Tech stack