Montreal-based StreamingFast Co-founder and CTO, Alexandre Bourget, made waves at the Messari Summit 2023 with the introduction of Substreams. This indexing technology is poised to transform the way blockchain data is consumed.
StreamingFast is a protocol infrastructure company that focuses on processing and indexing blockchain data at high speeds.
The company has been involved in core protocol development, indexing services, and performance improvements, with a particular focus on Ethereum, Solana, Flow, Polkadot, Cosmos, Polygon, and BNB Chain.
It is also known for its open-source contributions to The Graph ecosystem and collaboration with top-tier sports and entertainment properties.
The Birth of Substreams
Bourget, a data scientist and software architect, highlighted how StreamingFast effectively tackled indexing challenges. He emphasized that the company faced significant hurdles related to cost, reliability, and consistency.
To overcome these obstacles, they leveraged flat files and a streaming-first engine, resulting in exceptional performance.
During his presentation, Bourget shared that after joining the Graph Network, they were able to create a slow-to-sync subgraph in just 10 hours using the technology he introduced.
This remarkable achievement prompted the Graph Network to adopt the technology as the cornerstone of its high-performance and multi-chain strategy.
The technology he presented emphasized two important components: “fire hose” and “substreams.” The concept of the fire hose involves efficiently extracting data from blockchains through a highly parallelized approach.
This method tackles concerns such as cost, reliability, and consistency by leveraging flat files and prioritizing streaming engines.
In contrast, substreams were introduced as a versatile transformation engine for fire hose data, capable of massive parallelization.
It operates in batch mode for historical data processing and in real-time for low-latency applications. According to the speaker, substreams deliver exceptional performance without compromising quality.
Why Substreams
Bourget explained that Substreams was developed to address common challenges in the blockchain industry, such as the need to constantly reinvent solutions for each blockchain protocol and the complexity of understanding smart contracts. This technology offers a unified, open, and programmable indexing stack for organizing blockchain data.
He emphasized that the ultimate goal and mission of Substreams is to create a world where data is open and accessible, supported by a shared intelligence layer that is both cost-effective and high-performing. The aim is to foster global creativity by promoting greater openness in data.
Furthermore, Bourget acknowledged that while the industry’s growth presents opportunities, particularly in attracting new developers, there remains a challenge in integrating diverse tools and experiences due to variations in data availability.
