NEAR Protocol is a user-friendly, carbon-neutral blockchain, built from the ground up to be performant, secure, and unparalleled scalability. In technical terms, NEAR is a layer one, sharded, proof-of-stake blockchain built with usability in mind. In simple terms, NEAR is a blockchain for everyone.

Today, we are excited to announce the NEAR BigQuery Public Dataset for anyone who wants to query blockchain data in an easy and cost-effective way.

Why BigQuery Public Dataset

Until now, a user’s data query needs were fulfilled by indexers. Those indexers were either supplied by NEAR Protocol or custom made. To build custom indexers required JSON files from the NEAR Lake storage layer to be transformed and loaded into a target database engine like PostgreSQL, and only then could a user execute queries against it. This approach is complex, time-consuming, and resource-draining. It requires constant monitoring to ensure databases have the most up-to-date information.
NEAR BigQuery Public Dataset changes this. It provides near real-time blockchain data that can be easily queried with SQL.

What we did

We built the NEAR LakeHouse in Databricks. The data is loaded into raw bronze files using Databricks Autoloader, and transformed with Databricks Delta Live Tables into cleaned and enriched silver tables following the Databricks Medallion Architecture. The silver tables are then copied into the GCP BigQuery Public Dataset ready for consumption.

The solution design

The code is open-source and can be found in our GitHub repository: near/near-public-lakehouse

To learn more about how to get started and the data available, please check our documentation:

https://docs.near.org/bos/queryapi/big-query

Benefits

• NEAR instant insights: Historic on-chain data queried at scale.
• Cost-effective: Eliminate the need to store and process bulk NEAR Protocol data; query as little or as much data as preferred.
• Easy to use: No prior experience with blockchain technology is required; bring a general knowledge of SQL to unlock insights.

Conclusion

NEAR BigQuery Public Dataset  is now available for anyone wanting to harness blockchain data for their own needs. BigQuery can help not only developers, but broader audiences including: 

• Users: Create queries to track NEAR Protocol assets, monitor transactions, or analyze on-chain events at a massive scale.
• Researchers: Use indexed data for data science tasks, including on-chain activities, identifying trends, or feeding AI/ML pipelines for predictive analysis.
• Startups: Use NEAR Protocol’s indexed data for deep insights on user engagement, smart contract utilization, or insights across tokens and NFT adoption.

Acknowledgments

We are grateful for the following contributors who helped us to deliver the NEAR BigQuery Public Dataset. 

• NEAR Foundation/Pagoda: Eduardo Ohe, Pavel Kudinov, Jo Yang, Abhishek Anirudhan, Yad Konrad, Olga Telezhnaya, Bohdan Khorolets,Morgan McCauley, Ernesto Cejas Padilla, Rob Tsai, Damián Parrino. 
• Google Cloud: Colleen Pimentel, Rodrigo de Freitas Vale, and Devan Mitchem.
• Databricks: Clayton Martin, and Alice Zhang.