Much indexing, such wow! Nevermined now includes an integration with The Graph’s hosted service

The speed at which the web3 tech stack is developing is impressive. And rightly so: as an industry we need a more robust toolkit. At Nevermined we are doing our part and we are super proud to announce that, together with our partner Keyko, we have integrated one of the top web3 protocols, The Graph, into Nevermined’s core toolkit.

As web3 developers we all know that feeling: “Dammit, why isn’t there a tool for this?”. Sure enough, we recently ran into one of those ‘dammit’ moments. However, we didn’t try and reinvent the wheel. Instead, we tapped into the open source collaboration spirit of web3 and fixed it, with the help of The Graph’s hosted service.

Web3-style, baby!

Nevermined is a platform that gives organisations the power to make the sharing of data easier, more secure and more governed. With Nevermined you can build bespoke data and analytics networks where different entities can share and monetize their data. Simply put, Nevermined makes walled data easy to share, easy to compute & easy to reward.

Our architecture consists of a few key technical components, but one part of the machine that was causing us headaches recently was our smart contract library. Smart contracts are critical for us in the sense that they record the agreements between a Data Owner and a Data Consumer. They are our USP to handle access control and tokenization in an innovative way (read more in ‘Under The Hood of Nevermined’).

The challenges came up when we were working on an implementation of Nevermined for a DeFi Data project (more to be announced soon). The reason for our frustration was two-fold.

• One, this project required lots and lots of smart contracts, each with quite a few events.
• And two, this project was using our smart contracts on Polygon, which we recently started to support.

Obviously, it’s amazing that Polygon allows for faster block times, but combined with our extensive stack of smart contracts, we were running into rate limit issues. To service the front end of the dapp and make data available to the end users, our infrastructure had to make too many on-chain queries.

Result 1: chain says ‘hold on’, causing wheels of death and a slow, unresponsive interface.

Result 2: “Dammit, why isn’t there a tool for this?”

99 PROBLEMS, BUT A QUERY LIMIT AIN’T ONE

After breathing in and out a few times, we remembered that our partners at Keyko had some good experiences with The Graph in the past.

The Graph is the indexing and query layer of web3. This makes it easier to query data that would normally be difficult to query directly, i.e. on-chain. In other words, it’s a library of open-source APIs for accessing blockchain data more easily.

For simple, one-dimensional data points, the Nevermined software can use on-chain queries. This could be questions like ‘Who’s the owner of token xyz’ or ‘what is the total supply of token contract abc’? That’s not too difficult.

But for more complex, aggregate data points, this doesn’t work, especially not if the Layer 2 solutions scale even further and the data queries grow exponentially. With Nevermined we want users to get answers to questions like:

• What was the average price of a trading pair over the last 7 days?
• What is the correlation between BAYC traits and trading activity in the last month, so I can make informed NFT investment strategies?
• How does our dapp/product benchmark on a certain KPI against industry averages?
• Give me a notification when a certain account acquires an NFT, i.e. you subscribe to a future event

Our ambition is that we want to enable advanced, proper data analytics. But, in order to provide answers to that type of questions in a web3 environment, we’d have to get access to that data in a more direct way.

Yes, we could crawl the data, store it locally and use standard web API tech to query that data. But that would be soooooo web2 — and that’s not an option. But then we realised Subgraphs are the solution for us.

Subgraphs are the core backbone of The Graph. We see them as the legs of our smart contracts, i.e. they’ll do the required running. They’re bespoke scripts that are made for specific smart contracts and that can be queried by a dapp with a standard GraphQL API.

The way it works is elegant in its design: each Subgraph identifies the important events of a contract and then goes out into the on-chain world to index the data related to those events.

If you feel ready for a deep dive on The Graph, their documentation will give you plenty of joy.

GRAPH TOOLS NEVERMINED TOOLS = POWER TOOLS FOR DATA

Why did we decide to use The Graph? We did look at other options, but The Graph had a few advantages.

One, they have been working on their code base for years and it shows. They’ve developed a standard for indexing.

Two, they have developed a set of tools, which made it easier for us to stand on the shoulders of giants. We basically used all of them.

• the Graph Typescript Library to write the actual Subgraphs
• the Codegen Graph TS to generate a TypeScript to query a subgraph
• the Graph CLI, which was particularly helpful for interacting with The Graph network and deploy the subgraphs to the indexer nodes.

Three, they understand the importance of collaboration and ecosystem growth in the space, so they were kind enough to support us with a grant.

But after the initial work, we’re repaying the favour by integrating our Subgraphs into our Nevermined toolkit. In the past all developers that worked with Nevermined had to rely on the Ethereum JSON-RPC, which admittedly was slow and unpredictable when it came to rate limits.

Integrating these Subgraphs into the core of the Nevermined SDK now gives developers 2 advantages.

One, it allowed us to kill the old way of handling Nevermined events. The grant allowed us to create no less than 40 Subgraphs. Together they index more than 120 events generated by the Nevermined smart contracts: from Access Control to NFTs and data provenance.

Two, when we amend our smart contracts, the Subgraphs get updated too.

So, from now on all web3 developers that use Nevermined to build data sharing solutions, get the Subgraph indexing functionality out of the box.

Web3 = Win-Win-Win

“The Graph Foundation is delighted to award a grant to Nevermined, whose work developing subgraphs will serve as a helpful public resource for apps looking to organise web3 data in an easy-to-read way, while providing examples for other subgraph developers to build upon,” shares Reem Chahrour, Ecosystem Manager of Grants at The Graph Foundation. “Subgraphs are open source, so these contributions will benefit the community as a whole.”

FROM DAMMIT TO WAGMI

Result 1: no more wheels of death.

But result 2 is that web3 entrepreneurs can truly start working on unlocking the value of web3.

Tools are only a means to an end. And apart from the obvious increase of robustness on an infrastructure level, the result of this integration with The Graph is what we believe to be a powerful combo on a conceptual and business level.

With The Graph, you can index relevant raw data sets and make them readable or available at scale. Then, with Nevermined’s toolkit, you can start being creative with that data: you can monetize the data, create aggregate analytics and create new data sets, invent new value streams and come up with new business models.

If you want to explore our Github repos, have a peek here and here.

Originally posted on 2022-05-12 on Medium.