General-grants-program: Polkadot Runtime Environment: Alternative Implementation Grant

Is there a preferred language for the implementation?

An alternate implementation in Rust or Go would be very useful early on, since a lot of the current blockchain stack is written in one or the other.

Down the road, it would be interesting to get an implementation in JavaScript (if it ever supports compilation to wasm) or Java/C#.

I'm due to release more information tomorrow (a little behind expected timeframe!).

The reference implementation is in Rust (i.e. Parity Tech), plus there is Jaco's JS implementation (a full Substrate client, but without block authoring).

The preferred implementations are: Java, Go, C++.

So, with some luck, I can release the full details tomorrow! :-)

@EdwardAThomson Are there timelines for the grant applications?

I'm seeking to help a team in this endeavour, specially using Go. If there's an existing team that is looking for help, please contact me. gfrеⅾеrісο аt іģսаzս ḋοt ѕοⅼսtіоոѕ or on Twitter: ģсѕfrеḋ.

@gcsfred2 The applications will be open for a while. At the moment there is a hope to have a couple more implementations, so the application process will remain open at least until we have a few interested teams working on this issue.

Hello, our team was in Berlin during blockchain week and had an incredible time getting to know people from the Parity team. We showed interest in writing an implementation of the PRE in Go. I will be filling out the application over the next few days. Thank you!

Hey ChainSafe team, that's great to hear! I see you have an email address on your profile. I will drop an email to say hello. Go is one of our preferred languages, so your application is of interest. We are starting to get quite a bit of interest now in writing alternative implementations.

Keen to be involved in core development of Polkadot. Form submitted! :)

Adding a list of relevant videos and blogs.

Videos:

• Gavin Wood presents on Polkadot at Ethereum London (June 2017)
• Gavin Wood presents Parity Substrate at EventHorizon18 (Berlin)
  
  
  
  • The slides: https://slides.com/paritytech/paritysubstrate#/
  
  
• Gavin Wood presents Substrate and Polkadot Governance at EdCon (Toronto)
  
  
  
  • Slides: https://slides.com/paritytech/polkadot-governance#/
  
  
• Robert Habermeier also presented Parity Substrate at the Berlin Truebit Meetup

Blogs etc:

• "What is Substrate?" Jack Fransham explains.
  "Substrate in a nutshell" by Dmitriy Kashitsyn.
• W3F wiki page for Polkadot Runtime Environment.

Technical Chat channel:

• Substrate Technical on Riot.

We'd recommend jumping into the Substrate Technical channel and start asking questions. Some developers are on-hand and answering questions, plus there is also a handful of community members who are becoming experts too!

Thanks for all of the interest so far. We have received a number of application forms, but the application process is still open in case any teams are wondering and were thinking of submitting a proposal. We are hoping to select 2 or 3 teams to develop alternative implementations, but we haven't selected anyone yet!

A further update on this topic. Many of you have been asking about timescales and the amount of effort involved.

I had a quick chat with Jaco, a dev in Parity. He is the one who created the implementation in JavaScript. Hopefully this gives you a better idea. He worked pretty much solo too!

How long would you expect a dev / team of devs to create an implementation of Substrate?

"Difficult to say - the issue is that my implementation found a lot of bugs in the Rust libp2p code, so I was waiting for (what felt like forever), for fixes. Too early.

So first thing, what language - if not JS, Go or Rust, they need to write their own libp2p implementation. (Maybe there is a way to use the Go/Rust versions as libraries)"

"Then the tricky bits are:

• implementation of Parity codec (2+ weeks to do properly) - needed for anything networking or RPC related
• implementation of a Trie DB - they can learn from what is done in JS, Rust, Ethereum clients, however Substrate has their own version. (Once again, it may be possible to use the Rust version as a lib)
• WASM environment, this stuff is not documented anywhere but in the code - took me around 1+ month here, but once again, implementations changed the whole time (too early)
  Then there are the bits and bobs - blockchain DB (over-and-above state trie), RPC calls, potential light client."

As a bare minimum, as he see it:

• Trie
• libp2p
• WASM environment

"If in the implementation is in Go, the Trie they can actually take the Ethereum version and "just" drop in blake2 and the parity-codec."

For this grant, we are looking for the consensus part to get done too.

Hello, just filled out the form for C++ implementation.

Hello, just filled out the form for C++ implementation.

Received! Thank you! @Yuvasee

The application process is still open. We will take a few more applications before making any decisions.

Did you know about our W3S conference? There are now reduced price and free tickets available:

https://medium.com/web3foundation/endowing-an-open-inclusive-community-at-web3-summit-3013456f4f84

Hi everyone, we've had some questions about the development process and we thought it would be good to share the answers with everyone.

1. Understanding the motivation for an alternative implementation

• To what extend is the present implementation a reference?

  • There is a specification is in a draft state. Hope to share it soon.
  • The current implementation is like a reference spec, but it isn't really.
  • RFP was started early, so we could start talking with teams.
  • We suggested that they get a proposal in early.

• An independent implementation is sought? In that case, is it OK if the new implementation is based on the Rust implementation of Substrate?

  • As independent as possible. Done in an idiomatic way that suite that language.
  • Not a direct translation.
  • Empower an additional community of developers. Don't lock to a particularly set of developers.
  • Also can help to identify ongoing discrepancies between implementations.
  • As it is a decentralised network, we need alternative implementations. Don't want to have one team control the development of the network.

• Better performance is sought? If so, what are the requirements/symptoms that we want to improve/cure?

  • Not really. Rust is fast enough. We really want to reach new developer communities and to improve decentralisation (network resiliency and governance).

• Better portability for multiple platforms?

  • Expect it to work on Linux and Mac. (Unix like platforms)
  • Would be great on Windows.

• Is there a rationale behind the preference for Go?

  • Want to include the existing Go community.
  • There is a Wasm interpreter in Go already. So it might be easier: https://github.com/perlin-network/life
  • We will probably fund at least 2 implementations. E.g. C++ and Go.
  • There is more to write in C++. Probably less reusable code. Possibly no libp2p in C++. Less Wasm support.

2. Understanding the implementation around WASM

• Can we re-use an existing WASM implementation?

  • Can re-use an existing interpreter in the language of the implementation.
  • It may be possible to use the Rust implementation with a C++ PRE, as there are bindings, but would be better to have a C++ interpreter. If the C++ implementation isn't great, then either make pull requests or fork and improve.

• Are we required to also implement native execution, or just the WASM interpreter?

  • Not necessary for spec compliance.
  • Can do an additional step.
  • There will be pieces of Wasm logic run frequently. Having the ability to run the logic as native code is important.

3. Understanding the “rules of engagement”

• Do formal specifications or acceptance tests exist, or could they be developed as a first phase of the project?

  • We need these, but don't have them.
  • Would be willing to consider the tests as a separate work package.

• Is there any formal deadline?

  • No deadline.
  • Ideally, an implementation by end of Q1 2019. Allows the implementation to start working with Polkadot.

• In what form should we apply for a grant and to whom should the request be addressed?

  • Application form (as in original post).
  • Best to get a rough proposal in earlier. Even if it is not very well fleshed out.

• Who will give the OK / decide at the end for the grant application?

  • Web3 Foundation. Peter will be the lead, but others will be involved (e.g. relevant people from Parity).
  • We may ask for additional information in order to clarify some things.

In progress specification: https://github.com/w3f/polkadot-re-spec/blob/master/polkadot_re_spec.pdf

We have an interest in supporting this effort by doing a minimal port of the implementation of Parity codec, and any other rust modules with the idea to support writing a runtime in Go. So not a full alternative implementation.

This task of course will be taken on by the team selected to do the full Go implementation. So I was looking to see if there is possibility of collaboration or if no work has started at all, to perhaps apply for a grant for a narrower scope of work?