Static-packaging

Now that we've completed RCon3, we need to turn our attention to how we might want to support packages for Rholang/RChain. Justin and I took a few moments to collect our thoughts on the issue.

The first thought: focusing on "dpkg/nix/yum" style package managers might not be the best direction to take at this time. These package managers are ideal for keeping track of static and dynamic linking of packages for applications. This is ideal for an operating system, but (for now, at least) isn't necessarily ideal for language use.

Instead, we should probably be taking our inspiration from Python's "pip/virtualenv", JavaScript's "npm" and "yarn" (well, JavaScript should probably serve as a warning for things to avoid doing too), and Common Lisp's "quicklisp". These package managers are designed to manage dependencies for projects, and thus provide tools to both control the version of the language being used, and the versions of individual packages.

To this end, we have three questions to consider:

1. Is there an existing package manager we could write a client for?
2. Is there an existing package standard we could write a client for?
3. Should we create our own standard?

Additionally, we have several requirements we need to consider for our package management system.

1. We need to be able to support multiple versions of Rholang/Solidity.
2. We need ways to support multiple versions of packages.
3. We need to provide support for e-commerce.
4. We need a delivery system of some sort for code and assets. -- tar, zip or git.

We also have a couple of questions that need to be answered.

Q. How will multiple versions of Rholang be handled? (IE, what is the the Rholang equivalent to "pragma"?)

Q. What is the likely syntax for the "let" functional sublanguage? (I would personally expect that the sublanguage should support variables and lambdas independent of the tuple-space, but treated as if they were unforgeable names/processes...)

Q. What, if anything, will be available for macro support?

Q. What can we expect, in terms of syntax, for importing files?

=

I, Alpheus, would like to take a moment to capture some of my thoughts about the discussion we had. (I'm a bit tired at the moment, so I need to take a moment to try to get my thoughts to solidify.)

First of all, I am concerned that we would need to have special syntax for dealing with imports. I would like to do something like this:

new myContract in {
    new subContract(`file:///subcontract.rho`) in {
        // do stuff here using "subContract"
    }
}

I initially thought that this would Just Work(TM), but now that I think of it, there's a serious obstacle with this working: intuitively, I would expect the import to be a strict text substitution, at least for now. Thus, if subcontract.rho looks something like this:

// subcontract.rho
new subContract in {
    contract subContract(hello, done) = {
        // subContract magic
    }
}

then myContract is going to look like this:

new myContract in {
    new subContract(`file:///subcontract.rho`) in {
        // subcontract.rho
        new subContract in {
            contract subContract(hello, done) = {
                // subContract management
            }
        }
        | // do stuff here using "subContract"
    }
}

but this means that // do stuff here using "subContract" is going to be out of the scope of subContract; that, and we now have two subContract names is the same namespace.

I would suppose that we probably need a new keyword, import, that would probably insert the specified namespace into the contract. That is,

new myContract in {
    import subContract(`file:///subcontract.rho`) in {
        // do stuff with subContract
    }
}

will become (with no changes to the above subcontract.rho file)

new myContract in {
    new subContract in {
        contract subContract(hello, done) = {
            // subContract magic
        }
        | // do stuff with subContract
    }
}

The syntax probably needs room for improvement: if "subContract" defines multiple unforgeable names that I would like to use, it would be awkward to have to say something like:

import subContract(`file:///subContract.rho`), anotherContract(`file:///subContract.rho`) in {
    // do stuff here
}

and for that matter, I might want to rename subContract to be another name more convenient for my purposes.

Perhaps something like

import
    from `file:///subContract.rho` subContract as mySubContract, anotherContract
    from `file:///yetAnotherContract.rho` yetAnotherContract
in {
    // do stuff here
}

which will probably translate into something like

new mySubContract, anotherContract, yetAnotherContract in {
    contract mySubContract(etc) = {
        // subContract magic
    }
    | contract anotherContract(etc) = {
        // anotherContract magic
    }
    | contract yetAnotherContract(etc) = {
        // yetAnotherContract
    }

    | // use mySubContract, anotherContract, and yetAnotherContract to
      // my heart's content.
}

Ok, so that was a much bigger "First" than I expected.

Second, my intuition indicates that we're going to want to create our own standard for package management. It's probably even going to be in the syntax. Now that I'm thinking about syntax, I thin what I'd like to see is something like this:

import
    from(`file:///subContract.rho`) subContract as mySubContract, anotherContract
    from(`git://yetAnotherContract.rho`, "10.0.1") yetAnotherContract
    from(`git://eCommerceContract.rho`, "0.5.1", myCredential) eCommerceContract as eContract
in {
    // do stuff here
}

In this example, "eCommerceContract" is a contract that requires a credential of some sort to download it (probably by purchasing it somehow), and is provided through myCredential.

One of the reasons I think this is that Rholang is going to be just different enough, it's going to be easier to support it through Rholang rather than by using yarn, or pip, or quicklisp. Having said that, these package managers aren't all that complex: you declare that you want a package, and optionally a library, and perhaps specify the repository where you expect to get this package. The package manager then pulls the packages you have said you want, matches up the versions, and then saves those packages locally so that you can use them.

Another reason: if we have good syntax support, then we probably don't need an external manager.

I can think of two different approaches to handling packages.

The first is how Python handles things: in the project directory, we use virtualenv to designate which version of Python we use, and then pip is used to download particular versions of the libraries we wish to use. I suspect that the latest version of the library available for the version of the Python we're using is used by default, but I think older versions can be specified from the command line. Once everything is downloaded, the versions are fixed, unless otherwise changed.

I am not aware of any mechanism to keep track of versions for use in other environments, or for updating libraries if desired. I'm fairly certain the packages come from a single repository of some sort, but I'm not sure how that repository is maintained.

I think this is generally how Quicklisp works for Common Lisp, and how Vim's plugins work as well -- except that in these cases, rather than having a single repository to pull from, most of the libraries are maintained in separate repositories (mostly Git on Github).

JavaScript represents another way packages are managed: for every project, we have a packages.json file that lists the packages we wish to have, and allows us to specify the versions of those packages we're willing to use. Then, a packages.json.lock file is created so that we could reproduce the same environment on other systems; this is particularly important because yarn and npm permit us to specify ranges of versions for each package that we would like to use.