Note: We plan to do a thorough documentation pass in the coming weeks to clean up and document the codebase. This document is a very rough draft intended to help intrepid adventurers navigate the jungle until then.

• Cardinality - The number of results a thing would return.
• Intermediate - A value which is necessary to derive the final result, but not part of the result.
• Referential Transparency - The property of, for a given input, always returning exactly the same output regardless of time or context.

To understand the API for creating Databases (Eve modules) and Providers (Eve functions), you need to understand a little about our evaluation strategy. In many ways, it is more similar to a database query engine than a conventional executor. To evaluate a query, the compiler breaks it down into a list of joins (called Scans). Functions are no different--to Eve, a function is just a join of its arguments to a mapping of inputs and outputs. These joins are then performed to run your program using an algorithm called Generic Join. Generic Join is capable of avoiding very high cardinality intermediates at the cost of doing more work during the query. In general, this gives significantly more consistent performance and alleviates the need for an advanced query planner like SQL uses.

In order to let Generic Join work its magic, our function providers (henceforth, Providers) need to do a little bit of extra work. A provider must be able to:

1. Test whether a given value is a valid output for the given input.
2. Calculate the cardinality of the output for a given input.
3. Return the output for a given input.

Additionally, providers include a mapping of attribute names for inputs (its AttributeMapping and outputs ReturnMapping), which tell the evaluator how to invoke your provider and interpret its results. Additionally, the provider has to keep some promises to the runtime or very bad things will happen:

1. A provider must be synchronous
2. A provider must have a finite cardinality
3. A provider must be referentially transparent.

Eve is not a lazy language. If #2 is violated, Eve is unable to evaluate it. If #3 is violated, even stranger things happen. Eve may successfully execute the query but end up with a wrong answer. Worse, this problem may only be visible if certain specific sequences of events occur. For an example of what this means, Eve's random function always returns the same value for a given input. To sample new random values over time, new inputs (such as the time or the id of transient objects) must be used.

A Provider is created by subclassing the Constraint class. It must implement all of the following abstract methods:

class MyProvider extends Constraint {
  // This can optionally be implemented to do any initial setup that may be required.
  // Just pass the arguments through in a call to super
  constructor(id: string, args: any[], returns: any[]);

  // Maps input attributes on the function to the array indexes they should reside in.
  static AttributeMapping:{[attribute:string]: number};
  // Maps output attributes on the function to the array indexes they will reside in.
  static ReturnMapping:{[attribute:string]: number};

  // Returns an object containing the values for the inputs and optionally outputs of the function.
  resolve(prefix):{args: any[], returns?: any[]};

  // Given a variable to solve for and a prefix of solved variables, return
  // a proposal for that variable
  abstract getProposal(tripleIndex: TripleIndex, proposed: Variable, prefix: any) : Proposal | undefined;

  // Resolve a proposal you provided into the actual values for a variable
  abstract resolveProposal(proposal: Proposal, prefix: any[]) : any[];

  // Test if a prefix adheres to the constraint being implemented
  abstract test(prefix: any) : boolean;
}

The resolve function lets you map from already computed values in the prefix to the attributes you expose on your Provider. The evaluator will not run your provider until all of its required inputs are present in the prefix. To retrieve the input (and possibly output) values for your function:

  // Let's say you have the following Attribute/ReturnMappings:
  static AttributeMapping = {
    "value": 0,
    "to": 1,
  }
  static ReturnMapping = {
    "converted": 0,
  }

  // You can retrieve the value for the "to" attribute with:
  let {args, returns} = this.resolve(prefix);
  let to = args[this.AttributeMapping.to];

Retrieving the output values may sound strange, but given Eve's unordered semantics, it's possible that another provider got first dibs on declaring the allowable values for your output(s). In that case, your provider needs to determine if the other provider's value satisfies your output mapping. If not, your provider will fail the test to kill the row.

The three abstract methods you must implement are:

The getProposal method, which determines what the cardinality of the result will be for the given inputs. The provider then updates its proposalObject accordingly. If the provider would fail or otherwise have no result, it should assign a cardinality of zero. It returns the proposal object. The proposalObject is returned.

The resolveProposal method actually runs the function on the inputs from the prefix. It returns an array of the outputs for the given input.

The test method allows a proposal to accept or reject a proposed value as a valid output for its inputs. The returns attribute on the result object of this.resolve(prefix) is guaranteed to be available here. If the returns array contains only valid outputs for each of its attributes, the test returns true, otherwise false.

Let's walk through an existing provider as an example:

// Urlencode a string
class Urlencode extends Constraint {
  static AttributeMapping = {
    "text": 0
  };
  static ReturnMapping = {
    "value": 0
  };

  // To resolve a proposal, we urlencode a text
  resolveProposal(proposal, prefix) {
    let {args, returns} = this.resolve(prefix);
    let value = args[this.AttributeMapping["text"]];
    let converted;
    converted = encodeURIComponent(value);
    return [converted];
  }

  test(prefix) {
    let {args, returns} = this.resolve(prefix);
    let value = args[this.AttributeMapping["text"]];

    let converted = encodeURIComponent(value);

    return converted === returns[this.ReturnMapping["value"]];
  }

  // Urlencode always returns cardinality 1
  getProposal(tripleIndex, proposed, prefix) {
    let proposal = this.proposalObject;
    proposal.cardinality = 1;
    proposal.providing = proposed;
    return proposal;
  }
}

// ...

providers.provide("urlencode", Urlencode);

This provider implements the urlencode function. It defines a single input attribute, text, and an output attribute value.

• The resolveProposal method retrieves the input as above, feeds it through the native encodeUriComponent function, and returns its value in the ReturnMapping["value"] slot (0).
• The test function also runs encodeUriComponent on its input, but compares it to the already proposed value in returns. In some cases it's not necessary to actually evaluate the function to test validity. E.g., square root can't possibly work on a string, so the specific correct value doesn't matter. However, evaluating the function and comparing the result will always return the correct result and is a good place to start.
• The getProposal function here is pretty boring. Since it is valid to urlencode any eve value (including numbers and booleans), the provider has a constant cardinality of 1. In the case where a value may be invalid (e.g., requiring a number but receiving a string), the getProposal function should be the one to catch this. In that case, it should return a cardinality of zero. In the future, we plan to also have a channel available for providers to warn users about bad inputs, but this does not yet exist.

Finally, the Provider is registered in the global providers registry, which makes it available for documents to use.

Databases are Eve's version of modules. They include an Eve document to run, which by convention searches in the Database of the same name. E.g., the editor DB imports a document named editor.eve whose blocks look at the @editor Database for input. All of this is technically configurable, but its good practice to be as obvious as possible. As usual exceptions apply (e.g., most databases will listen for records in @event and may even read/write in databases (e.g. editor writing commands for the editor into the @browser DB).

In the future, it will be possible to bundle a set of native providers into your DB, but this currently isn't really possible for third parties.

NOTE: Databases are in need of a refactor for extensibility. Creating a custom Database is currently a very invasive process. For this reason we are unlikely to accept pull requests for new Databases until the refactor happens except in exceptional cases.

A custom Database is created by subclassing the Database class.

export class MyDB extends Database {
  blocks: Block[];

  // Used to build the document this Database contains, if any.
  // In the future, there may be a more elegant mechanism for both this and
  // supplying providers from within the DB.
  constructor();

  // Invoked when a new evaluation using this DB is opened.
  register(evaluation: Evaluation);

  // Invoked when an evaluation using this DB is closed.
  unregister(evaluation: Evaluation);

  // Invoked when an evaluation using this DB has completed a change.
  // `changes` contains the changing state.
  onFixpoint(currentEvaluation: Evaluation, changes: Changes);
}

Unless otherwise specified, be sure to invoke the super's method when overriding a method.

Let's look at some examples. For a simple, purely native DB, let's look at @editor in src/runtime/databases/browserSession.ts.

export class BrowserEditorDatabase extends Database {
  constructor() {
    super();
    let source = eveSource.get("/examples/editor.eve");
    if(source) {
      let {results, errors} = parser.parseDoc(source, "editor");
      if(errors && errors.length) console.error("Editor DB Errors", errors);
      let {blocks, errors: buildErrors} = builder.buildDoc(results);
      if(buildErrors && buildErrors.length) console.error("Editor DB Errors", buildErrors);
      this.blocks = blocks;
    }
  }
}

We only need to override the constructor here, and the entirety of that is some boilerplate for retrieving and building the source for the document editor.eve which powers @editor. We unfortunately don't have a nice way to send out errors here yet, so we console.error any unexpected errors to at provide some warning of foul play. Finally, we attach the built document's blocks to the DB's blocks attribute. Running evaluations using this DB will take these into account as necessary.

Next, let's look at a Database that provides native functionality. @http provides a simple interface for sending and receiving JSON requests in src/runtime/databases/http.ts.

export class HttpDatabase extends Database {

  sendRequest(evaluation, requestId, request) {
    var oReq = new XMLHttpRequest();
    oReq.addEventListener("load", () => {
      let body = oReq.responseText;
      let scope = "http";
      let responseId = `${requestId}|response`;
      let changes = evaluation.createChanges();
      changes.store(scope, requestId, "response", responseId, this.id);
      changes.store(scope, responseId, "tag", "response", this.id);
      changes.store(scope, responseId, "body", body, this.id);
      let contentType = oReq.getResponseHeader("content-type");
      if(contentType && contentType.indexOf("application/json") > -1 && body) {
        let id = eavs.fromJS(changes, JSON.parse(body), this.id, scope, `${responseId}|json`);
        changes.store(scope, responseId, "json", id, this.id);
      }
      evaluation.executeActions([], changes);
    });
    let method = "GET";
    if(request.method) {
      method = request.method[0];
    }

    oReq.open(method, request.url[0]);

    if(request.headers) {
      let headers = this.index.asObject(request.headers[0]);
      for(let header in headers) {
        oReq.setRequestHeader(header, headers[header][0]);
      }
    }

    if(request.body) {
      oReq.send(request.body[0]);
    } else if(request.json) {
      let object = this.index.asObject(request.json[0], true, true);
      oReq.setRequestHeader("Content-Type", "application/json");
      oReq.send(JSON.stringify(object));
    } else {
      oReq.send();
    }
  }

  onFixpoint(evaluation: Evaluation, changes: Changes) {
    let name = evaluation.databaseToName(this);
    let result = changes.result({[name]: true});
    let handled = {};
    let index = this.index;
    let actions = [];
    for(let insert of result.insert) {
      let [e,a,v] = insert;
      if(!handled[e]) {
        handled[e] = true;
        if(index.lookup(e,"tag", "request") && !index.lookup(e, "tag", "sent")) {
          let request = index.asObject(e);
          if(request.url) {
            actions.push(new InsertAction("http|sender", e, "tag", "sent", undefined, [name]));
            this.sendRequest(evaluation, e, request);
          }
        }
      }
    }
    if(actions.length) {
      setTimeout(() => {
        // console.log("actions", actions);
        evaluation.executeActions(actions);
      })
    }
  }

The important bit here is an override on onFixpoint, which looks for new records in @http that match the signature for a new request. For each of these it adds an InsertAction to mark the request as sent and then actually does so. At the end, we asynchronously execute any actions we generated to avoid changing the current state until after all Databases have used it. This is vitally important to upholding Eve's guarantee that changes are always executed in T + 1, and without it very bad things could happen.

Finally, we need to tell the evaluation to use our new database. In the future this will be detected by searching a DB registry (much like the provider registry) for DBs used in an evaluation's blocks, but for now we hardwire it in src/runtime/runtimeClient.ts in the makeEvaluation method. On the freshly created evaluation ev, we call registerDatabase(name: string, db: Database) on instances of each DB we'd like it to use.