# Custom Functions and modules in Eve **Note**: *We plan to do a thorough documentation pass in the coming weeks to clean up and document the codebase. This document is a very rough draft intended to help intrepid adventurers navigate the jungle until then.* ## Terms Sheet - **Cardinality** - The number of results a thing would return. - **Intermediate** - A value which is necessary to derive the final result, but not part of the result. - **Referential Transparency** - The property of, for a given input, always returning exactly the same output regardless of time or context. ## Evaluating Eve To understand the API for creating Databases (Eve modules) and Providers (Eve functions), you need to understand a little about our evaluation strategy. In many ways, it is more similar to a database query engine than a conventional executor. To evaluate a query, the compiler breaks it down into a list of joins (called Scans). Functions are no different--to Eve, a function is just a join of its arguments to a mapping of inputs and outputs. These joins are then performed to run your program using an algorithm called Generic Join. Generic Join is capable of avoiding very high cardinality intermediates at the cost of doing more work during the query. In general, this gives significantly more consistent performance and alleviates the need for an advanced query planner like SQL uses. ## Providers In order to let Generic Join work its magic, our function providers (henceforth, Providers) need to do a little bit of extra work. A provider must be able to: 1. Test whether a given value is a valid output for the given input. 2. Calculate the cardinality of the output for a given input. 3. Return the output for a given input. Additionally, providers include a mapping of attribute names for inputs (its `AttributeMapping` and outputs `ReturnMapping`), which tell the evaluator how to invoke your provider and interpret its results. Additionally, the provider has to keep some promises to the runtime or very bad things will happen: 1. A provider must be synchronous 2. A provider must have a finite cardinality 3. A provider must be referentially transparent. Eve is not a lazy language. If #2 is violated, Eve is unable to evaluate it. If #3 is violated, even stranger things happen. Eve may successfully execute the query but end up with a wrong answer. Worse, this problem may only be visible if certain specific sequences of events occur. For an example of what this means, Eve's random function always returns the same value for a given input. To sample new random values over time, new inputs (such as the time or the id of transient objects) must be used. ### Implementation A Provider is created by subclassing the `Constraint` class. It must implement all of the following abstract methods: ``` typescript class MyProvider extends Constraint { // This can optionally be implemented to do any initial setup that may be required. // Just pass the arguments through in a call to super constructor(id: string, args: any[], returns: any[]); // Maps input attributes on the function to the array indexes they should reside in. static AttributeMapping:{[attribute:string]: number}; // Maps output attributes on the function to the array indexes they will reside in. static ReturnMapping:{[attribute:string]: number}; // Returns an object containing the values for the inputs and optionally outputs of the function. resolve(prefix):{args: any[], returns?: any[]}; // Given a variable to solve for and a prefix of solved variables, return // a proposal for that variable abstract getProposal(tripleIndex: TripleIndex, proposed: Variable, prefix: any) : Proposal | undefined; // Resolve a proposal you provided into the actual values for a variable abstract resolveProposal(proposal: Proposal, prefix: any[]) : any[]; // Test if a prefix adheres to the constraint being implemented abstract test(prefix: any) : boolean; } ``` The `resolve` function lets you map from already computed values in the prefix to the attributes you expose on your Provider. The evaluator will not run your provider until all of its required inputs are present in the prefix. To retrieve the input (and possibly output) values for your function: ``` typescript // Let's say you have the following Attribute/ReturnMappings: static AttributeMapping = { "value": 0, "to": 1, } static ReturnMapping = { "converted": 0, } // You can retrieve the value for the "to" attribute with: let {args, returns} = this.resolve(prefix); let to = args[this.AttributeMapping.to]; ``` Retrieving the output values may sound strange, but given Eve's unordered semantics, it's possible that another provider got first dibs on declaring the allowable values for your output(s). In that case, your provider needs to determine if the other provider's value satisfies your output mapping. If not, your provider will fail the test to kill the row. The three abstract methods you must implement are: The `getProposal` method, which determines what the cardinality of the result will be for the given inputs. The provider then updates its `proposalObject` accordingly. If the provider would fail or otherwise have no result, it should assign a cardinality of zero. It returns the proposal object. The `proposalObject` is returned. The `resolveProposal` method actually runs the function on the inputs from the prefix. It returns an array of the outputs for the given input. The `test` method allows a proposal to accept or reject a proposed value as a valid output for its inputs. The `returns` attribute on the result object of `this.resolve(prefix)` is guaranteed to be available here. If the `returns` array contains only valid outputs for each of its attributes, the test returns true, otherwise false. Let's walk through an existing provider as an example: ``` typescript // Urlencode a string class Urlencode extends Constraint { static AttributeMapping = { "text": 0 }; static ReturnMapping = { "value": 0 }; // To resolve a proposal, we urlencode a text resolveProposal(proposal, prefix) { let {args, returns} = this.resolve(prefix); let value = args[this.AttributeMapping["text"]]; let converted; converted = encodeURIComponent(value); return [converted]; } test(prefix) { let {args, returns} = this.resolve(prefix); let value = args[this.AttributeMapping["text"]]; let converted = encodeURIComponent(value); return converted === returns[this.ReturnMapping["value"]]; } // Urlencode always returns cardinality 1 getProposal(tripleIndex, proposed, prefix) { let proposal = this.proposalObject; proposal.cardinality = 1; proposal.providing = proposed; return proposal; } } // ... providers.provide("urlencode", Urlencode); ``` This provider implements the `urlencode` function. It defines a single input attribute, `text`, and an output attribute `value`. - The `resolveProposal` method retrieves the input as above, feeds it through the native `encodeUriComponent` function, and returns its value in the `ReturnMapping["value"]` slot (0). - The `test` function also runs `encodeUriComponent` on its input, but compares it to the already proposed value in `returns`. In some cases it's not necessary to actually evaluate the function to test validity. E.g., square root can't possibly work on a string, so the specific correct value doesn't matter. However, evaluating the function and comparing the result will always return the correct result and is a good place to start. - The `getProposal` function here is pretty boring. Since it is valid to urlencode any eve value (including numbers and booleans), the provider has a constant cardinality of 1. In the case where a value may be invalid (e.g., requiring a number but receiving a string), the `getProposal` function should be the one to catch this. In that case, it should return a cardinality of zero. In the future, we plan to also have a channel available for providers to warn users about bad inputs, but this does not yet exist. Finally, the Provider is registered in the global providers registry, which makes it available for documents to use. ## Databases Databases are Eve's version of modules. They include an Eve document to run, which by convention searches in the Database of the same name. E.g., the editor DB imports a document named `editor.eve` whose blocks look at the `@editor` Database for input. All of this is technically configurable, but its good practice to be as obvious as possible. As usual exceptions apply (e.g., most databases will listen for records in `@event` and may even read/write in databases (e.g. editor writing commands for the editor into the `@browser` DB). In the future, it will be possible to bundle a set of native providers into your DB, but this currently isn't really possible for third parties. **NOTE**: *Databases are in need of a refactor for extensibility. Creating a custom Database is currently a very invasive process. For this reason we are unlikely to accept pull requests for new Databases until the refactor happens except in exceptional cases.* ### Implementation A custom Database is created by subclassing the `Database` class. ``` typescript export class MyDB extends Database { blocks: Block[]; // Used to build the document this Database contains, if any. // In the future, there may be a more elegant mechanism for both this and // supplying providers from within the DB. constructor(); // Invoked when a new evaluation using this DB is opened. register(evaluation: Evaluation); // Invoked when an evaluation using this DB is closed. unregister(evaluation: Evaluation); // Invoked when an evaluation using this DB has completed a change. // `changes` contains the changing state. onFixpoint(currentEvaluation: Evaluation, changes: Changes); } ``` Unless otherwise specified, be sure to invoke the `super`'s method when overriding a method. Let's look at some examples. For a simple, purely native DB, let's look at `@editor` in `src/runtime/databases/browserSession.ts`. ``` typescript export class BrowserEditorDatabase extends Database { constructor() { super(); let source = eveSource.get("/examples/editor.eve"); if(source) { let {results, errors} = parser.parseDoc(source, "editor"); if(errors && errors.length) console.error("Editor DB Errors", errors); let {blocks, errors: buildErrors} = builder.buildDoc(results); if(buildErrors && buildErrors.length) console.error("Editor DB Errors", buildErrors); this.blocks = blocks; } } } ``` We only need to override the constructor here, and the entirety of that is some boilerplate for retrieving and building the source for the document `editor.eve` which powers `@editor`. We unfortunately don't have a nice way to send out errors here yet, so we `console.error` any unexpected errors to at provide some warning of foul play. Finally, we attach the built document's blocks to the DB's `blocks` attribute. Running evaluations using this DB will take these into account as necessary. Next, let's look at a Database that provides native functionality. `@http` provides a simple interface for sending and receiving JSON requests in `src/runtime/databases/http.ts`. ``` typescript export class HttpDatabase extends Database { sendRequest(evaluation, requestId, request) { var oReq = new XMLHttpRequest(); oReq.addEventListener("load", () => { let body = oReq.responseText; let scope = "http"; let responseId = `${requestId}|response`; let changes = evaluation.createChanges(); changes.store(scope, requestId, "response", responseId, this.id); changes.store(scope, responseId, "tag", "response", this.id); changes.store(scope, responseId, "body", body, this.id); let contentType = oReq.getResponseHeader("content-type"); if(contentType && contentType.indexOf("application/json") > -1 && body) { let id = eavs.fromJS(changes, JSON.parse(body), this.id, scope, `${responseId}|json`); changes.store(scope, responseId, "json", id, this.id); } evaluation.executeActions([], changes); }); let method = "GET"; if(request.method) { method = request.method[0]; } oReq.open(method, request.url[0]); if(request.headers) { let headers = this.index.asObject(request.headers[0]); for(let header in headers) { oReq.setRequestHeader(header, headers[header][0]); } } if(request.body) { oReq.send(request.body[0]); } else if(request.json) { let object = this.index.asObject(request.json[0], true, true); oReq.setRequestHeader("Content-Type", "application/json"); oReq.send(JSON.stringify(object)); } else { oReq.send(); } } onFixpoint(evaluation: Evaluation, changes: Changes) { let name = evaluation.databaseToName(this); let result = changes.result({[name]: true}); let handled = {}; let index = this.index; let actions = []; for(let insert of result.insert) { let [e,a,v] = insert; if(!handled[e]) { handled[e] = true; if(index.lookup(e,"tag", "request") && !index.lookup(e, "tag", "sent")) { let request = index.asObject(e); if(request.url) { actions.push(new InsertAction("http|sender", e, "tag", "sent", undefined, [name])); this.sendRequest(evaluation, e, request); } } } } if(actions.length) { setTimeout(() => { // console.log("actions", actions); evaluation.executeActions(actions); }) } } ``` The important bit here is an override on `onFixpoint`, which looks for new records in `@http` that match the signature for a new request. For each of these it adds an `InsertAction` to mark the request as sent and then actually does so. At the end, we *asynchronously* execute any actions we generated to avoid changing the current state until after all Databases have used it. This is vitally important to upholding Eve's guarantee that changes are always executed in T + 1, and without it very bad things could happen. Finally, we need to tell the evaluation to use our new database. In the future this will be detected by searching a DB registry (much like the provider registry) for DBs used in an evaluation's blocks, but for now we hardwire it in `src/runtime/runtimeClient.ts` in the `makeEvaluation` method. On the freshly created evaluation `ev`, we call `registerDatabase(name: string, db: Database)` on instances of each DB we'd like it to use.