Skip to main content

Enterprise schema

The key to solving enterprise problems with ELARA is an end-to-end definition of an enterprise. We call this an called an enterprise schema. In summary, an enterprise schema describes all components of a solution spanning ingestion, transformation, simulation, optimisation, analytics, visualisation and user/system interaction, with respect to data.

Components#

A schema may describe a solution with any of the following components:

ExpressionsExpression components facilitate creation of statically typed functional data expressions (values and variables), as well as providing a set of standard libraries for common expression patterns.
Because all components within an enterprise schema are structured expressions, the enteprise schema is an expression of the enterprise with respect to the available data.
TableTable components facilitate creation and management of streaming tables of expressions. Based on incremental data flow, every table will have a expression for a unique primary key. Rather than creating tables explicitly, schema components output one or more tables.
DatasourceDatasource components facilitate generation of expressions from external systems (such as an api) or files (such as a json file), and generate output tables.
PipelinesPipeline components facilitate creation of chained operations applied to expressions exposed by tables, including tables output from other pipeline. Pipelines evaluate expressions to incrementally cleanse, transform, and combine data in real time, as changes occur.
StructureStructure components facilitate defining the relevant enterprise concepts (such as the processes undertaken by the enterprise) as instances or expressions of tables. Decisions can be defined within structure components, telling ELARA to apply mathematical optimisation to facilitate manual, guided or automatic exploration of business outcomes.
AnalyticsAnalytics components facilitate creation of partitioned in-memory views of tables to load into UI pages in a secure application.
AdminAdmin components facilitate creation and management of UI users, as well as access levels to grant to different roles. The components also provide abstraction of environment variables to manage solution and deployment secrets safely.
ApplicationApplication components facilitate creation of secure web-based user interfaces containing pages of visuals, allowing decision makers to interact in real-time with ELARA and third party systems.

Code#

The ELARASchema Typescript interface provides insight into what an enterprise schema looks like as an object. As described in the cli overview rather than creating a ELARASchema object directly, the EDK manages the lifecycle of a ELARASchema which may also exist in a JSON form.

// the ELARASchema Typescript interfaceexport type ELARASchema = {    // Admin Components    // the collection of Users in the solution    user?: Record<string, User>,    // the collection of EnvironmentVariables in the solution    environment?: Record<string, EnvironmentVariable>        // Data Components    // the collection of streaming Tables in the solution    table?: Record<string, Table>,
    // Datasource Components    // the collection of DataSources in the solution    datasource?: Record<string, DataSource>,        // Pipeline Components    // the collection of Pipelines in the solution    pipeline?: Record<string, Pipeline>,        // Structure Components    // the collection of structure collections in the solution    structure?: Record<string, Structure>    // the collection of structure mappings in the solution     mapping?: Record<string, Mapping>,    // the collection of structure instances in the solution     instance?: Record<string, Record<string, Instance>>,    // the collection of Scenarios in the solution    scenario?: Record<string, Scenario>        // Analytics Components    // the collection of Views in the solution    view?: Record<string, View>,        // Application Components    // the collection of Applications in the solution    application?: Record<string, Application>    // the collection of Pages in the solution    page?: Record<string, Page>,    // the collection of Visuals in the solution    visual?: Record<string, Visual>,
    // the solution name    name?: string,}

Plugins#

As discussed in the cli overview a solution is defined by combining one or more ELARASchema components into a solution, or in other words a full solution ELARASchema is defined by merging multiple partial solution ELARASchemas together.

Because of this fundamental composability of solutions, we are able to create plugins to generate procedural or hard-coded ELARASchema objects.

For example we might create a CustomPlugin in a Typescript module custom-plugin-module and publish to npm. Our CustomPlugin is a function that given a DataSourceDefinition, produces an ELARASchema containing a DataSource, Analytics and Application components (such as Views & Pages), as shown below:

import { ELARASchema, JsonSourceSchema, DataSourceDefinition, TablesPlugin, mergeSchemas } from '@elaraai/edk/lib';
// the only requirement for our function is that it returns an ELARASchema object.export function CustomPlugin(datasource: DataSourceDefinition): ELARASchema {    // create a partial a DataSource component ELARASchema    let datasource_schema = DataSourceSchema(datasource)    // TablesPlugin is a plugin that creates an ELARASchema     //    containing a Pages,Visuals and Views from input tables.    let page_schema = TablesPlugin({         // pass in the tables created for the DataSource        tables: datasource_schema.table,         // chhose some nice looking icon (https://materialdesignicons.com/)        icon: 'upload-network'     });    // merge the schemas together!    return mergeSchemas(datasource_schema,page_schema)}

Now that we have a plugin, we could inject this into a solution that has been created with edk init. For example using the ApplicationPlugin within a plugin created with edk add plugin we could include our CustomPlugin:

import { ApplicationPlugin, Const, Schema, SuperUser } from '@elaraai/edk/lib';
// import our plugin (this could have been published in an npm module alternatively)import { DataSourcePlugin } from 'custom-plugin-module';
// a plugin instance in a project always return a Schemaexport default Schema(    // inject the result of the ApplicationPlugin into the project Schema    ApplicationPlugin({        // give our Application a name        name: "Datasource Application",        // inject the DataSourcePlugin ELARASchema output into our Application        schemas: {            DataSources: DataSourcePlugin(                ArraySource({                    name: 'Array',                    primary_key: Variable('id', 'string'),                    selections: {                        id: Parse(Variable('id', 'string'))                        date: Parse(Variable('date', 'datetime')),                        array: Parse(Variable('object', ArrayType(DictType('float'))),                    },                    rows: [{                        id: "1234",                        date: '2021-01-04T00:02:24.961Z',                        array: [                            { "one": 5.0, "two": 5.0 },                            { "two": 4.0, "three": 5.0 },                        ]                      }]                })            ),        },        // create a user for our Application        users: [            SuperUser({                email: 'super@example.com',                name: 'Admin',                password: Const('admin')            })        ],    }))

The EDK package provides some useful predefined plugins which we will demonstrate when we develop a project.

Summary#

Now that we have defined the overall concepts of a solution, we can define how an enterprise creates value with the structure.