CLI reference
The ELARA EDK (ELARA Development Kit) and EDK-IO CLIs are a command-line interfaces to create, develop and deploy ELARA solutions.
Installation#
The EDK and EDK-IO are public npm packages. The installation guide demonstrates how to install both packages to start developing ELARA solutions.
EDK CLI#
Once installed the EDK can be invoked from the command line, for example to read the EDK help:
$ edk --helpUsage: edk [options] [command]
Options: -h, --help display help for command
Commands: init [options] initialise a new ELARA project add add an asset to the project remove [options] <asset> remove an asset from the project links [options] <asset> show the directly linked assets in a project list [options] list the assets in the project update [options] update the assets in the project build [options] build the schema from assets deploy [options] <deployment> deploy the assets in a project help [command] display help for commandProject Structure#
The edk init command produces a workspace folder where a Node.js v14 is initialised along with ELARA project files. The basic directory structure for project is as follows:
Example│ elara.json -- this is the project configuration| schema.json -- this is a json description of the ELARA Schematic| package.json -- a node project artefact| tsconfig.json -- a typescript project artefact└───gen│ │ ... generated files will go here ...│ │ ...│ ││└───src│ │ ... working source files will go here ...│ │ ...│ ││ ...Aside from the schema.json schema file and a elara.json environment project configuration file, the CLI manages the lifecycle of definition and generated code files for assets available in the ELARA Libraries. Note that for every asset that exists there is duality of definition code (*.def.ts) and generated code *.ts. These concepts are critical to understanding the ability to incrementally build solutions using ELARA, so are expanded on below.
Definition Files#
Definition files (*.def.ts) contain typescript definitions for ELARA Schematic assets available in the ELARA Libraries, such as a DataSource definitions. The skeleton files are generated when an asset is added with the EDK. For example:
$ edk add datasource array --name "Example Array" Will create an asset in the elara.json, as well as add a example_array.source.def.ts into the src directory, containing the a definition. As an example given the skeleton example_array.source.def.ts a streaming DataSource can be defined to produce some primitive values:
import { ArraySourceSchema, Variable, Parse } from '@elaraai/edk/lib'
return default ArraySourceSchema({ name: 'Example Array', primary_key: Variable('id', 'string'), selections: { id: Parse(Variable('id', 'string')) date: Parse(Variable('date', 'datetime')), value: Parse(Variable('value', 'number')), }, rows: [{ id: "1234", date: '2021-01-04T00:02:24.961Z', value: 124.0 },{ id: "1235", date: '2021-01-05T00:02:24.961Z', value: 135.0 }]})Generated Files#
The EDK is able to incrementally build *.def.ts files into schematic assets, for example:
$ edk build Will incrementally build any required assets, for example using our example above will produce a generated asset file gen/example_array.source.ts, which which will describe the Array DataSource source asset as a Typescript object literal:
// GENERATED FILE, DO NOT EDITexport default { type: 'array' as const, source_type: 'single_source' as const, name: "Example Array", rows: [{ id: "1234", date: '2021-01-04T00:02:24.961Z', value: 124.0 },{ id: "1235", date: '2021-01-05T00:02:24.961Z', value: 135.0 }], output_table: { name: "Source.Example Array", hash: "f32296eafc965ea94322e1018dbab9b6c2df20e7dc54c6ea717d82c1589930d6", fields: { id: { type: 'string' as const, ast_type: 'Variable' as const, name: "string" }, date: { type: 'datetime' as const, ast_type: 'Variable' as const, name: "date" }, date: { type: 'number' as const, ast_type: 'Variable' as const, name: "value" }, }, primary_key: { type: 'string' as const, ast_type: 'Variable' as const, name: "id" }, partitions: { all: { name: "Source.Example Array.all", partition_key: null, label: null, dir: "desc" as const, }, }, }, // ... // other asset contents // ...}// GENERATED FILE, DO NOT EDITThe build command will also update any related assets in the project schema.json build artefact. The generated files are used by the EDK as a build cache, but also serve an important purpose in solution development. For example with the above gen/example_array.source.ts we are able to use the incremental streaming data produced by the datasource, in some other schematic asset, and conveniently Typescript will maintain type checking since the imported object is an object literal. For example:
$ edk add pipeline --name "Example Aggregation" Will add a Pipeline To apply some streaming incremental transformation to any changes generated by the array data source, for example:
import { ArraySourceSchema, Variable, Parse } from '@elaraai/edk/lib'
// import the generated ArrayDataSourceimport array from '../gen/example_array.source.ts'
return default PipelineSchema({ name: 'Example Aggregation', input_table: array.output_table, operations: [ // get the total of value per week AggregateOperation({ group_field: Variable('Group', 'string'), // group records by week group_value: Print(Floor(array.output_table.fields.date, 'week')), aggregations: { // the week will be unique since its part of the key Week: Unique(Floor(array.output_table.fields.date, 'week')), // sum the values within each group TotalValue: Sum(array.output_table.fields.value), }, }) ]})The same logic applies across an entire solution, definitions create assets that can be used to build other assets, spanning ingestion, transformation, simulation, optimisation, visualisation, and user and system interaction. The EDK manages dependencies to ensure that build occurs in the correct order, and static Typescript type checking and build-time ELARA Expression parsing ensures compliance with language syntax and semantics.
EDK-IO CLI#
Once installed the EDK-IO can be invoked from the command line, for example to read the EDK-IO help:
$ edk-io --helpUsage: edk-io [options] [command]
Options: -V, --version output the version number -h, --help display help for command
Commands: store manage the file store detect detect datasource expressions help [command] display help for command