Skip to main content

Send Data

In this tutorial, you will learn how to define a Datasink to send data from a live Solution via File Transfer Protocol (FTP). You will

  • define an FTP Datasink, and
  • launch a Solution and observe the export of data via your Datasink.

The starting point of code for this tutorial is the endpoint of the

previous tutorial. The code is available in the StackBlitz instance below for convenience.

If you wish to use the above code, simply download the Project, navigate to the Project directory and run npm install to set up the Project for development.

Define an FTP Datasink

In the

previous tutorial, you defined a Pipeline Daily Difference in Revenue.CSV which transforms the output Datastream of your Daily Difference in Revenue Pipeline to a BlobType Datastream of CSV format. In this tutorial exercise, you will define a Datasink to automatically export this data via FTP.

You define a Datasink using the

SinkBuilder() class from EDK Core. First, initialise a new SinkBuilder() instance in your src/my_tutorial.ts Asset. Set the target Datastream to be exported as the output Datastream of your Daily Difference in Revenue.CSV Pipeline.

  import { ..., PipelineBuilder, SinkBuilder, Template } from "@elaraai/core"

  ...

  const encode_exercise_one = new PipelineBuilder("Daily Difference in Revenue.CSV")
      .from(select_exercise_one.outputStream())
      .toCsv({
          ...
      })

  const ftp_datasink = new SinkBuilder("Daily Difference in Revenue.CSV")
      .from(encode_exercise_one.outputStream())

  export default Template(
      ...
      ftp_datasink
  )

To define a Datasink that will export data to an FTP File URI, you use the

.ftp() method of SinkBuilder(). As it's first argument, you provide a TypeScript object with configuration properties. The only required property is a uri, though there are additional properties for username, password, private key, etc.

In this tutorial, you will transmit data to a public FTP Test URI,

DLP Test, with the following access details:

uri: ftp://ftp.dlptest.com/
username: dlpuser
password: rNrKYTX9g7z3RgJRmxWuGHbeu

Note that this is a public FTP server courtesy of

DLP Test. It is not maintained or affiliated with ELARA. Use this service at your own risk.

Call the

.ftp() method in your Datasink definition with the above configuration parameters. For the uri parameter, concatenate the uri abive with a file path and name for the exported file (e.g. test_revenue.csv).

...

const ftp_datasink = new SinkBuilder("Daily Difference in Revenue.CSV")
    .from(encode_exercise_one.outputStream())
    .ftp({
        uri: () => Const("ftp://ftp.dlptest.com/test_revenue.csv"),
        username: () => Const("dlpuser"),
        password: () => Const("rNrKYTX9g7z3RgJRmxWuGHbeu")
    })

...
Avoid hardcoded credentials

In a normal situation credentials should be protected, the recommended practice for a real solution is to define crendetials as streams (as shown below), this concept can be extended to uri's.

...

const ftp_username = new SourceBuilder("Username").writable(StringType)
const ftp_password = new SourceBuilder("Password").writable(StringType)

const ftp_datasink = new SinkBuilder("Daily Difference in Revenue.CSV")
    .from(encode_exercise_one.outputStream())
    .input({ name: "Username", stream: ftp_username.outputStream()  })
    .input({ name: "Password", stream: ftp_password.outputStream()  })
    .ftp({
        uri: () => Const("ftp://ftp.dlptest.com/test_revenue.csv"),
        username: (_, inputs) => inputs.Username,
        password: (_, inputs) => inputs.Password,
    })
...

You've now defined an FTP Datasink. In the next tutorial step, you will launch a Solution and test your Datasink. For convenience, a Project containing the code changes above is available in the StackBlitz instance below, ready to install and launch.

Observe Data Export via Datasink

First, build and launch a Solution from your Project.

$ edk template build
ℹ Compiled my_tutorial from src/my_tutorial.ts
✔ build succeeded
$ edk template deploy -ycw send_data
✔ deploy succeeded, Successfully deployed to workspace send_data with id xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

To validate that your Datasink has succeeded in exporting data, you validate the Datasink Task by running edk task logs in the command-line.

$ edk task logs "Sink.Daily Difference in Revenue.CSV" -w send_data

The command-line output indicates that your Task has succeeded and your Datasink has successfully exported data to the FTP server.

Listing task logs for Sink.Daily Difference in Revenue.CSV available in tenant with identifier: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx and workspace send_data.


UUID                                 REASON         STATUS  LEVEL MESSAGE              LOGGEDAT                
xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx new_input_data success info  Executing ftp source 2023-02-12T01:13:49.116Z
xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx new_input_data success info  Executing ftp source 2023-02-12T01:13:53.171Z

Listing task logs for Pipeline.Parse Products available in tenant with identifier: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx and workspace send_data.

To validate at the FTP server side, access the FTP server using the credentials provided:

uri: ftp.dlptest.com
username: dlpuser
password: rNrKYTX9g7z3RgJRmxWuGHbeu

Next steps

In this module, you learnt about Datasinks and learnt how to define a Datasink to send data from a live Solution via File Transfer Protocol (FTP).

In the

next module, you will learn how to generate dashboard visualisations of data and view them in the web browser.

On this page