Data application

Data applications are applications running on the Chainslake platform written by Chainslake itself or other programmers, check out all in Data Apps Store. Data Apps will help expand data (get data from new chains, new data sources), add new transformations… Please checkout Tutorial 1 to learn step by step how to build a Data App.

Example: binance_cex.trade_minute

binance_cex.trade_minute is a data app that is responsible for getting trade data on the binance exchange (by minute) and saving it to the binance_cex.trade_minute table

Clone Chainslake’s data-builder project on github, you can find the job of this app in the jobs/binance/cex directory. Now let’s look at the code of this app, there are a few things you need to know. First, open spark directory on your favorite IDE.

Install and test

This is a Spark project using Scala language, so you need to install the development environment according to the instructions in the Readme.md file then run test and check result data in your local.

Review code

App data will be placed in sub packages of the chainslake package. A sub package includes:

  • Main: The place where the application starts, responsible for loading the general configuration of the application.
  • JobFactory: Navigate to the application corresponding to app_name in the configuration
  • binance.TradeMinute: Processing logic for binance_cex.trade_minute. There are a few things to keep in mind:
    • Every data app must extends from chainslake.job.TaskRun and implement the onProcess method or extends chainslake.job.JobInf and implement the run method.
    • onProcess: function contains the job processing logic, with the parameters from, to passed in indicating the range of data that needs to be processed. Since the trade_minute table has frequent_type as minute, the from and to passed in will be timestamp.

      You should read about table design to know about the frequent_types of tables.

    • getLatestInput, getFirstInput: override these 2 functions only if your job can determine the start and end points of the data, otherwise Chainslake will automatically get this information from the tables in list_input_tables configuration

How to build and submit your app

  1. You need to folk data-builder project to your account on github
  2. Code and test your app on your local
  3. Commit and push your work to your repository
  4. Create Pull request to merge your work to main branch of data-builder