Versions Compared

Old Version 4

changes.mady.by.user Matthew Krafczyk

Saved on

compared with

New Version 5

changes.mady.by.user Matthew Krafczyk

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

...

Here, the expression is evaluated with a constant key. This means the Unit Type with id 'lumen' will always be used for the lumensUOM field of the destination type.

C3.ai resources on Canonical Transform Types

Conclusion

Once you've defined a transform, when data arrives in a Canonical Type, the appropriate transform will be called to populate the data model. You can actually define multiple transforms as well for the same Canonical Type. So you can populate multiple data model types from the same Canonical Types.

C3.ai resources on Canonical Transform Types

...

For instance, for the SmartBulb types we can see that the SmartBulb type includes several other types like 'Manufacturer' mentioned earlier. This allows us to for instance, select SmartBulbs based on Manufacturer.

Basic Data Sources

CSV Files

JSON Files

Seed Data

Sending Data Via POST

Complex Data Sources

Custom External Database

C3 Supported Database technologies

...

Now that we know what the C3 AI Suite does with data once it enters through a Canonical Type, we can explore how to get data to this entry point.

CSV Files

Probably the easiest method is through a CSV file. Simply define a csv file with appropriate column names, create a Canonical type whose fields match those names, and C3 can use this data to create Canonical Types. Consider the SmartBulb.csv seed data (we'll get into seed data soon) located in the lightbulbAD package at 'seed/CanonicalSmartBulb/SmartBulb.csv'.

Code Block
Manufacturer,BulbType,Wattage,SN,StartDate,Latitude,Longitude
Bell,LED,9.5,SMBLB1,2011-01-01T04:00:00-0800,37.48596719,-122.2428196
GE,LED,9.5,SMBLB2,2011-01-01T04:00:00-0800,37.49115273,-122.2319523
GE,LED,9.5,SMBLB3,2011-01-01T04:00:00-0800,37.49099652,-122.2324551
...

If this file is placed or sent to the right place, the C3 AI Suite will 'ingest' it which will start the Data Integration system described above.

JSON Files

Data can also be sent in the JSON format. Generally, the JSON format can have two formats:

First, we define an array of a specific Type:

Code Block
{
  "type": "[Unit]",
  "value": [
    { "id": "PCS", "name": "pieces", "symbol": "PCS", "concept": "count" },
    ...
  ]
}

Second, we define an array of multiple possibly different types:

Code Block
[
  {
    "type": "Unit",
    "id": "PCS",
    "name": "pieces",
    "symbol": "PCS",
    "concept": "count"
  },
  {
    "type": "Unit",
    ...
  },
  ...
]

Consider the following data which can work with the lightbulbAD package:

Code Block
[
  {
    "status": 1,
    "end": "2012-10-23T12:00:00.000-07:00",
    "temperature": 121,
    "power": 63,
    "lumens": 14,
    "start": "2012-10-23T11:00:00.000-07:00",
    "parent": {
      "id": "SBMS_serialNo_SMBLB74"
    },
    "voltage": 944.6,
    "type": "SmartBulbMeasurement",
    "id": "SBMS_serialNo_SMBLB74#BJ"
  },
  {
    "status": 1,
    "end": "2013-09-12T22:00:00.000-07:00",
    "temperature": 13,
    "power": 58,
    "lumens": 919.1,
    "start": "2013-09-12T21:00:00.000-07:00",
    "parent": {
      "id": "SBMS_serialNo_SMBLB74"
    },
    "voltage": 120,
    "type": "SmartBulbMeasurement",
    "id": "SBMS_serialNo_SMBLB74#V"
  },

Seed Data

Now that we've discussed some basic data formats, we can talk about where to put the data. The first place is the 'seed' directory in your C3 package. Place your seed data file in a subdirectory of the 'seed' folder with the same name as the canonical type the data is destined for. For example:

Code Block
seed/CanonicalSmartBulb/SmartBulb.csv

Now, when you provision your C3 Package, C3 will take this file and run it through the Data Ingestion pipeline.

One negative to this method is it's primarily for small amounts of data. Any data you send in this method will bloat your C3 package, and if you choose to use the browser provisioner, this data will be loaded into browser memory which can be quite limiting. We'll now discuss another method to send data to your C3 cluster which can work with practically unlimited amounts of data.

Sending Data Via POST and RESTful API

C3 creates a REST API endpoint for each Canonical Type. This is accessible through the path: 'https://<vanity_url>/import/1/<tenant>/<tag>/<CanonicalType>/<FileName>'.

  • <vanity_url>: Your vanity_url
  • <tenant>: Your tenant
  • <tag>: Your tag
  • <CanonicalType>: The target canonical type.
  • <FileName>: Destination name of the file.

You can send data to this endpoint with an HTTP PUT.

When sending the HTTP PUT, ensure the following headers are defined:

  • Authorization: An auth token generated by 'Authenticator.generateC3AuthToken()'
  • Content-Type: 'text/csv' for CSV file, 'application/json' for JSON file.

This import API endpoint acts to essentially copy a file from your local file system into the C3 System. C3 will associate this file to your tenant/tag and Canonical Type. It will remember the <FileName> in case you try to send it again.

Helper Scripts and Tools

There are a couple of tools you can use to execute these HTTP PUT commands.

send-file.py: DTI Developed helper script

If you download the git repository available here: https://github.com/c3aidti/c3-helper-scripts

We can use the script `send-file.py`. This script uses the python requests module to form the HTTP PUT command. Helpfully, it also reports percent completion which is great if the file you're uploading is large or your internet connection is slow. To use this command, first clone the github repo, then execute the script inside with python (assuming you've installed the requests module). You want to execute it as follows:

Code Block
python send-file.py --vanity-url <vanity_url> --tenant <tenant> --tag <tag> --api-endpoint <endpoint> --file <path_to_file> --auth-token <auth_token>

Replace the placeholders as follows:

  • <vanity_url>: Your vanity_url
  • <tenant>: Your tenant
  • <tag>: Your tag
  • <endpoint>: The 'location' you want to copy the file to after 'https://<vanity_url>/import/1/<tenant>/<tag>/' This is usually '<CanonicalType>/<FileName>'.
  • <path_to_file>: The path to the file you want to upload on your local file system.
  • <auth_token>: An authorization token generated by 'Authenticator.generateC3AuthToken()'.

curl

You can form an appropriate PUT command with curl. Please follow the detailed instructions on how to do this at C3's official documentation: https://developer.c3.ai/docs/7.12.17/topic/di-curl

POSTMAN

Postman is a GUI tool dedicated to this type of custom POST/PUT command creation. Please see the C3.ai documentation on how to use POSTMAN here: https://developer.c3.ai/docs/7.12.17/topic/tutorial-data-loading-and-processing

Monitoring Data Integration

Source Systems and Source Collections

https://developer.c3.ai/docs/7.12.17/topic/di-source-systems-and-source-collections

SourceFile

https://developer.c3.ai/docs/7.12.17/topic/di-monitoring-and-troubleshooting

Complex Data Sources

Custom External Database

It is also possible to 'add' an external database into C3 as well. Through the SqlSourceSystem, SqlSourceCollection, and External type, you can define a 'persistable' type whose storage exists on the external database. When you execute fetch commands on this type, instead of querying an internal database as the C3 AI Suite normally does, it will call out to this external database to perform the appropriate query. Please see the detailed C3 developer documentation describing how this works here: https://developer.c3.ai/docs/7.12.0/topic/tutorial-external-types

C3 Supported Database technologies

C3 supports numerous connectors to existing databases. Please see C3's documentation on what's available and how to use them here: https://developer.c3.ai/docs/7.12.17/topic/di-connectors

Other Data Formats

Support for other data formats will need to be visited on a case-by-case basis. Contact the DTI at help@c3dti.ai for help determining the right way to ingest your custom data format.