The C3.ai system contains many tools to help scientists with their data analysis. We want researchers
to have the option to get started using C3 right away. This guide details how to connect to a C3 cluster,
fetch data you want to use, and convert it into a form that you can more easily analyze.

This guide serves as a starting point for your C3 journey, but does not expose most of C3's API or potential.

Connecting to a C3 Cluster

There are a couple options for connecting to the C3 Cluster:

Fetching Instances of Types

All data in C3 is represented by a 'Type'. Data for a specific type can be 'fetched' from C3 using the 'fetch' API.
In each language, each Type has a 'fetch' function to which a FetchSpec Type can be passed. This function then retrieves
the data in a FetchResult Type which can be opened and used for data analysis.

Fetching is governed by the FetchSpec Type and you can find full documentation of that type here. However,
as a simple API description, may not be particularly helpful at first. Generally, the FetchSpec type defines a set of
constraints to apply when gathering Type data. it can be 'empty' i.e. without constraints, but generally you'll
want to apply one or more to get reasonable results.

The most useful properties of the FetchSpec are:

filter: Defines an expression to evaluate for each type. When the expression evaluates as true, that type is fetched.
limit: Fetch only 'limit' results. Can be useful to debug a fetch which might grab a lot of results.
include: Define specific properties of the Type to include in the fetch result. If not defined, all properties will be grabbed.
order: An expression which defines the order to return the results in.

Examples of Fetch operations

Consider the DTI housing example located here: https://github.com/c3aidti/HouseCoverageExample

In this example, the Type BlockInfo defines information aggregated about census blocks. We can for example, fetch BlockInfo
types for which the 'prp_bf_lr' property is defined. Then we can order them based on their 'id' properly.

BlockInfo.fetch({
	'limit': -1,
	'filter': 'exists(prp_bf_lr)',
	'order': 'descending(id)',
	'include': 'pct_i_l,pct_t_l,prp_res_lr,pop10_ha_lr,hu10_ha_lr,eroom_ha_lr,med10_age,prp_bf_lr',
})

Here's how you would perform the same fetch in python:

raw_data = c3.BlockInfo.fetch(spec={
    'limit': -1,
    'filter': 'exists(prp_bf_lr)',
    'order': 'descending(id)',
    'include': 'pct_i_l,pct_t_l,prp_res_lr,pop10_ha_lr,hu10_ha_lr,eroom_ha_lr,med10_age,prp_bf_lr'
})

More examples of fetch commands can be found in our examples here:

Here is a list of C3 documentation mentioning fetching:

FetchSpec Documentation: https://developer.c3.ai/docs/7.12.0/type/FetchSpec
Fetching in Python: https://developer.c3.ai/docs/7.12.0/topic/ds-jupyter-notebooks

Converting Fetch results to usable forms in Jupyter Notebook

For most data analysis situations, FetchResults need to be massaged a little bit to be useful.
Here, we show the typical ways FetchResults can be used.

Python

A Code Example in Jupyter Notebook:

## continue from above ##
import pandas as pd
df = pd.DataFrame(raw_data.objs.toJson())
df.head()
df.drop('meta', axis=1, inplace=True)
df.drop('type', axis=1, inplace=True)
df.drop('version', axis=1, inplace=True)
df.drop('id', axis=1, inplace=True)
df.head()

Users can then use the resulting dataframe as they normally would.

ExpressionEngineFunctions

Several spots in the C3 API allow for the use of so-called 'ExpressionEngineFunctions'. These functions take a variety of
arguments and perform a variety of processing tasks. For example, the function 'contains' takes two strings and checks
whether the first argument contains the second argument. The function 'lowerCase' takes an input string, and returns
that string with all uppercase letters replaced with lowercase ones. In addition to those string processing functions,
many math functions exist as well such as 'log' which operates on a variety of input data types.

These functions are very helpful, and can be used in a number of places such as:

'filter' field of FetchSpec
'expression' field of ComplexMetric

Official C3 documentation for ExpressionEngineFunctions: https://developer.c3.ai/docs/7.12.0/type/ExpressionEngineFunction

Executing Metrics on Time series data

TODO:

Page tree

DTI Shallow C3 Primer