Quickstart
A quick start guide for Featureform with Docker.
This quickstart uses the standalone containered version of Featureform. It can be used to connect to the same providers as the Kubernetes Hosted version, but lacks the scaling capabilities.
This quickstart will walk through creating a few simple features, labels, and a training set from a fraud detection dataset using Postgres and Redis.
Requirements
- Python 3.7+
- Docker
Step 1: Install the Featureform CLI
pip install featureform
Step 2: Start Featureform
To start Featureform, we can run:
featureform deploy docker --quickstart
This will pull and start the Featureform container, as well as a Postgres and Redis containers containing some example data.
Users can optionally also use ClickHouse, should they wish to use an OLAP database for their offline store, by specifying
the --include_clickhouse
flag i.e.
featureform deploy docker --quickstart --include_clickhouse
Step 3: Set the Featureform Host
export FEATUREFORM_HOST=localhost:7878
Step 4: Apply Definitions
Featureform definitions can be stored as both a local file and URL. Multiple files can be applied at the same time.
We’ll set the --insecure
flag since we’re using an unencrypted endpoint on the container.
featureform apply quickstart/definitions.py --insecure
Step 5: Dashboard and Serving
The dashboard is available at localhost
In the dashboard, you should be able to see that 2 Sources, 1 Feature, 1 Label, and 1 Training Set has been created.
You can also check the status of the training set with:
featureform get training-set fraud_training quickstart --insecure
When the status of these resources is READY, you can serve them with:
python quickstart/serving.py
python quickstart/training.py
This will download and run sample serving and training scripts, which will serve a single feature value and a sample of a training data set.
Step 5: Teardown
To teardown the quickstart run:
featureform stop docker
How Does It Work?
Now that we have everything running, we’ll walk through what was done to create the training set and feature.
Apply
If we download the definitions.py file, we can see what Featureform is doing when we run featureform apply
.
First we register the Postgres and Redis containers as providers so Featureform is aware of them.
import featureform as ff
postgres = ff.register_postgres(
name="postgres-quickstart",
host="host.docker.internal", # The docker dns name for postgres
port="5432",
user="postgres",
password="password",
database="postgres",
)
redis = ff.register_redis(
name = "redis-quickstart",
host="host.docker.internal", # The docker dns name for redis
port=6379,
)
We can then register our sources.
The first source we’ll register is our Transactions table that exists in Postgres. This is so Featureform is aware that the Transactions table exists and can be used as a dependency.
We can then create a Transformation source off of our Transactions table. This is done using an SQL query that is executed in Postgres and saved in a table.
transactions = postgres.register_table(
name="transactions",
table="transactions", # This is the table's name in Postgres
)
@postgres.sql_transformation()
def average_user_transaction():
return "SELECT CustomerID as user_id, avg(TransactionAmount) " \
"as avg_transaction_amt from {{transactions.default}} GROUP BY user_id"
We can then register our feature, label, and training set.
The feature is registered off of the table we created with our SQL Transformation.
The label is registered off of our base Transactions table.
A Training Set can be created by joining our feature and label together.
@ff.entity
class User:
avg_transactions = ff.Feature(
average_user_transaction[["user_id", "avg_transaction_amt"]], # We can optional include the `timestamp_column` "timestamp" here
type=ff.Float32,
inference_store=redis,
)
fraudulent = ff.Label(
transactions[["customerid", "isfraud"]], variant="quickstart", type=ff.Bool
)
ff.register_training_set(
"fraud_training",
label=("fraudulent", "quickstart"),
features=["avg_transactions"],
)
The ff.entity
decorator will use the lowercased class name as the entity name. The class attributes avg_transactions
and fraudulent
will be registered as a feature and label, respectively, associated with the user
entity. Indexing into the sources (e.g. average_user_transaction
) with a [["<ENTITY COLUMN>", "<FEATURE/LABEL COLUMN>"]]
, returns the required parameters to the Feature
and Label
registration classes.
When registering more than one variant, we can use the Variants
registration class:
@ff.entity
class User:
avg_transactions = ff.Variants(
{
"quickstart": ff.Feature(
average_user_transaction[["user_id", "avg_transaction_amt"]],
type=ff.Float32,
inference_store=redis,
),
"quickstart_v2": ff.Feature(
average_user_transaction[["user_id", "avg_transaction_amt"]],
type=ff.Float32,
inference_store=redis,
),
}
)
fraudulent = ff.Label(
transactions[["customerid", "isfraud"]], variant="quickstart", type=ff.Bool
)
Serving
We can serve single features from Redis with the Serving Client. The features()
method takes the name of the feature
and an entity that we want the value for.
from featureform import Client
serving = Client(insecure=True)
user_feat = serving.features([("avg_transactions", "quickstart")], {"user": "C1214240"})
print("User Result: ")
print(user_feat)
Training
We can serve a training dataset from Postgres with the Serving Client as well. This example takes the name of the training set and returns 25 rows of the training set.
from featureform import Client
client = Client(insecure=True)
dataset = client.training_set("fraud_training", "quickstart")
for i, batch in enumerate(dataset):
print(batch)
if i > 25:
break