Getting Started

The following guide will help you to get started using Connect. Once you have your project ID and API key and you have decided how to model your events, you can start pushing events and executing queries.

Installing the SDK

Getting started is easy - simply install the .NET SDK using NuGet. For example, from the package manager console:

Install-Package ConnectSdk

Initializing the client

Before you can start pushing events or executing queries, you must initialize the Connect client with your project ID and API key.

You can either do this statically (when you don't wish to manage the client instances):

Connect.Initialize(new BasicConfiguration("API_KEY", "PROJECT_ID"))

Or you can create a specific instance of a client (e.g. to push to/query multiple projects):

IConnect connect = new ConnectClient(new BasicConfiguration("API_KEY", "PROJECT_ID"));

Both these classes confirm to the same API, so which you use is purely personal preference. The static method will be used in the remaining examples.

Pushing events

The SDK supports pushing events directly into collections. All methods take advantage of asynchronous programming with async/await.

Single event

To push a single event to a collection, you can specify the collection name and the event to push:

var result = await Connect.Push("mycollection", new {
    product = "Something",
    cost = 2.01m
});

Multiple events

To push multiple events to a collection, you can specify the collection name and an enumerable of events to push:

var events = new[] 
{
    new { product = "Something", cost = 2.01m },
    new { product = "Something Else", cost = 4.02m }
}

var result = await Connect.Push("mycollection", events);

Queuing events

Events can be stored/cached locally and pushed to Connect later. This provides a level of reliability to mitigate the risk of network outages and is best practice.

This can be achieved in the SDK by "adding" events into collections.

Queuing a single event

await Connect.Add("mycollection", new { Name = "Something", Cost = 2.01m });

Queuing multiple events

var events = new[] 
{
    new { product = "Something", cost = 2.01m },
    new { product = "Something Else", cost = 4.02m }
}

await Connect.Add("mycollection", events);

Pushing pending events

Once you have queued events, you must call PushPending() to have those events pushed to Connect. Ideally, this would be done regularly at an appropriate time in your application lifecycle.

var response = await Connect.PushPending();

var status = response.Status; // status of the batch push (EventPushResponseStatus)
var httpStatusCode = response.HttpStatusCode; // HTTP status code
var errorMessage = response.ErrorMessage; // error message if applicable

// loop through the collections
foreach (var collectionResult in response.ResponsesByCollection) {
    var collectionName = collectionResult.Key;

    // loop through the results of individual events
    foreach (var result in collectionResult.Value) {
        var originalEvent = result.Event; // original event pushed
        var status = result.Status; // status of the individual event
        var errorMessage = result.ErrorMessage; // error message if applicable
        var fieldErrors = result.FieldErrors; // Dictionary of field to error message
    }
}

Bulk importing events

Currently, this SDK does not support bulk importing events.

However, you can use the HTTP API to run bulk imports if you need.

Restrictions on pushing

There are a number of restrictions on the properties you can use in your events and the limitations on querying which influences how you should structure your events.

Refer to restrictions in the modeling your events section.

Reliability of events

You can ensure delivery of events reliably by queuing the events. You should then handle the response from PushPending() to verify that all the events were successfully pushed.

Events also allow a custom ID to be sent in the event document which will prevent duplicates (i.e. guarantees idempotence even if the event is delivered multiple times). For example:

var result = await Connect.Push("mycollection", new {
    id = "12345",
    product = "Something",
    cost = 2.01m
});

Timestamps

All events have a single timestamp property which records when the event being pushed occurred. Events cannot have more than one date/time property. If you feel you need more than one date/time property, you probably need to reconsider how you're modeling your events.

Querying

You can only run time interval queries or timeframe filters on the timestamp property. No other date/time property in an event is supported for querying.

By default, if no timestamp property is sent with the event, the SDK will use the current date and time as the timestamp of the event.

The timestamp, however, can be overridden to, for example, accommodate historical events or maintain accuracy of event times when events are queued. For example:

var result = await Connect.Push("mycollection", new {
    timestamp = DateTime.Now,
    product = "Something",
    cost = 2.01m
});

Timezones

Timestamps are always recorded in UTC. If you supply a timestamp in a timezone other than UTC, it will be converted to UTC. When you query your events, you can specify a timezone so things like time intervals will be returned in local time.

Querying events

Querying events in Connect is easy. Using the SDK, you can construct and execute queries using async/await.

Once you have initialized the client with a valid read key, you can start querying your collections immediately.

For example, to get the sum of the price property in a collection called purchases, you would build the following query:

var queryResponse = await Connect.Query("purchases")
    .Select(new {
        totalPrice = Aggregations.Sum("price"),
        averagePrice = Aggregations.Avg("price"),
        minPrice = Aggregations.Min("price"),
        maxPrice = Aggregations.Max("price"),
        purchaseCount = Aggregations.Count()
    })
    .Where(new {
        product = Filters.Eq("12 red roses")
    })
    .Daily()
    .ThisMonth()
    .GroupBy("country")
    .Timezone("Australia/Brisbane")
    .Execute();

Metadata

The query results return metadata on the executed query. This includes any groups used, the time interval, if specified, and the timezone, if specified, for the query.

For example, the following query:

var queryResponse = await Connect.Query("purchases")
    .Select(new {
        totalPrice = Aggregations.Sum("price")
    })
    .GroupBy("product")
    .Daily()
    .ThisMonth()
    .Timezone("Australia/Brisbane")
    .Execute();

The metadata would be:

Console.WriteLine(string.Format("Groups: {0}", string.Join(", ", queryResponse.Metadata.Groups.ToArray())));
Console.WriteLine("Interval: " + queryResponse.Metadata.Interval.ToString());
Console.WriteLine("Timezone: " + queryResponse.Metadata.Timezone);

// output is:
//
// Groups: product
// Interval: Daily
// Timezone: Australia/Brisbane

The metadata can assist you if you wish to visualize the query results as it provides useful information on how to display that data.

Aggregations

You can perform various aggregations over events you have pushed. Simply specify in the query's select which properties you wish to aggregate and which aggregation you wish to use. You also must specify an "alias" for the result set that is returned.

For example, to query the purchases collection and perform aggregations on the price event property::

var queryResponse = await Connect.Query("purchases")
    .Select(new {
        itemsSold = Aggregations.Count(),
        totalPrice = Aggregations.Sum("price"),
        averagePrice = Aggregations.Avg("price"),
        minPrice = Aggregations.Min("price"),
        maxPrice = Aggregations.Max("price")
    })
    .Execute();

This would return a result like:

// no important metadata because no group by, interval or timezone
var metadata = queryResponse.Metadata;

// no group by, so only a single result
var singleResult = queryResponse.Results.First();

// get the values from the result
var itemsSold = (double)singleResult["itemsSold"];
var totalPrice = (double)singleResult["totalPrice"];
var averagePrice = (double)singleResult["averagePrice"];
var minPrice = (double)singleResult["minPrice"];
var maxPrice = (double)singleResult["maxPrice"];

You can also use the AggregationOperation enum to specify the aggregation operation:

var queryResponse = await Connect.Query("purchases")
    .Select(new {
        totalPrice = Aggregations.Use(AggregationOperation.Sum, "price"),
        averagePrice = Aggregations.Use(AggregationOperation.Avg, "price"),
        minPrice = Aggregations.Use(AggregationOperation.Min, "price"),
        maxPrice = Aggregations.Use(AggregationOperation.Max, "price"),
        purchaseCount = Aggregations.Use(AggregationOperation.Count)
    })
    .Execute();

Or a string:

var queryResponse = await Connect.Query("purchases")
    .Select(new {
        totalPrice = Aggregations.Use("Sum", "price"),
        averagePrice = Aggregations.Use("Avg", "price"),
        minPrice = Aggregations.Use("Min", "price"),
        maxPrice = Aggregations.Use("Max", "price"),
        purchaseCount = Aggregations.Use("Count")
    })
    .Execute();

If you are using C# 6.0 or above you may find it useful to statically include the Aggregations class:

using static ConnectSdk.Querying.Aggregations

This enables you to use methods like Count() without prefixing with Aggregations

The following aggregations are supported:

Aggregations.Count (AggregationOperation.Count or "Count")
Aggregations.Sum (AggregationOperation.Sum or "Sum")
Aggregations.Avg (AggregationOperation.Avg or "Avg")
Aggregations.Min (AggregationOperation.Min or "Min")
Aggregations.Max (AggregationOperation.Max or "Max")

Limitations

Aggregations only work on numeric properties. If you try to aggregate a string property, you will receive a null result. If you try to aggregate a property with multiple types (e.g. some strings, some numbers), only the numeric values will be added - the rest are ignored.

Filters

You can filter the events you wish to include in your queries by specifying one or more filters. For example:

var queryResponse = await Connect.Query("purchases")
    .Select(new {
        totalPrice = Aggregations.Sum("price")
    })
    .Where("product", "12 red roses")
    .Execute();

The above will filter the query results for only those events that have a product property equalling "12 red roses". The above is also shorthand for the eq operator. To illustrate this, the following is identical to the above:

var queryResponse = await Connect.Query("purchases")
    .Select(new {
        totalPrice = Aggregations.Sum("price")
    })
    .Where(new {
        product = Filters.Eq("12 red roses")
    })
    .Execute();

The following match operators are currently supported:

Filters.Eq (equal to)
Filters.Neq (not equal to)
Filters.Gt (greater than)
Filters.Gte (greater than or equal to)
Filters.Lt (less than)
Filters.Lte (less than or equal to)
Filters.Exists (whether or not a property exists)
Filters.StartsWith (starts with - string values only)
Filters.EndsWith (ends with - string values only)
Filters.Contains (contains - string values only)
Filters.In (property is one of a list of values)

Exists filter

The exists filter will filter the query results for only events that either have or don't have the specified property and where the specified property is or isn't null respectively. You supply a boolean value with the exists operator to specify whether to include or exclude the events.

Missing properties vs null values

We treat missing properties and properties with a null value the same for the purpose of the exists filter. While we plan to change this behavior in the future, you should consider setting a default value as opposed to a null value on properties if you wish to make a distinction.

For example, the following will filter for events that have a property called gender:

var queryResponse = await Connect.Query("purchases")
    .Select(new {
        TotalPrice = Aggregations.Sum("Price")
    })
    .Where(new {
        gender = Filters.Exists(true)
    })
    .Execute();

Whereas the following will filter for events that do not have a property called gender:

var queryResponse = await Connect.Query("purchases")
    .Select(new {
        totalPrice = Aggregations.Sum("price")
    })
    .Where(new {
        gender = Filters.Exists(false)
    })
    .Execute();

In filter

The in filter allows you to specify a list of values for which a property should match.

For example, the following will filter for events that have a category of either "Bikes", "Books" or "Magazines":

var queryResponse = await Connect.Query("purchases")
    .Select(new {
        totalPrice = Aggregations.Sum("price")
    })
    .Where(new {
        category = Filters.In(new[] {"Bikes", "Books", "Magazines"})
    })
    .Execute();

You can also use the shorthand of this:

var queryResponse = await Connect.Query("purchases")
    .Select(new {
        totalPrice = Aggregations.Sum("price")
    })
    .Where("category", new[] {"Bikes", "Books", "Magazines"})
    .Execute();

Note: All values in the list must be of the same type (i.e. string, numeric or boolean). Mixed types are currently not supported.

Combining filter expressions

You can also combine filter expressions to filter multiple values on the same property.

For example, the following will filter for events with a price property greater than 5 but less than 10:

var queryResponse = await Connect.Query("purchases")
    .Select(new {
        totalPrice = Aggregations.Sum("price")
    })
    .Where(new Dictionary<string, Filter[]>{
        {"price", new[]{ Filters.Gt(5), Filters.Lt(10) }}
    })
    .Execute();

Chaining filters

Filters can be chained by calling the Where method multiple times. The query below will query for purchase that are in the "Shirts" category and have a delivery type of "Standard" or "Express":

var queryResponse = await Connect.Query("purchases")
    .Select(new {
        totalPrice = Aggregations.Sum("price")
    })
    .Where(new {
        category = Filters.Eq("Shirts")
    })
    .Where(new {    
        delivery = Filters.In(new [] {"Express", "Standard"})
    })
    .Execute();

Filter dictionaries

The SDK also supports specifying filters as dictionaries. This especially useful if you wish to specify a nested property path. For example, to filter on product.category:

var queryResponse = await Connect.Query("purchases")
    .Select(new {
        totalPrice = Aggregations.Sum("price")
    })
    .Where(new Dictionary<string, Filter[]>{
        {"product.category", Filters.Eq("Shirts")}
    })
    .Execute();

"Or" filters

Currently, "or" filters are not supported.

Timeframes

You can restrict query results by specifying the timeframe for the query which will filter for events only within that specific timeframe.

If no timeframe is specified, events will not be filtered by time; the query will match events from all time.

For example, the following query filters for events only for this month:

var queryResponse = await Connect.Query("purchases")
    .Select(new {
        totalPrice = Aggregations.Sum("price")
    })
    .ThisMonth()
    .Execute();

There are two types of timeframes:

Relative timeframes - a timeframe relative to the current date and time.
Absolute timeframes - a timeframe between two specific dates and times.

Relative timeframes

Relative timeframes can be specified as either a string or a complex type containing exact numbers of "periods" to filter.

Timezones

By default, all relative timeframes are UTC by default. See the timezone section to specify your own timezone.

The following are supported timeframes:

ThisMinute()
LastMinute()
ThisHour()
LastHour()
Today()
Yesterday()
ThisWeek()
LastWeek()
ThisMonth()
LastMonth()
ThisQuarter()
LastQuarter()
ThisYear()
LastYear()

You can also specify a relative timeframe with an enum:

var queryResponse = await Connect.Query("purchases")
    .Select(new {
        totalPrice = Aggregations.Sum("price")
    })
    .WithRelativeTimeframe(RelativeTimeWindow.ThisMonth)
    .Execute();

Or a string:

var queryResponse = await Connect.Query("purchases")
    .Select(new {
        totalPrice = Aggregations.Sum("price")
    })
    .WithRelativeTimeframe("this_month")
    .Execute();

You can also specify exactly how many current/previous periods you wish to include in the query results.

For example, if you want to filter by the today and the last 2 days:

var queryResponse = await Connect.Query("purchases")
    .Select(new {
        totalPrice = Aggregations.Sum("price")
    })
    .Current(3, TimeType.Days)
    .Execute();

Or, to filter by the last 2 months, excluding the current month:

var queryResponse = await Connect.Query("purchases")
    .Select(new {
        totalPrice = Aggregations.Sum("price")
    })
    .Previous(2, TimeType.Months)
    .Execute();

If you are using C# 6.0 or above you may find it useful to statically include the RelativeWindow and TimeType enums.

using static ConnectSdk.Querying.RelativeWindow
using static ConnectSdk.Querying.TimeType

This enables you to use the enums without prefixes.

The following periods are supported for complex, relative timeframes:

TimeType.Minutes
TimeType.Hours
TimeType.Days
TimeType.Weeks
TimeType.Months
TimeType.Quarters
TimeType.Years

Weeks

Our weeks start on a Sunday and finish on a Saturday. In the future, we plan to support specifying on which day of the week you'd like to start your weeks.

Absolute timeframes

You can specify an absolute timeframe to filter events that occurred between specific dates. For example:

var queryResponse = await Connect.Query("purchases")
    .Select(new {
        totalPrice = Aggregations.Sum("price")
    })
    .Between(new DateTime(2015, 01, 01), new DateTime(2015, 01, 06))
    .Execute()

The Connect API only accepts UTC dates for an absolute timeframe, therefore if you specify a DateTime with a kind of DateTimeKind.Local, it will be converted to UTC before querying.

You can also use StartingAt or EndingAt to filter events that occurred after or before a specific date, respectively:

var queryResponse = await Connect.Query("purchases")
    .Select(new {
        totalPrice = Aggregations.Sum("price")
    })
    .StartingAt(new DateTime(2015, 01, 01))
    .Execute()

You can only specify a timezone and an absolute timeframe when you have also specified a time interval. If a timezone is specified on an absolute timeframe query that does not have a time interval an error will occur.

Group by

You can group the query results by one or more properties from your events.

Missing properties vs null values

We treat missing properties and properties with a null value the same for the purpose of grouping. This means that all events with a null value for a property or missing that property altogether will be grouped into the "null" value for that query. While we plan to change this in the future, you should consider setting a default value as opposed to a null value on properties if you wish to make a distinction.

For example, to query total sales by country:

var queryResponse = await Connect.Query("purchases")
    .Select(new {
        totalPrice = Aggregations.Sum("price")
    })
    .GroupBy("country")
    .Execute();

This would return a result like:

var metadata = queryResponse.Metadata;
// metadata.Groups is new string[] {"country"}

foreach (var result in queryResponse.Results) {
    Console.WriteLine(string.Format("{0}: {1}", result["country"], result["totalPrice"]));
}

// output is:
//
// Australia: 1000000
// Italy: 2500000
// United States: 10000000

Grouping by multiple properties

You can also group by multiple properties in your events by providing multiple property names.

For example, to query total sales by country and product category:

var queryResponse = await Connect.Query("purchases")
    .Select(new {
        totalPrice = Aggregations.Sum("price")
    })
    .GroupBy("country", "product.category")
    .Execute();

This would return a result like:

var metadata = queryResponse.Metadata;
// metadata.Groups is new string[] {"country", "product.category"}

foreach (var result in queryResponse.Results) {
    Console.WriteLine(string.Format("{0} - {1}: {2}", result["country"],
        result["product.category"], result["totalPrice"]));
}

// output is:
//
// Australia - Bikes: 500000
// Austrlaia - Cars: 500000
// Italy - Scooters: 2500000
// United States - Mobile Phones: 8000000
// United States - Laptops: 2000000

Time intervals

Time intervals allow you to group results by a time period, so that you could analyze your events over time.

For example, to query daily total sales this month:

var queryResponse = await Connect.Query("purchases")
    .Select(new {
        totalPrice = Aggregations.Sum("price")
    })
    .ThisMonth()
    .Daily()
    .Execute();

This would return a result like:

var metadata = queryResponse.Metadata;
// metadata.Interval is Interval.Daily

foreach (var intervalResult in queryResponse.Results) {
    // intervalResult is QueryIntervalResult<IDictionary<string, object>>

    // single result because no group by
    var result = intervalResult.Results.First();

    Console.WriteLine(string.Format("{0:d} to {1:d}: {2}", intervalResult.Start,
        intervalResult.End, result["totalPrice"]));
}

// output is (en-US dates):
//
// 02/01/2015 to 02/02/2015: 500000
// 02/02/2015 to 02/03/2015: 150000
// 02/03/2015 to 02/04/2015: 25000

The following time intervals are supported:

minutely
hourly
daily
weekly
monthly
quarterly
yearly

Time interval with group by

You can also combine a time interval with a group by in your query.

For example, to query daily total sales this month by country:

var queryResponse = await Connect.Query("purchases")
    .Select(new {
        totalPrice = Aggregations.Sum("price")
    })
    .ThisMonth()
    .Daily()
    .GroupBy("country")
    .Execute();

This would return a result like:

var metadata = queryResponse.Metadata;
// metadata.Interval is Interval.Daily
// metadata.Groups is new string[] { "country" }

foreach (var intervalResult in queryResponse.Results) {
    // intervalResult is QueryIntervalResult<IDictionary<string, object>>

    Console.WriteLine(string.Format("Totals for {0:d} to {1:d}:",
        intervalResult.Start, intervalResult.End));

    foreach (var result in intervalResult.Results) {
        Console.WriteLine(string.Format("{0}: {1}", result["country"], result["totalPrice"]));
    }

    Console.WriteLine();
}

// output is (en-US dates):
//
// Totals for 02/01/2015 to 02/02/2015:
// Australia: 100000
// Italy: 100000
// United States: 300000
//
// Totals for 02/02/2015 to 02/03/2015:
// Australia: 25000
// Italy: 25000
// United States: 100000
//
// Totals for 02/03/2015 to 02/04/2015:
// Australia: 5000
// Italy: 5000
// United States: 15000
//

Timezones

By default, Connect uses UTC as the timezone for queries with relative timeframes or time intervals. You can override this to be the timezone of your choice by specifying the timezone in your query.

You can only specify a timezone when you have specified a time interval and/or a relative timeframe. If you try to specify a timezone without one of these set, an error will be returned.

You can specify a numeric (decimal) value for an hours offset of UTC, for example:

var queryResponse = await Connect.Query("purchases")
    .Select(new {
        totalPrice = Aggregations.Sum("price")
    })
    .Timezone(10)
    .Execute();

You can also specify a string which contains an IANA time zone identifier, for example:

var queryResponse = await Connect.Query("purchases")
    .Select(new {
        totalPrice = Aggregations.Sum("price")
    })
    .Timezone("Australia/Brisbane")
    .Execute();

Error handling

When you Execute() a query, you will receive a QueryResponse in response to the query. You should check the Status of this response to ensure it is successful and, if not, handle the error gracefully.

QueryResponse is an enum with the following values:

Successful - the query was successful
Unauthorized - the project ID or API key used were not correct
QueryFormatError - the query was not correctly formatted
NetworkError - a network error occurred while running the query
GeneralError - a general/unknown error occurred while running the query

If the result is not successful, you will be able to find a general error message in ErrorMessage as well as field specific errors (if applicable) in FieldErrors.

Exceptions are only thrown when a valid response is not received from the Connect API. These would capture issues like client network connectivity.

Exporting events

Currently, this SDK does not support exporting events.

However, you can use HTTP API to perform exports as required.

Deleting collections

Currently, this SDK does not support deleting collections.

However, you can use the one of the following methods to delete collections if required:

Projects and keys

Connect allows you to manage multiple projects under a single account so that you can easily segregate your collections into logical projects.

You could use this to separate analytics for entire projects, or to implement separation between different environments (e.g. My Project (Prod) and My Project (Dev)).

To start pushing and querying your event data, you will need both a project ID and an API key. This information is available to you via the admin console inside each project under the "Keys" tab:

Screenshot of project keys in Connect admin console

By default, you can choose from four different types of keys, each with their own specific use:

Push/Query Key - you can use this key to both push events and execute queries.
You should only use this key in situations where it is not possible to isolate merely pushing or querying.
Push Key - you can only use this key to push events.
You should use this key in your apps where you are tracking event data, but do not require querying.
Query Key - you can only use this key to execute queries.
You should use this key in your reporting interfaces where you do not wish to track events.
Master Project Key - you can use this key to execute all types of operations on a project, including pushing, querying and deleting collections.
Keep this key safe - it is intended for very limited use and definitely should not be included in your main apps.

You must use your project ID and desired key to begin using Connect:

Connect.Initialize(new BasicConfiguration("YOUR_API_KEY", "YOUR_PROJECT_ID"));

Security

Security is a vital component to the Connect service and we take it very seriously. It is important to consider how to ensure your data remains secure.

API Keys

API keys are the core security mechanisms by which you can push and query your data. It is important to keep these keys safe by controlling where these keys exist and who has access to them.

Each key can either push, query or both. The most important key is the Project Master Key which can perform all of these actions, as well as administrative functions such as deleting data. Read more about the keys here.

Keeping API Keys Secure

You should carefully consider when and which API keys to expose to users.

Crucially, you should never expose your Project Master Key to users or embed it in client applications. If this key does get compromised, you can reset it.

If you embed API keys in client applications, you should consider these keys as fully accessible to anyone having access to that client application. This includes both mobile and web applications.

Pushing events securely

While you can use a Push Key to prevent clients from querying events, you cannot restrict the collections or events clients can push to the API. Unfortunately, this is the nature of tracking events directly client-side and opens the door to malicious users potentially sending bad data.

In many circumstances, this is not an issue as users can already generate bad data simply by using your application in an incorrect way, generating events with bad or invalid data. In circumstances where you absolutely cannot withstand bad event data, you should consider pushing the events server-side from a service under your control.

Finally, if a Push Key is compromised or being used maliciously, you can always reset it by resetting the master key.

Querying events securely

To query events, you must use an API key that has query permissions. By default, a Query Key has full access to all events in all collections in your project. If this key is exposed, a client could execute any type of query on your collections.

You have a number of options on querying events securely:

For internal querying or dashboard, you may consider it acceptable to expose the normal Query Key in client applications. Keep in mind that this key can execute any query on any collection in the project.
Generate a filtered key, which applies a specific set of filters to all queries executed by clients with the key.
Only allow clients to execute queries via a service you control, which in turn executes queries via the Connect API server-side.

Finally, if a Query Key is compromised or being used maliciously, you can always reset it by resetting the master key.

Resetting the master key

Resetting the Project Master Key will invalidate the previous key and generate a new, random key. This action will also reset all other keys for the project (including the push, query and any filter keys generated).

Doing this is irreversible and would prevent all applications with existing keys from pushing to or querying the project.

You can only reset the master key in the projects section of the admin console.

Filtered keys

Filtered keys allows you to create an API key that can either push or query, and in the case of querying, apply one or more filters to all queries executed with the key.

This allows you to have finer control over security and what data clients can access, especially in multi-tenant environments.

Filters are only applied to queries

Any filters specified in your filtered key only apply to querying. We currently do not support applying filters to restrict the pushing of events.

Filtered keys can only push or query (as you specify), never administrative functions or deleting data.

Generating a filtered key

Filtered keys are generated and encrypted with the Project Master Key. You do not have to register the filtered key with the Connect service.

To generate a filtered key, you must supply the master key, key settings and filters. For example:

var canPush = true;
var canQuery = true;
var keySettings = new KeySettings(canPush, canQuery);

string filteredKey = Connect.FilteredKeyQuery()
    .Where("customer.firstName", "Tom")
    .GenerateFilteredKey("YOUR_MASTER_KEY", keySettings);

Property	Type	Description
`filters`	`object`	The filters to apply all queries executed when using the key. This uses the same specification for defining filters when querying normally.
`canQuery`	`boolean`	Whether or not the key can be used to execute queries. If `false`, the `filters` property is ignored (as it does not applying to pushing).
`canPush`	`boolean`	Whether or not the key can be used to push events.

You would use the resulting key when creating a client or to provide to client applications (e.g. in a browser using the JavaScript SDK).

Modeling your events

When using Connect to analyze and visualize your data, it is important to understand how best to model your events. The way you structure your events will directly affect your ability to answer questions with your data. It is therefore important to consider up-front the kind of questions you anticipate answering.

What is an event?

An event is an action that occurs at a specific point in time. To answer "why did this event occur?", our event needs to contain rich details about what the "world" looked like at that point in time.

Put simply, events = action + time + state.

For example, imagine you are writing an exercise activity tracker app. We want to give users of your app the ability to analyze their performance over time. This is an event produced by our hypothetical activity tracker app:

var myEvent = new {
    type = "cycling",
    timestamp = DateTime.UtcNow,
    duration = 67,
    distance = 21255,
    caloriesBurned = 455,
    maxHeartRate = 182,
    user = new {
        id = 698396,
        firstName = "Bruce",
        lastName = "Jones",
        age = 35
    }
};

Action

What happened? In the above example, the action is an activity was completed.

In most circumstances, we group all events of the same action into a single collection. In this case, we could call our collection activityCompleted, or alternatively, just activity.

Time

When did it happen? In the above example, we specified the start time of the activity as the value of the timestamp property. The top-level timestamp property is a special property in Connect. This is because time is an essential property of event data - it's not optional.

When an event is pushed to Connect, the current time is assigned to the timestamp property if no value was provided by you.

State

What do we know about this action? What do we know about the entities associated with this action? What do we know about the "world" at this moment in time? Every property in our event, besides the timestamp and the name of the collection, serves to answer those questions. This is the most important aspect of our event - it's where all the answers live.

The richer the data you provide in your event, the more questions you can answer for your users, therefore it's important to enrich your events with as much information as possible. In stark contrast to the relational model where you would store this related information in separate tables and join at query time, in the event model this data is denormalized into each event, so as to know the state of the "world" at the point in time of the event.

Collections

It is important when modeling your events to consider how you intend to group those events into collections. This is a careful balance between events being broad enough to answer queries for your users, while specific enough to be manageable.

In our activity example, the activity contains different properties based on what the type of activity. Our cycling activity contains properties associated with the bike that was used, while a kayaking activity may contain properties associated with a kayak that is used.

Because a kayaking event may have different properties to a running event, it might seem logical to put each of them in distinct collections. However, if we had distinct cycling, running and kayaking collections, we would lose the opportunity to query details that are common to all activities.

As a general rule, consider the common action among your events and decide if the specific variants of that action warrant grouping those events together.

Structuring your events

Events have the following core properties:

Denormalized
Immutable
Rich/nested
Schemaless

It is also important to consider how to group events into collections to enable future queries to be answered.

Events are denormalized

Consider our example event again, notice the age property of the user:

var myEvent = new {
    type = "cycling",
    ...
    user = new {
        id = 698396,
        firstName = "Bruce",
        lastName = "Jones",
        age = 35
    }
};

The user's age is going to be duplicated in every activity he/she completes throughout the year. This may seem inefficient; however, remember that Connect is about analyzing. This denormalization is a real win for analysis; the key is that event data stores state over time, rather than merely current state. This helps us answer questions about why something happened, because we know what the "world" looked like at that point of time.

For example imagine we wanted to chart the average distance cycled per ride, grouped by the age of the rider at the time of the ride. We could simply execute the following query:

var queryResponse = await Connect.Query("activity")
  .Select(new {
    AverageDistance = Aggregations.Avg("distance")
  })
  .GroupBy("user.age")
  .Execute();

It's this persistence of state over time that makes event data perfect for analysis.

Events are immutable

By their very nature, events cannot change, as they always record state at the point in time of the event. This is also the reason to record as much rich information about the event and "state of the world" as possible.

For example, in our example event above, while Bruce Jones may now be many years older, at the time he completed his bike ride, he was 35 years of age. By ensuring this event remains immutable, we can correctly analyze bike riding over time by 35-year-olds.

Consider events as recording history - as much as we'd occasionally like to, we can't change history!

Events are rich and nested

Events are rich in that they specify very detailed state. They specify details about the event itself, the entities involved and the state of the "world" at that point in time.

Consider our example activity event - the top level type property describes something about the activity itself (a run, a bike ride, a kayak etc.). The user property specifies rich information about the actor who performed the event. In this case it's the person who completed the activity, complete with their name and age.

In reality, though, we may decide to include a few other nested entities in our event, for example:

var myEvent = new {
    type = "cycling",
    ...
    user = new {
        id = 698396,
        firstName = "Bruce",
        ...
    },
    bike = new {
        id = 231806,
        brand = "Specialized",
        model = "S-Works Venge"
    },
    weather = new {
        condition = "Raining",
        temperature = 21,
        humidity = 99,
        wind = 17
    }
};

Note our event now includes details about the bike used and the weather conditions at the time of the activity. By adding this extra bike state information to our event, we have opened up extra possibilities for interrogating our data. For example, we can now query the average distance cycled by each model of bike that was built by "Specialized":

var queryResponse = await Connect.Query("activity")
  .Select(new {
    AverageDistance = Aggregations.Avg("distance")
  })
  .Where("bike.brand", "Specialized")
  .GroupBy("bike.model")
  .Execute();

The weather also provides us with exciting insights - what did the world look like at this point in time? What was the weather like? Storing this data allows us to answer yet more questions. We can test our hypothesis that "older people are less scared of riding in the rain" by simply charting the following query:

var queryResponse = await Connect.Query("activity")
  .Select(new {
    AverageDistance = Aggregations.Avg("distance")
  })
  .GroupBy("user.age", "weather.condition")
  .Execute();

As you can see, the richer and more denormalized the event, the more interesting answers can be derived when later querying.

Events are schemaless

Events in Connect should be considered semi-structured - that is, they have an inherent structure, but it is not defined. This means you can, and should, push as much detailed information about an event and the state of the "world" as possible. Moreover, this allows you to improve your schema over time and add extra information about new events as that information becomes available.

Restrictions

While you can post almost any event structure to Connect, there are a few, by-design restrictions.

Property names

You cannot have any property in the root document beginning with "tp_". This is because we prefix our own internal properties with this. Internally, we merge our properties into your events for performance at query time.
The property "_id" is reserved and cannot be pushed.
The properties "id" and "timestamp" have special purposes. These allow consumers to specify a unique ID per event and override the event's timestamp respectively. You cannot use the "id" property in queries. Refer to "reliability of events" and "timestamps" for information.
The length of property names can't exceed 255 characters. If you need property names longer than this, you probably need to reconsider the structure of your event!
Properties cannot include a dot in their names. This is because dots are used in querying to access nested properties. The following is an example of an invalid event property due to a dot in the name:

var result = await Connect.Push("mycollection", new Dictionary<string, object> {
    { "invalid.property", "value" }
});

Arrays

While you can create events with arrays, it is currently not possible to take advantage of these arrays at query time. Therefore, you should avoid using arrays in your events unless you plan to export the raw events.

Distinct count

Distinct count is currently not supported for querying, therefore you should consider how to structure your event if your application relies on this.