Getting Started
The following guide will help you to get started using Connect. Once you have your project ID and API key and you have decided how to model your events, you can start pushing events and executing queries.
Installing the SDK
The quickest and easiest way to install the SDK is via CocoaPods: 
To do this, simply add the following line to your podfile and run pod update:
pod 'ConnectClient'
Initializing the client
Before you can start pushing events, you must initialize the Connect client with your project ID and API key.
You can either initialize a shared client (when you don't want to manage client instances):
#import "Connect.h"
[TPConnectClient sharedClientWithProjectId:@"YOUR_PROJECT_ID"
apiKey:@"YOUR_API_KEY"];
TPConnectClient.sharedClientWithProjectId("YOUR_PROJECT_ID", apiKey: "YOUR_API_KEY")
Or you can create a specific instance of a client (e.g. to push to/query multiple projects):
#import "Connect.h"
TPConnectClient *client = [TPConnectClient clientWithProjectId:@"YOUR_PROJECT_ID"
apiKey:@"YOUR_API_KEY"];
let client = TPConnectClient.clientWithProjectId("YOUR_PROJECT_ID", apiKey: "YOUR_API_KEY")
Pushing events
The SDK supports pushing events directly into collections.
Single event
To push a single event to a collection, you can specify the collection name and the event to push:
NSDictionary *event = @{
@"product": @"Something",
@"cost": @(2.01)
};
[[TPConnectClient sharedClient] pushEvent:event
toCollection:@"mycollection"
completionHandler:^(BOOL success, NSError *error) {
}];
let event = [
"product": "Something",
"cost": 2.01
]
TPConnectClient.sharedClient()?.pushEvent(event, toCollection: "mycollection") {
(success, error) in
}
Batches of events
To push multiple events to one or more collections, you can create a dictionary of collection names to arrays of events:
NSArray *purchases = @[
@{
@"name": @"Phone",
@"price": @465.55
},
@{
@"name": @"Case",
@"price": @5.55
}
];
NSDictionary *eftpos = @{
@"type": @"visa",
@"total": @571.10
};
NSDictionary *batch = @{
@"purchases": purchases,
@"eftpos": @[eftpos]
};
[[TPConnectClient sharedClient] pushEventBatch:batch
completionHandler:^(NSDictionary * results, NSError * error) {
}];
let batch = [
"purchases": [
[
"name": "phone",
"price": 465.55
],
[
"name": "case",
"price": 5.55
]
],
"eftpos": [[
"type": "visa",
"total": 571.10
]]
]
TPConnectClient.sharedClient()?.pushEventBatch(batch) {
(results, error) in
}
Queuing events
Events can be stored/cached locally and pushed to Connect later. This provides a level of reliability to mitigate the risk of network outages or device crashes.
This can be achieved in the SDK by "adding" events into collections:
NSError *error;
[[TPConnectClient sharedClient] addEvent:event
toCollection:@"mycollection"
withError:&error];
var error: NSError?
TPConnectClient.sharedClient()?.addEvent(event, toCollection: "mycollection", withError: &error)
Periodically or on a specific trigger, you must push the pending events in a batch to Connect:
[[TPConnectClient sharedClient] pushAllPendingEventsWithCompletionHandler:^(NSDictionary * results, NSError * error) {
}];
TPConnectClient.sharedClient()?.pushAllPendingEventsWithCompletionHandler {
(results, error) in
}
The results is an NSDictionary keyed by collection name. The value of the NSDictionary is an NSArray of
TPEventAPIResponse
indicating the response to an individual event.
Bulk importing events
Currently, this SDK does not support bulk importing events.
However, you can use the HTTP API to run bulk imports if you need.
Restrictions on pushing
There are a number of restrictions on the properties you can use in your events and the limitations on querying which influences how you should structure your events.
Refer to restrictions in the modeling your events section.
Reliability of events
You can ensure delivery of events reliably by queuing the events.
You should then handle the response from pushAllPendingEvents to verify that all the events were successfully pushed.
Events also allow a custom ID to be sent in the event document which will prevent duplicates (i.e. guarantees idempotence even if the event is delivered multiple times). For example:
NSDictionary *event = @{
@"id": "12345678",
@"product": @"Something",
@"cost": @(2.01)
};
let event = [
"id": "12345678",
"product": "Something",
"cost": 2.01
]
Timestamps
All events have a single timestamp property which records when the event being pushed occurred. Events cannot
have more than one date/time property. If you feel you need more than one date/time property, you probably need
to reconsider how you're modeling your events.
Querying
You can only run time interval queries or timeframe filters on the
timestampproperty. No other date/time property in an event is supported for querying.
By default, if no timestamp property is sent with the event, the SDK will use the current date and time as
the timestamp of the event.
The timestamp, however, can be overridden to, for example, accommodate historical events or maintain accuracy of event times when events are queued. For example:
NSDictionary *event = @{
@`timestamp`: [NSDate date],
@"product": @"Something",
@"cost": @(2.01)
};
let event = [
`timestamp`: NSDate(),
"product": "Something",
"cost": 2.01
]
Timezones
Timestamps are always recorded in UTC. If you supply a timestamp in a timezone other than UTC, it will be converted to UTC. When you query your events, you can specify a timezone so things like time intervals will be returned in local time.
Querying events
Currently, this SDK does not support querying events. However, you have the following options to query:
- HTTP API - send queries and receives results directly via JSON
- JavaScript SDK - query events and build visualizations client-side in the browser
- .NET SDK - query events using the fluent .NET query syntax
We're really looking forward to supporting querying in all SDKs soon!
Exporting events
Currently, this SDK does not support exporting events.
However, you can use HTTP API to perform exports as required.
Deleting collections
Currently, this SDK does not support deleting collections.
However, you can use the one of the following methods to delete collections if required:
Projects and keys
Connect allows you to manage multiple projects under a single account so that you can easily segregate your collections into logical projects.
You could use this to separate analytics for entire projects, or to implement separation between different environments (e.g. My Project (Prod) and My Project (Dev)).
To start pushing and querying your event data, you will need both a project ID and an API key. This information is available to you via the admin console inside each project under the "Keys" tab:
By default, you can choose from four different types of keys, each with their own specific use:
Push/Query Key- you can use this key to both push events and execute queries.
You should only use this key in situations where it is not possible to isolate merely pushing or querying.Push Key- you can only use this key to push events.
You should use this key in your apps where you are tracking event data, but do not require querying.Query Key- you can only use this key to execute queries.
You should use this key in your reporting interfaces where you do not wish to track events.Master Project Key- you can use this key to execute all types of operations on a project, including pushing, querying and deleting collections.
Keep this key safe - it is intended for very limited use and definitely should not be included in your main apps.
You must use your project ID and desired key to begin using Connect:
#import "Connect.h"
[TPConnectClient sharedClientWithProjectId:@"YOUR_PROJECT_ID"
apiKey:@"YOUR_API_KEY"];
TPConnectClient.sharedClientWithProjectId("YOUR_PROJECT_ID", apiKey: "YOUR_API_KEY")
Security
Security is a vital component to the Connect service and we take it very seriously. It is important to consider how to ensure your data remains secure.
API Keys
API keys are the core security mechanisms by which you can push and query your data. It is important to keep these keys safe by controlling where these keys exist and who has access to them.
Each key can either push, query or both. The most important key is the Project Master Key which can perform all of
these actions, as well as administrative functions such as deleting data. Read more about the keys here.
Keeping API Keys Secure
You should carefully consider when and which API keys to expose to users.
Crucially, you should never expose your Project Master Key to users or embed it in client applications.
If this key does get compromised, you can reset it.
If you embed API keys in client applications, you should consider these keys as fully accessible to anyone having access to that client application. This includes both mobile and web applications.
Pushing events securely
While you can use a Push Key to prevent clients from querying events, you cannot restrict the collections or events
clients can push to the API. Unfortunately, this is the nature of tracking events directly client-side and opens the
door to malicious users potentially sending bad data.
In many circumstances, this is not an issue as users can already generate bad data simply by using your application in an incorrect way, generating events with bad or invalid data. In circumstances where you absolutely cannot withstand bad event data, you should consider pushing the events server-side from a service under your control.
Finally, if a Push Key is compromised or being used maliciously, you can always reset it by resetting the master key.
Querying events securely
To query events, you must use an API key that has query permissions. By default, a Query Key has full access to all
events in all collections in your project. If this key is exposed, a client could execute any type of query on your
collections.
You have a number of options on querying events securely:
For internal querying or dashboard, you may consider it acceptable to expose the normal
Query Keyin client applications. Keep in mind that this key can execute any query on any collection in the project.Generate a filtered key, which applies a specific set of filters to all queries executed by clients with the key.
Only allow clients to execute queries via a service you control, which in turn executes queries via the Connect API server-side.
Finally, if a Query Key is compromised or being used maliciously, you can always reset it by resetting the master key.
Resetting the master key
Resetting the Project Master Key will invalidate the previous key and generate a new, random key. This action will also
reset all other keys for the project (including the push, query and any filter keys generated).
Doing this is irreversible and would prevent all applications with existing keys from pushing to or querying the project.
You can only reset the master key in the projects section of the admin console.
Filtered keys
Filtered keys allows you to create an API key that can either push or query, and in the case of querying, apply one or more filters to all queries executed with the key.
This allows you to have finer control over security and what data clients can access, especially in multi-tenant environments.
Filters are only applied to queries
Any filters specified in your filtered key only apply to querying. We currently do not support applying filters to restrict the pushing of events.
Filtered keys can only push or query (as you specify), never administrative functions or deleting data.
Generating a filtered key
Filtered keys are generated and encrypted with the Project Master Key. You do not have to register the
filtered key with the Connect service.
It is important that you never generate filtered keys client-side and always ensure the are generated by a secure, server-side service.
Create a JSON object describing the access allowed by the key, including any filters, for example:
{ "filters": { "customerId": 1234 }, "canQuery": true, "canPush": false }Property Type Description filtersobjectThe filters to apply all queries executed when using the key. This uses the same specification for defining filters when querying normally. canQuerybooleanWhether or not the key can be used to execute queries. If false, thefiltersproperty is ignored (as it does not applying to pushing).canPushbooleanWhether or not the key can be used to push events. Serialize the JSON object to a string.
Generate a random 128-bit initialization vector (IV).
Encrypt the JSON string using AES256-CBC with PKCS7 padding (128-bit block size) using the 128-bit random IV we generated in the previous step and the
Project Master Keyas the encryption key.Convert the IV and cipher text to hex strings and combine them (IV first, followed by the cipher text), separating them with a hyphen (-). The resulting key should look something like this:
5D07A77D87D5B20FA5508303F748A43B-DDFA284A9341068A704A846E83ACF49069D960632C4F74A44B5EE330073F79A8324ADC91023F88F63AAE4507F3D119B5C7F31A2D7D9616408E9665EC6C1DEBE3
You can now use this API key to either push events or execute queries depending on the canPush
and canQuery properties, respectively.
Finally, if a filtered key is compromised or being used maliciously, you can always reset it by resetting the master key.
Modeling your events
When using Connect to analyze and visualize your data, it is important to understand how best to model your events. The way you structure your events will directly affect your ability to answer questions with your data. It is therefore important to consider up-front the kind of questions you anticipate answering.
What is an event?
An event is an action that occurs at a specific point in time. To answer "why did this event occur?", our event needs to contain rich details about what the "world" looked like at that point in time.
Put simply, events = action + time + state.
For example, imagine you are writing an exercise activity tracker app. We want to give users of your app the ability to analyze their performance over time. This is an event produced by our hypothetical activity tracker app:
NSDictionary *myEvent = @{
@"type": @"cycling",
@"timestamp": [NSDate date],
@"duration": @67,
@"distance": @21255,
@"caloriesBurned": @455,
@"maxHeartRate": @182,
@"user": @{
@"id": @698396,
@"firstName": @"Bruce",
@"lastName": @"Jones",
@"age": @35
}
};
let myEvent = [
"type": "cycling",
"timestamp": NSDate(),
"duration": 67,
"distance": 21255,
"caloriesBurned": 455,
"maxHeartRate": 182,
"user": [
"id": 698396,
"firstName": "Bruce",
"lastName": "Jones",
"age": 35
]
]
Action
What happened? In the above example, the action is an activity was completed.
In most circumstances, we group all events of the same action into a single collection.
In this case, we could call our collection activityCompleted, or alternatively, just activity.
Time
When did it happen? In the above example, we specified the start time of the activity as the value of the timestamp property. The top-level timestamp property is a special property in Connect. This is because time is an essential property of event data - it's not optional.
When an event is pushed to Connect, the current time is assigned to the timestamp property if no value was provided by you.
State
What do we know about this action? What do we know about the entities associated with this action? What do we know about the "world" at this moment in time? Every property in our event, besides the timestamp and the name of the collection, serves to answer those questions. This is the most important aspect of our event - it's where all the answers live.
The richer the data you provide in your event, the more questions you can answer for your users, therefore it's important to enrich your events with as much information as possible. In stark contrast to the relational model where you would store this related information in separate tables and join at query time, in the event model this data is denormalized into each event, so as to know the state of the "world" at the point in time of the event.
Collections
It is important when modeling your events to consider how you intend to group those events into collections. This is a careful balance between events being broad enough to answer queries for your users, while specific enough to be manageable.
In our activity example, the activity contains different properties based on what the type of activity. Our cycling activity contains properties associated with the bike that was used, while a kayaking activity may contain properties associated with a kayak that is used.
Because a kayaking event may have different properties to a running event, it might seem logical to put each of them in distinct collections. However, if we
had distinct cycling, running and kayaking collections, we would lose the opportunity to query details that are common to all activities.
As a general rule, consider the common action among your events and decide if the specific variants of that action warrant grouping those events together.
Structuring your events
Events have the following core properties:
- Denormalized
- Immutable
- Rich/nested
- Schemaless
It is also important to consider how to group events into collections to enable future queries to be answered.
Events are denormalized
Consider our example event again, notice the age property of the user:
NSDictionary *myEvent = @{
@"type": @"cycling",
...
@"user": @{
@"id": @698396,
@"firstName": @"Bruce",
@"lastName": @"Jones",
@"age": @35
}
};
let myEvent = [
"type": "cycling",
...
"user": [
"id": 698396,
"firstName": "Bruce",
"lastName": "Jones",
"age": 35
]
]
The user's age is going to be duplicated in every activity he/she completes throughout the year. This may seem inefficient; however, remember that Connect is about analyzing. This denormalization is a real win for analysis; the key is that event data stores state over time, rather than merely current state. This helps us answer questions about why something happened, because we know what the "world" looked like at that point of time.
For example imagine we wanted to chart the average distance cycled per ride, grouped by the age of the rider at the time of the ride. We could simply execute the following query:
var query = connect.query('activity')
.select({ averageDistance: { avg: 'distance' } })
.groupBy('user.age');
var chart = connect.chart(query, '#chart', {
title: 'Average distance per activity by age',
chart: { type: 'bar' }
});
It's this persistence of state over time that makes event data perfect for analysis.
Events are immutable
By their very nature, events cannot change, as they always record state at the point in time of the event. This is also the reason to record as much rich information about the event and "state of the world" as possible.
For example, in our example event above, while Bruce Jones may now be many years older, at the time he completed his bike ride, he was 35 years of age. By ensuring this event remains immutable, we can correctly analyze bike riding over time by 35-year-olds.
Consider events as recording history - as much as we'd occasionally like to, we can't change history!
Events are rich and nested
Events are rich in that they specify very detailed state. They specify details about the event itself, the entities involved and the state of the "world" at that point in time.
Consider our example activity event - the top level type property describes something about the activity itself (a run, a bike ride, a kayak etc.). The user property specifies rich information about the actor who performed the event. In this case it's the person who completed the activity, complete with their name and age.
In reality, though, we may decide to include a few other nested entities in our event, for example:
NSDictionary *myEvent = @{
@"type": @"cycling",
...
@"user": @{
@"id": @698396,
@"firstName": @"Bruce",
...
},
@"bike": @{
@"id": @231806,
@"brand": @"Specialized",
@"model": @"S-Works Venge"
},
@"weather": @{
@"condition": @"Raining",
@"temperature": @21,
@"humidity": @99,
@"wind": @17
}
};
let myEvent = [
"type": "cycling",
...
"user": [
"id": 698396,
"firstName": "Bruce",
...
],
"bike": [
"id": 231806,
"brand": "Specialized",
"model": "S-Works Venge"
],
"weather": [
"condition": "Raining",
"temperature": 21,
"humidity": 99,
"wind": 17
]
]
Note our event now includes details about the bike used and the weather conditions at the time of the activity. By adding this extra bike state information to our event, we have opened up extra possibilities for interrogating our data. For example, we can now query the average distance cycled by each model of bike that was built by "Specialized":
var query = connect.query('activity')
.select({ averageDistance: { avg: 'distance' } })
.groupBy('bike.model');
.filter({
'bike.brand': 'Specialized'
});
The weather also provides us with exciting insights - what did the world look like at this point in time? What was the weather like? Storing this data allows us to answer yet more questions. We can test our hypothesis that "older people are less scared of riding in the rain" by simply charting the following query:
var query = connect.query('activity')
.select({ averageDistance: { avg: 'distance' } })
.groupBy(['user.age', 'weather.condition']);
As you can see, the richer and more denormalized the event, the more interesting answers can be derived when later querying.
Events are schemaless
Events in Connect should be considered semi-structured - that is, they have an inherent structure, but it is not defined. This means you can, and should, push as much detailed information about an event and the state of the "world" as possible. Moreover, this allows you to improve your schema over time and add extra information about new events as that information becomes available.
Restrictions
While you can post almost any event structure to Connect, there are a few, by-design restrictions.
Property names
You cannot have any property in the root document beginning with "tp_". This is because we prefix our own internal properties with this. Internally, we merge our properties into your events for performance at query time.
The property "_id" is reserved and cannot be pushed.
The properties "id" and "timestamp" have special purposes. These allow consumers to specify a unique ID per event and override the event's timestamp respectively. You cannot use the "id" property in queries. Refer to "reliability of events" and "timestamps" for information.
The length of property names can't exceed 255 characters. If you need property names longer than this, you probably need to reconsider the structure of your event!
Properties cannot include a dot in their names. This is because dots are used in querying to access nested properties. The following is an example of an invalid event property due to a dot in the name:
NSDictionary *event = @{
@"invalid.property": @"value"
};
let event = [
"invalid.property": "value"
]
Arrays
While you can create events with arrays, it is currently not possible to take advantage of these arrays at query time. Therefore, you should avoid using arrays in your events unless you plan to export the raw events.
Distinct count
Distinct count is currently not supported for querying, therefore you should consider how to structure your event if your application relies on this.