Telemetry Policy

Effective as of 27 november 2023. LAst Updated: 26 february 2024.

Welcome to Unify! 
Whether you’re interested in deploying models efficiently or developing seamlessly, we want to make sure we provide you with the best experience. For that end, we need to process some of your data as you make use of our APIs.

In the sections below, we outline our telemetry policy for each of:
Ivy Transpiler Telemetry Policy
⦁ About The Policy
⦁ Why do we collect data?
⦁ How do we collect data?
⦁ Telemetry Modes
⦁ Setting Modes
Model Hub Telemetry Policy
⦁ About The Policy
⦁ Why do we collect data?
⦁ How do we collect data?

Ivy Transpiler Telemetry Policy

About the policy
We only collect data when using Ivy’s transpiler, and we want to give users as much flexibility as possible when it comes to data collection. Therefore, if you have not opted out, there are five different data sharing modes for you to choose from. 😄
Keep in mind that any collected data is string-based and as concise as possible, this is, no training data, model parameter, or input array to your function or model will leave your computer at any time in terms of telemetry.

Why do we collect data?
We use the data to learn about your usage of the transpiler, helping us prioritize the development of features and the improvement of the most used functions across all the versions of all the frameworks when it comes to compilation and transpilation. The objective is to make Ivy a much better framework, optimized for actual usage by our community. More specifically, this means more time will be spent on improving the functions, frameworks, and computation graphs that matter most for your projects.

How do we collect data?
When you use your transpiler API key, telemetry data will be stored in a database as outlined in the Telemetry Modes section below. If you wish your data to be anonimized, you can opt out from data association in your account settings. In that case, the key is a securely hashed version of the IP address from where the call was made.

Telemetry Modes
This section outlines the different modes available to all users, as well as the data that is collected for each one when calling ivy.compile, ivy.transpile, or ivy.unify.

Stealth Mode:
No data is collected at all. This mode is limited to users with the enterprise license only.

API Calls Mode
:
We collect the Ivy functions signature and timestamp. For example, the message produced by this function call would be:

comp_fn = ivy.compile(fn, to=”torch”, include_generators=True)

{
    "function":
ivy.compile,
    "timestamp": 108586,
    "flags": {"to": "torch", "include_generators": "True"}
}

This is helpful to understand the typical usage of Ivy, making it possible to focus the development on the more widely used features and use cases of Ivy.

Functional API Mode:
We collect the Ivy functions signature, timestamp, and the set of used functions from the corresponding functional API, without their frequencies. A sample message in this case would be similar to:

comp_fn = ivy.compile(fn, to=”torch”, include_generators=True)

{
    "function":
ivy.compile,
    "timestamp": 108586,
    "flags": {"to": "torch", "include_generators": "True"},
    "functions": ["torch.add", "torch.matmul"]
}


As mentioned above, this helps prioritize the addition of new functions and fixing of existing ones based on actual user usage of them, rather than relying on guess work and potentially focusing our efforts on functions that may not be wanted or needed by the users.

Functional API Frequency Mode:
We collect the Ivy functions signature, timestamp, and the set of used functions from the corresponding functional API along with the number of calls to each function. A sample message if this mode is selected would be:

comp_fn = ivy.compile(fn, to=”torch”, include_generators=True)

{
    "function":
ivy.compile,
    "timestamp": 108586,
    "flags": {"to": "torch", "include_generators": "True"},
    "functions": [["torch.add", 2], ["torch.matmul", 1]]
}


Similar to the previous mode, this helps even more to prioritize certain functions that need to be included, fixed, or optimized based on their actual usage. The usage frequencies give another very helpful data point to be even more accurate with function prioritization, as it’s now possible to focus on the functions depending on how commonly used they are.

Graph Mode:
We collect Ivy functions signature, timestamp and a text-based representation of the computation graph. This mode is set by default on pip install ivy. If this mode is selected, telemetry messages will be similar to:

comp_fn = ivy.compile(fn, to=”torch”, include_generators=True)

{
    "function":
ivy.compile,
    "timestamp": 108586,
    "flags": {"to": "torch", "include_generators": "True"},
    "graph": "<graph_representation>"
}


where <graph_representation> would be a string containing:

p708217 = args[0]
p862415 = args[1]
p817145 = torch.add(p708217, p862415)
p913484 = torch.add(p708217, p817145)
p868273 = torch.matmul(p913484, p817145)
return p868273


In this case, telemetry helps not only with the function-level prioritization, but also with sub-function fixes and optimizations in Ivy’s functional API and Ivy’s frontends, as it’s now possible to distinguish which parameters or flags are used more often by users of the compiler and the transpiler.
Furthermore, minimal version of the computational graph allows us to recreate and explore any error that arises during compilation or transpilation, which once again enables the prioritization of bug fixes and features based on real-world usage.

Below, you can find a summary of the modes and their corresponding data levels:

stealth
api-calls
functional-api
api frequency
graph
Ivy signature
Time stamp
Functions
Frequency
Graph repr.

Setting The Telemetry Mode

By default, the telemetry mode is
graph. To change this setting, you should modify the config.json file that you can find inside of your .ivy directory. More specifically, you will have to change the data_level field. For example, if you want to limit the telemetry to avoid sending the computational graph, you can modify config.json so that it looks like this:

{
   "data_level": "functional-api-frequency"
}



Model Hub Telemetry Policy

About the policy
When using our Model Hub, either through the web interface or your API key, we collect some of the input data to process the requests.Specifically, we only collect data needed to compute the cost of the request and performance of the provider. This translates into collecting request-specific metadata like the number of input and output tokens when querying language modes. We also store performance metrics about the service such as the request latency. Conversely, we do not store the actual inputs and outputs.

Why do we collect data?
We use the data to learn about your usage of the model hub, helping us understand what part of the service is most useful for you, and focus more of our development efforts on maintaining and expanding integrations and features related that would most benefit your usage. We want to make the Model Hub a centralized place for efficient model deployment, and we need performance data to guide users to the best deployment options for their needs as part of our ongoing benchmarks of deployment options.

How do we collect data?
When you use the Model Hub, either via the web interface or through your API key, we process the data you send to any of our API endpoints, and store metadata about it. When querying the Model Hub endpoints, your queries are redirected to different providers depending on how you specify your request inputs. Therefore, keep in mind that these services may interact with your data to different extents. This means your input is also subject to the selected providers’ respective telemetry policies as well. For more details on the providers’ policies, we encourage you to visit their websites to learn more.


We hope this helps you understand what data we process as part of our services and how they contribute to improving each!