Package com.cognite.client

Class Datasets

java.lang.Object

com.cognite.client.Datasets

public abstract class Datasets
extends Object

This class represents the Cognite datasets api endpoint It provides methods for reading and writing DataSet

Field Summary

Fields

Modifier and Type	Field	Description
protected static org.slf4j.Logger	LOG

Constructor Summary

Constructors

Constructor	Description
Datasets()

Method Summary

Modifier and Type	Method	Description
protected Request	addAuthInfo(Request request)	Adds the required authentication information into the request object.
protected Aggregate	aggregate(ResourceType resourceType, Request requestParameters)	Performs an item aggregation request to Cognite Data Fusion.
Aggregate	aggregate(Request requestParameters)	Performs an item aggregation to Cognite Data Fusion.
protected List<String>	buildPartitionsList(int noPartitions)	Builds an array of partition specifications for parallel retrieval from the Cognite api.
protected List<Item>	deDuplicate(Collection<Item> itemList)	De-duplicates a collection of Item.
abstract CogniteClient	getClient()
protected Iterator<CompletableFuture<ResponseItems<String>>>	getListResponseIterator(ResourceType resourceType, Request requestParameters)
protected boolean	itemsHaveId(Collection<Item> items)	Returns true if all items contain either an externalId or id.
Iterator<List<DataSet>>	list()	Returns all DataSet objects.
Iterator<List<DataSet>>	list(Request requestParameters)	Return all DataSet object that matches the filters set in the Request.
Iterator<List<DataSet>>	list(Request requestParameters, String... partitions)	Return all DataSet objects that matches the filters set in the Request for the specific partitions.
protected Iterator<List<String>>	listJson(ResourceType resourceType, Request requestParameters, String... partitions)	Will return the results from a list / filter api endpoint.
protected Iterator<List<String>>	listJson(ResourceType resourceType, Request requestParameters, String partitionKey, String... partitions)	Will return the results from a list / filter api endpoint.
protected Map<String,Item>	mapItemToId(Collection<Item> items)	Maps all items to their externalId (primary) or id (secondary).
static Datasets	of(CogniteClient client)	Construct a new Datasets object using the provided configuration.
protected List<Item>	parseItems(List<String> input)	Parses a list of item object in json representation to typed objects.
protected String	parseName(String json)	Returns the name attribute value from a json input.
protected String	parseString(String itemJson, String fieldName)	Try parsing the specified Json path as a String.
List<DataSet>	retrieve(long... id)	Retrieve datasets by internal id.
List<DataSet>	retrieve(String... externalId)	Retrieve datasets by externalId.
List<DataSet>	retrieve(List<Item> items)	Retrieves datasets by externalId / id.
protected List<String>	retrieveJson(ResourceType resourceType, Collection<Item> items)	Retrieve items by id.
protected List<String>	retrieveJson(ResourceType resourceType, Collection<Item> items, Map<String,Object> parameters)	Retrieve items by id.
protected List<Map<String,Object>>	toRequestItems(Collection<Item> itemList)	Converts a list of Item to a request object structure (that can later be parsed to Json).
List<DataSet>	upsert(List<DataSet> datasets)	Creates or update a set of DataSet objects.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

protected static final org.slf4j.Logger LOG

Constructor Detail

Datasets

public Datasets()

Method Detail

of

public static Datasets of(CogniteClient client)

Construct a new Datasets object using the provided configuration. This method is intended for internal use--SDK clients should always use CogniteClient as the entry point to this class.

Parameters:

client - The CogniteClient to use for configuration settings.

Returns:

The datasets api object.

list

public Iterator<List<DataSet>> list()
                             throws Exception

Returns all DataSet objects.

Example:

 
     List<DataSet> listDataSetsResults = new ArrayList<>();
     client.datasets()
           .list()
           .forEachRemaining(batch -> listDataSetsResults.addAll(batch));
 
 

API Reference - Filter data sets

Throws:

Exception

See Also:

list(Request), CogniteClient, CogniteClient.datasets()

list

public Iterator<List<DataSet>> list(Request requestParameters)
                             throws Exception

Return all DataSet object that matches the filters set in the Request. The results are paged through / iterated over via an Iterator--the entire results set is not buffered in memory, but streamed in "pages" from the Cognite api. If you need to buffer entire results set, then you have to stream these results into your own data structure. The datasets are retrieved using multiple, parallel request streams towards the Cognite api. The number of parallel streams are set in the ClientConfig.

Example:

 
     List<DataSet> listDataSetsResults = new ArrayList<>();
     client.datasets()
           .list(Request.create()
                            .withFilterParameter("writeProtected", true))
           .forEachRemaining(batch -> listDataSetsResults.addAll(batch));
 
 

API Reference - Filter data sets

Parameters:

requestParameters - The filters to use for retrieving datasets.

Returns:

An Iterator to page through the results set.

Throws:

Exception

See Also:

list(Request,String...), CogniteClient, CogniteClient.datasets()

list

public Iterator<List<DataSet>> list(Request requestParameters,
                                    String... partitions)
                             throws Exception

Return all DataSet objects that matches the filters set in the Request for the specific partitions. This method is intended for advanced use cases where you need direct control over the individual partitions. For example, when using the SDK in a distributed computing environment. The results are paged through / iterated over via an Iterator--the entire results set is now buffered in memory, but streamed in "pages" from the Cognite api. If you need to buffer the entire results set, then you have to stream these results into your own data strcture.

Example:

 
     List<DataSet> listDataSetsResults = new ArrayList<>();
     client.datasets()
           .list(Request.create()
                            .withFilterParameter("writeProtected", true),
                                 "1/8","2/8","3/8","4/8","5/8","6/8","7/8","8/8")
           .forEachRemaining(batch -> listDataSetsResults.addAll(batch));
 
 

API Reference - Filter data sets

Parameters:

requestParameters - The filters to use for retrieving the datasets

partitions - The partitions to include

Returns:

An Iterator to page through the results set

Throws:

Exception

See Also:

listJson(ResourceType,Request,String...), CogniteClient, CogniteClient.datasets()

retrieve

public List<DataSet> retrieve(String... externalId)
                       throws Exception

Retrieve datasets by externalId.

Example:

 
      List<DataSet> retrievedDataSets = client.datasets().retrieve("1","2");
 
 

API Reference - Retrieve data sets

Parameters:

externalId - The externalIds to retrieve

Returns:

The retrieved datasets.

Throws:

Exception

See Also:

retrieve(List), CogniteClient, CogniteClient.datasets()

retrieve

public List<DataSet> retrieve(long... id)
                       throws Exception

Retrieve datasets by internal id.

Example:

 
      List<DataSet> retrievedDataSets = client.datasets().retrieve(1,2);
 
 

API Reference - Retrieve data sets

Parameters:

id - The ids to retrieve

Returns:

The retrieved datasets.

Throws:

Exception

See Also:

retrieve(List), CogniteClient, CogniteClient.datasets()

retrieve

public List<DataSet> retrieve(List<Item> items)
                       throws Exception

Retrieves datasets by externalId / id.

Example:

 
      List<Item> items = List.of(Item.newBuilder().setExternalId("1").build());
      List<DataSet> retrievedDataSets = client.datasets().retrieve(items);
 
 

API Reference - Retrieve data sets

Parameters:

items - The item(s) externalId / id to retrieve.

Returns:

The retrieved datasets.

Throws:

Exception

See Also:

retrieveJson(ResourceType, Collection), CogniteClient, CogniteClient.datasets()

aggregate

public Aggregate aggregate(Request requestParameters)
                    throws Exception

Performs an item aggregation to Cognite Data Fusion. The default aggregation is a total item count based on the (optional) filters in the request. Multiple aggregation types are supported. Please refer to the Cognite API specification for more information on the possible settings.

Example:

 
      Aggregate aggregateResult = client.datasets()
                  .aggregate(Request.create()
                  .withFilterParameter("writeProtected", true));
 
 

API Reference - Aggregate data sets

Parameters:

requestParameters - The filtering and aggregates specification.

Returns:

The aggregation results.

Throws:

Exception

See Also:

aggregate(ResourceType,Request), CogniteClient, CogniteClient.datasets(), Cognite API v1 specification

upsert

public List<DataSet> upsert(List<DataSet> datasets)
                     throws Exception

Creates or update a set of DataSet objects. If it si a new DataSet object (based on the id / externalId, thenit will be created. If an Iterator object already exists in Cognite Data Fusion, it will be updated. The update behaviour is specified via the update mode in the ClientConfig settings.

Example:

 
      List<DataSet> upsertDataSetsList = // List of DataSet;
      client.datasets().upsert(upsertDataSetsList);
 
 

API Reference - Create data sets
API Reference - Update the attributes of data sets.

Parameters:

datasets - The datasets to upsert

Returns:

The upserted datasets

Throws:

Exception

See Also:

CogniteClient, CogniteClient.datasets()

getClient

public abstract CogniteClient getClient()

buildPartitionsList

protected List<String> buildPartitionsList(int noPartitions)

Builds an array of partition specifications for parallel retrieval from the Cognite api. This specification is used as a parameter together with the filter / list endpoints. The number of partitions indicate the number of parallel read streams. Employ one partition specification per read stream.

Example:

 
      List<String> partitions = buildPartitionsList(getClient().getClientConfig().getNoListPartitions());
 
 

Parameters:

noPartitions - The total number of partitions

Returns:

a List of partition specifications

listJson

protected Iterator<List<String>> listJson(ResourceType resourceType,
                                          Request requestParameters,
                                          String... partitions)
                                   throws Exception

Will return the results from a list / filter api endpoint. For example, the filter assets endpoint. The results are paged through / iterated over via an Iterator--the entire results set is not buffered in memory, but streamed in "pages" from the Cognite api. If you need to buffer the entire results set, then you have to stream these results into your own data structure. This method support parallel retrieval via a set of partition specifications. The specified partitions will be collected and merged together before being returned via the Iterator.

Example:

 
      Iterator<List<String>> result = listJson(resourceType, requestParameters, partitions);
 
 

Parameters:

resourceType - The resource type to query / filter / list. Ex. event, asset, time series.

requestParameters - The query / filter specification. Follows the Cognite api request parameters.

partitions - An optional set of partitions to read via.

Returns:

an Iterator over the results set.

Throws:

Exception

See Also:

listJson(ResourceType,Request,String,String...)

listJson

protected Iterator<List<String>> listJson(ResourceType resourceType,
                                          Request requestParameters,
                                          String partitionKey,
                                          String... partitions)
                                   throws Exception

Will return the results from a list / filter api endpoint. For example, the filter assets endpoint. The results are paged through / iterated over via an Iterator--the entire results set is not buffered in memory, but streamed in "pages" from the Cognite api. If you need to buffer the entire results set, then you have to stream these results into your own data structure. This method support parallel retrieval via a set of partition specifications. The specified partitions will be collected and merged together before being returned via the Iterator.

Example:

 
      Iterator<List<String>> result = listJson(resourceType, requestParameters, partitionKey, partitions);
 
 

Parameters:

resourceType - The resource type to query / filter / list. Ex. event, asset, time series.

requestParameters - The query / filter specification. Follows the Cognite api request parameters.

partitionKey - The key to use for the partitions in the read request. For example partition or cursor.

partitions - An optional set of partitions to read via.

Returns:

an Iterator over the results set.

Throws:

Exception

retrieveJson

protected List<String> retrieveJson(ResourceType resourceType,
                                    Collection<Item> items)
                             throws Exception

Retrieve items by id. Will ignore unknown ids by default.

Example:

 
      Collection<Item> items = //Collection of items with ids;
      List<String> result = retrieveJson(resourceType, items);
 
 

Parameters:

resourceType - The item resource type (Event, Asset, etc.) to retrieve.

items - The item(s) externalId / id to retrieve.

Returns:

The items in Json representation.

Throws:

Exception

See Also:

retrieveJson(ResourceType,Collection,Map)

retrieveJson

protected List<String> retrieveJson(ResourceType resourceType,
                                    Collection<Item> items,
                                    Map<String,Object> parameters)
                             throws Exception

Retrieve items by id. This version allows you to explicitly set additional parameters for the retrieve request. For example: <"ignoreUnknownIds", true> and <"fetchResources", true>.

Example:

 
      Collection<Item> items = //Collection of items with ids;
      Map<String, Object> parameters = //Parameters;
      List<String> result = retrieveJson(resourceType, items, parameters);
 
 

Parameters:

resourceType - The item resource type (Event, Asset, etc.) to retrieve.

items - The item(s) externalId / id to retrieve.

parameters - Additional parameters for the request. For example <"ignoreUnknownIds", true>

Returns:

The items in Json representation.

Throws:

Exception

aggregate

protected Aggregate aggregate(ResourceType resourceType,
                              Request requestParameters)
                       throws Exception

Performs an item aggregation request to Cognite Data Fusion. The default aggregation is a total item count based on the (optional) filters in the request. Some resource types, for example Event, supports multiple types of aggregation.

Example:

 
      Aggregate aggregateResult = aggregate(resourceType,requestParameters);
 
 

Parameters:

resourceType - The resource type to perform aggregation of.

requestParameters - The request containing filters.

Returns:

The aggregation result.

Throws:

Exception

See Also:

Cognite API v1 specification

addAuthInfo

protected Request addAuthInfo(Request request)
                       throws Exception

Adds the required authentication information into the request object. If the request object already have complete auth info nothing will be added. The following authentication schemes are supported: 1) API key. When using an api key, this service will look up the corresponding project/tenant to issue requests to.

Example:

 
      Request requestParams = addAuthInfo(request);
 
 

Parameters:

request - The request to enrich with auth information.

Returns:

The request parameters with auth info added to it.

Throws:

Exception

getListResponseIterator

protected Iterator<CompletableFuture<ResponseItems<String>>> getListResponseIterator(ResourceType resourceType,
                                                                                     Request requestParameters)
                                                                              throws Exception

Throws:

Exception

parseItems

protected List<Item> parseItems(List<String> input)
                         throws Exception

Parses a list of item object in json representation to typed objects.

Example:

 
      List<String> input = //List of json;
      List<Item> resultList = parseItems(input);
 
 

Parameters:

input - the item list in Json string representation

Returns:

the parsed item objects

Throws:

Exception

toRequestItems

protected List<Map<String,Object>> toRequestItems(Collection<Item> itemList)

Converts a list of Item to a request object structure (that can later be parsed to Json).

Example:

 
      Collection<Item> itemList = //Collection of items;
      List<Map<String, Object>> result = toRequestItems(itemList);
 
 

Parameters:

itemList - The items to parse.

Returns:

The items in request item object form.

deDuplicate

protected List<Item> deDuplicate(Collection<Item> itemList)

De-duplicates a collection of Item.

Example:

 
      Collection<Item> itemList = //Collection of items;
      List<Item> result = deDuplicate(itemList);
 
 

Parameters:

itemList -

Returns:

itemsHaveId

protected boolean itemsHaveId(Collection<Item> items)

Returns true if all items contain either an externalId or id.

Example:

 
      Collection<Item> items = //Collection of items;
      boolean result = itemsHaveId(items);
 
 

Parameters:

items -

Returns:

mapItemToId

protected Map<String,Item> mapItemToId(Collection<Item> items)

Maps all items to their externalId (primary) or id (secondary). If the id function does not return any identity, the item will be mapped to the empty string. Via the identity mapping, this function will also perform deduplication of the input items.

Example:

 
      Collection<Item> items = //Collection of items;
      Map<String, Item> result = mapItemToId(items);
 
 

Parameters:

items - the items to map to externalId / id.

Returns:

the Map with all items mapped to externalId / id.

parseString

protected String parseString(String itemJson,
                             String fieldName)

Try parsing the specified Json path as a String.

Example:

 
      String json = //String of json object
      String result = parseString(json, "name");
 
 

Parameters:

itemJson - The Json string

fieldName - The Json path to parse

Returns:

The Json path as a String.

parseName

protected String parseName(String json)

Returns the name attribute value from a json input.

Example:

 
      String json = //String of json object
      String result = parseName(json);
 
 

Parameters:

json - the json to parse

Returns:

The name value