Package com.cognite.client

Class DataPoints

    • Field Detail

      • LOG

        protected static final org.slf4j.Logger LOG

    • Constructor Detail

      • DataPoints

        public DataPoints()

    • Method Detail

      • of

        public static DataPoints of​(CogniteClient client)
        Construct a new DataPoints object using the provided configuration. This method is intended for internal use--SDK clients should always use CogniteClient as the entry point to this class.
        Parameters:
        client - The CogniteClient to use for configuration settings.
        Returns:
        the assets api object.

      • retrieve

        public Iterator<List<TimeseriesPoint>> retrieve​(Request requestParameters)
                                         throws Exception
        Returns all TimeseriesPoint objects that matches the filters set in the Request. Please note that only root-level filter and aggregate specifications are supported. That is, per-item specifications of time filters and/or aggregations are not supported. If you need to apply different time and/or aggregation specifications, then these should be submitted in separate requests--each using root-level specifications. The results are paged through / iterated over via an Iterator--the entire results set is not buffered in memory, but streamed in "pages" from the Cognite api. If you need to buffer the entire results set, then you have to stream these results into your own data structure.

        Example:

         
      List<TimeseriesPoint> results = new ArrayList<>();
      client.timeseries().dataPoints()
           .retrieve(Request.create().withRootParameter("includeOutsidePoints", true))
           .forEachRemaining(items-> results.addAll(items));
        API Reference - Retrieve data points
        Parameters:
        requestParameters - the filters to use for retrieving the timeseries.
        Returns:
        an Iterator to page through the results set.
        Throws:
        Exception
        See Also:
        CogniteClient, CogniteClient.timeseries(), Timeseries.dataPoints()

      • upsert

        public List<TimeseriesPointPost> upsert​(@NotNull
                                        List<TimeseriesPointPost> dataPoints)
                                 throws Exception
        Creates or update a set of TimeseriesPointPost objects. TimeseriesPointPost is the write-optimized version of a time series data point while TimeseriesPoint is the read-optimized version. If it is a new TimeseriesPointPost object (based on the id / externalId + timestamp, then it will be created. If an TimeseriesPoint object already exists in Cognite Data Fusion, it will be updated. The algorithm runs as follows: 1. Write all TimeseriesPointPost objects to the Cognite API. 2. If one (or more) of the objects fail, check if it is because of missing time series objects--create temp headers. 3. Retry the failed TimeseriesPointPost objects.

        Example:

         
 // Create the time series header.
 List<TimeseriesMetadata> upsertTimeseriesList = List.of(TimeseriesMetadata.newBuilder()
          .setExternalId("my-external-id")
          .setName("test_ts")
          .setIsString(false)
          .setIsStep(false)
          .setDescription("Description")
          .setUnit("TestUnits")
          .putMetadata("type", "sdk-data-generator")
          .putMetadata("sdk-data-generator", "sdk-data-generator")
          .build());
  client.timeseries().upsert(upsertTimeseriesList);

  // Add time series data points.
  List<TimeseriesPointPost> dataPoints = List.of(
          TimeseriesPointPost.newBuilder()
                         .setExternalId("my-external-id")
                         .setTimestamp(Instant.parse("2020-12-03T10:15:30.00Z").toEpochMilli())
                         .setValueNum(ThreadLocalRandom.current().nextLong(-10, 20))
                         .build(),
         TimeseriesPointPost.newBuilder()
                         .setExternalId("my-external-id")
                         .setTimestamp(Instant.parse("2020-12-03T10:16:30.00Z").toEpochMilli())
                         .setValueNum(ThreadLocalRandom.current().nextLong(-10, 20))
                         .build()
  );
  client.timeseries().dataPoints().upsert.upsert(dataPoints);
        API Reference - Insert data points
        Specified by:
        upsert in interface UpsertTarget<TimeseriesPointPost,​TimeseriesPointPost>
        Parameters:
        dataPoints - The data points to upsert
        Returns:
        The upserted data points
        Throws:
        Exception
        See Also:
        CogniteClient, CogniteClient.timeseries()

      • uploadQueue

        public UploadQueue<TimeseriesPointPost,​TimeseriesPointPost> uploadQueue()
        Returns an upload queue. The upload queue helps improve performance by batching items together before uploading them to Cognite Data Fusion. This queue is tuned with a capacity of 500k elements for high-throughput data points.
        Returns:
        The upload queue.

      • retrieveLatest

        public List<TimeseriesPoint> retrieveLatest​(@NotNull
                                            List<Item> items)
                                     throws Exception
        Retrieves the latest (newest) data point for a time series. The Item must specify the externalId / id of the time series. Optionally, you can specify Item.exclusiveEnd to set an upper time boundary. That is, the response will contain the latest data point before the upper time boundary.

        Example:

         
      List<Item> byInternalIds = List.of(Item.newBuilder().setId(10).build());
      List<TimeseriesPoint> result =
              client.timeseries().dataPoints()
                  .retrieveLatest(byInternalIds);
        API Reference - Retrieve latest data point
        Parameters:
        items - The time series to retrieve data point(s) from.
        Returns:
        The latest data point(s)
        Throws:
        Exception
        See Also:
        CogniteClient, CogniteClient.timeseries(), Timeseries.dataPoints()

      • getMaxFrequency

        public double getMaxFrequency​(Request requestParameters,
                              Instant startOfWindow,
                              Instant endOfWindow)
                       throws Exception
        Calculate the max frequency of the TS items in the query. Only numeric data points are considered. In case of string data points, this method returns 0 (i.e. no splitting per time window for string time series). This method is intended for advanced use cases with distributed computing frameworks that implement their own split and parallelization algorithms.

        Example:

         
      double maxFrequency = getMaxFrequency(requestParameters,
                     Instant.ofEpochMilli(startTimestamp),
                     Instant.ofEpochMilli(endTimestamp));
        Parameters:
        requestParameters -
        startOfWindow -
        endOfWindow -
        Returns:
        Throws:
        Exception

      • buildPartitionsList

        protected List<String> buildPartitionsList​(int noPartitions)
        Builds an array of partition specifications for parallel retrieval from the Cognite api. This specification is used as a parameter together with the filter / list endpoints. The number of partitions indicate the number of parallel read streams. Employ one partition specification per read stream.

        Example:

         
      List<String> partitions = buildPartitionsList(getClient().getClientConfig().getNoListPartitions());
        Parameters:
        noPartitions - The total number of partitions
        Returns:
        a List of partition specifications

      • listJson

        protected Iterator<List<String>> listJson​(ResourceType resourceType,
                                          Request requestParameters,
                                          String... partitions)
                                   throws Exception
        Will return the results from a list / filter api endpoint. For example, the filter assets endpoint. The results are paged through / iterated over via an Iterator--the entire results set is not buffered in memory, but streamed in "pages" from the Cognite api. If you need to buffer the entire results set, then you have to stream these results into your own data structure. This method support parallel retrieval via a set of partition specifications. The specified partitions will be collected and merged together before being returned via the Iterator.

        Example:

         
      Iterator<List<String>> result = listJson(resourceType, requestParameters, partitions);
        Parameters:
        resourceType - The resource type to query / filter / list. Ex. event, asset, time series.
        requestParameters - The query / filter specification. Follows the Cognite api request parameters.
        partitions - An optional set of partitions to read via.
        Returns:
        an Iterator over the results set.
        Throws:
        Exception
        See Also:
        listJson(ResourceType,Request,String,String...)

      • listJson

        protected Iterator<List<String>> listJson​(ResourceType resourceType,
                                          Request requestParameters,
                                          String partitionKey,
                                          String... partitions)
                                   throws Exception
        Will return the results from a list / filter api endpoint. For example, the filter assets endpoint. The results are paged through / iterated over via an Iterator--the entire results set is not buffered in memory, but streamed in "pages" from the Cognite api. If you need to buffer the entire results set, then you have to stream these results into your own data structure. This method support parallel retrieval via a set of partition specifications. The specified partitions will be collected and merged together before being returned via the Iterator.

        Example:

         
      Iterator<List<String>> result = listJson(resourceType, requestParameters, partitionKey, partitions);
        Parameters:
        resourceType - The resource type to query / filter / list. Ex. event, asset, time series.
        requestParameters - The query / filter specification. Follows the Cognite api request parameters.
        partitionKey - The key to use for the partitions in the read request. For example partition or cursor.
        partitions - An optional set of partitions to read via.
        Returns:
        an Iterator over the results set.
        Throws:
        Exception

      • retrieveJson

        protected List<String> retrieveJson​(ResourceType resourceType,
                                    Collection<Item> items)
                             throws Exception
        Retrieve items by id. Will ignore unknown ids by default.

        Example:

         
      Collection<Item> items = //Collection of items with ids;
      List<String> result = retrieveJson(resourceType, items);
        Parameters:
        resourceType - The item resource type (Event, Asset, etc.) to retrieve.
        items - The item(s) externalId / id to retrieve.
        Returns:
        The items in Json representation.
        Throws:
        Exception
        See Also:
        retrieveJson(ResourceType,Collection,Map)

      • retrieveJson

        protected List<String> retrieveJson​(ResourceType resourceType,
                                    Collection<Item> items,
                                    Map<String,​Object> parameters)
                             throws Exception
        Retrieve items by id. This version allows you to explicitly set additional parameters for the retrieve request. For example: <"ignoreUnknownIds", true> and <"fetchResources", true>.

        Example:

         
      Collection<Item> items = //Collection of items with ids;
      Map<String, Object> parameters = //Parameters;
      List<String> result = retrieveJson(resourceType, items, parameters);
        Parameters:
        resourceType - The item resource type (Event, Asset, etc.) to retrieve.
        items - The item(s) externalId / id to retrieve.
        parameters - Additional parameters for the request. For example <"ignoreUnknownIds", true>
        Returns:
        The items in Json representation.
        Throws:
        Exception

      • aggregate

        protected Aggregate aggregate​(ResourceType resourceType,
                              Request requestParameters)
                       throws Exception
        Performs an item aggregation request to Cognite Data Fusion. The default aggregation is a total item count based on the (optional) filters in the request. Some resource types, for example Event, supports multiple types of aggregation.

        Example:

         
      Aggregate aggregateResult = aggregate(resourceType,requestParameters);
        Parameters:
        resourceType - The resource type to perform aggregation of.
        requestParameters - The request containing filters.
        Returns:
        The aggregation result.
        Throws:
        Exception
        See Also:
        Cognite API v1 specification

      • addAuthInfo

        protected Request addAuthInfo​(Request request)
                       throws Exception
        Adds the required authentication information into the request object. If the request object already have complete auth info nothing will be added. The following authentication schemes are supported: 1) API key. When using an api key, this service will look up the corresponding project/tenant to issue requests to.

        Example:

         
      Request requestParams = addAuthInfo(request);
        Parameters:
        request - The request to enrich with auth information.
        Returns:
        The request parameters with auth info added to it.
        Throws:
        Exception

      • parseItems

        protected List<Item> parseItems​(List<String> input)
                         throws Exception
        Parses a list of item object in json representation to typed objects.

        Example:

         
      List<String> input = //List of json;
      List<Item> resultList = parseItems(input);
        Parameters:
        input - the item list in Json string representation
        Returns:
        the parsed item objects
        Throws:
        Exception

      • toRequestItems

        protected List<Map<String,​Object>> toRequestItems​(Collection<Item> itemList)
        Converts a list of Item to a request object structure (that can later be parsed to Json).

        Example:

         
      Collection<Item> itemList = //Collection of items;
      List<Map<String, Object>> result = toRequestItems(itemList);
        Parameters:
        itemList - The items to parse.
        Returns:
        The items in request item object form.

      • deDuplicate

        protected List<Item> deDuplicate​(Collection<Item> itemList)
        De-duplicates a collection of Item.

        Example:

         
      Collection<Item> itemList = //Collection of items;
      List<Item> result = deDuplicate(itemList);
        Parameters:
        itemList -
        Returns:

      • itemsHaveId

        protected boolean itemsHaveId​(Collection<Item> items)
        Returns true if all items contain either an externalId or id.

        Example:

         
      Collection<Item> items = //Collection of items;
      boolean result = itemsHaveId(items);
        Parameters:
        items -
        Returns:

      • mapItemToId

        protected Map<String,​Item> mapItemToId​(Collection<Item> items)
        Maps all items to their externalId (primary) or id (secondary). If the id function does not return any identity, the item will be mapped to the empty string. Via the identity mapping, this function will also perform deduplication of the input items.

        Example:

         
      Collection<Item> items = //Collection of items;
      Map<String, Item> result = mapItemToId(items);
        Parameters:
        items - the items to map to externalId / id.
        Returns:
        the Map with all items mapped to externalId / id.

      • parseString

        protected String parseString​(String itemJson,
                             String fieldName)
        Try parsing the specified Json path as a String.

        Example:

         
      String json = //String of json object
      String result = parseString(json, "name");
        Parameters:
        itemJson - The Json string
        fieldName - The Json path to parse
        Returns:
        The Json path as a String.

      • parseName

        protected String parseName​(String json)
        Returns the name attribute value from a json input.

        Example:

         
      String json = //String of json object
      String result = parseName(json);
        Parameters:
        json - the json to parse
        Returns:
        The name value