Package com.cognite.client

Class Files

    • Field Detail

      • LOG

        protected static final org.slf4j.Logger LOG

    • Constructor Detail

      • Files

        public Files()

    • Method Detail

      • of

        public static Files of​(CogniteClient client)
        Constructs a new Files object using the provided client configuration. This method is intended for internal use--SDK clients should always use CogniteClient as the entry point to this class.
        Parameters:
        client - The CogniteClient to use for configuration settings.
        Returns:
        the assets api object.

      • list

        public Iterator<List<FileMetadata>> list​(Request requestParameters)
                                  throws Exception
        Returns all FileMetadata objects that matches the filters set in the Request. The results are paged through / iterated over via an Iterator--the entire results set is not buffered in memory, but streamed in "pages" from the Cognite api. If you need to buffer the entire results set, then you have to stream these results into your own data structure. The assets are retrieved using multiple, parallel request streams towards the Cognite api. The number of parallel streams are set in the ClientConfig.

        Example:

         
      List<FileMetadata> listResults = new ArrayList<>();
      client.files()
              .list(Request.create()
                             .withFilterParameter("source", "source"))
              .forEachRemaining(listResults::addAll);
        API Reference - Filter files
        Parameters:
        requestParameters - the filters to use for retrieving the assets.
        Returns:
        an Iterator to page through the results set.
        Throws:
        Exception
        See Also:
        list(Request,String...), CogniteClient, CogniteClient.files()

      • list

        public Iterator<List<FileMetadata>> list​(Request requestParameters,
                                         String... partitions)
                                  throws Exception
        Returns all FileMetadata objects that matches the filters set in the Request for the specified partitions. This is method is intended for advanced use cases where you need direct control over the individual partitions. For example, when using the SDK in a distributed computing environment. The results are paged through / iterated over via an Iterator--the entire results set is not buffered in memory, but streamed in "pages" from the Cognite api. If you need to buffer the entire results set, then you have to stream these results into your own data structure.

        Example:

         
      List<FileMetadata> listResults = new ArrayList<>();
      client.files()
              .list(Request.create()
                             .withFilterParameter("source", "source"),
                                  "1/8","2/8","3/8","4/8","5/8","6/8","7/8","8/8")
              .forEachRemaining(listResults::addAll);
        API Reference - Filter files
        Parameters:
        requestParameters - the filters to use for retrieving the files.
        partitions - the partitions to include.
        Returns:
        an Iterator to page through the results set.
        Throws:
        Exception
        See Also:
        listJson(ResourceType,Request,String...), CogniteClient, CogniteClient.files()

      • metadataUploadQueue

        public UploadQueue<FileMetadata,​FileMetadata> metadataUploadQueue()
        Returns an upload queue for FileMetadata. You can use this queue to push file header/metadata updates to files that have already been uploaded to CDF. The upload queue helps improve performance by batching items together before uploading them to Cognite Data Fusion.
        Returns:
        The upload queue.

      • fileContainerUploadQueue

        public UploadQueue<FileContainer,​FileMetadata> fileContainerUploadQueue()
        Returns an upload queue for FileContainer. You can use this queue to push file binaries and headers (FileContainer) to CDF. The upload queue helps improve performance by batching items together before uploading them to Cognite Data Fusion.
        Returns:
        The upload queue.

      • upload

        public List<FileMetadata> upload​(List<FileContainer> files)
                          throws Exception
        Uploads a set of file headers and binaries to Cognite Data Fusion. The file binary can either be placed in-memory in the file container (as a byte string) or referenced (by URI) to a blob store.

        Example:

         
      Path fileAOriginal = Paths.get("path + file");
      List<FileContainer> files = new ArrayList<>();
      FileMetadata fileMetadata = FileMetadata.newBuilder()
           .setExternalId("10")
           .setName("test_file_.test")
           .setSource("sdk-data-generator")
           .putMetadata("type", "sdk-data-generator")
          .build();

      FileContainer fileContainer = FileContainer.newBuilder()
           .setFileMetadata(fileMetadata)
           .setFileBinary(FileBinary.newBuilder()
                .setBinaryUri(fileAOriginal.toUri().toString()))
           .build();
      files.add(fileContainer);

      List<FileMetadata> uploadFileResult =
                  client.files().upload(files);
        API Reference - Upload file
        Specified by:
        upload in interface UploadTarget<FileContainer,​FileMetadata>
        Parameters:
        files - The files to upload.
        Returns:
        The file metadata / headers for the uploaded files.
        Throws:
        Exception
        See Also:
        upload(List, boolean), CogniteClient, CogniteClient.files()

      • upload

        public List<FileMetadata> upload​(@NotNull
                                 List<FileContainer> files,
                                 boolean deleteTempFile)
                          throws Exception
        Uploads a set of file headers and binaries to Cognite Data Fusion. The file binary can either be placed in-memory in the file container (as a byte string) or referenced (by URI) to a blob store. In case you reference the file by URI, you can choose to automatically remove the file binary from the (URI referenced) blob store after a successful upload to Cognite Data Fusion. This can Be useful in situations where you perform large scala data transfers utilizing a temp backing store.

        Example:

         
      Path fileAOriginal = Paths.get("path + file");
      List<FileContainer> files = new ArrayList<>();
      FileMetadata fileMetadata = FileMetadata.newBuilder()
           .setExternalId("10")
           .setName("test_file_.test")
           .setSource("sdk-data-generator")
           .putMetadata("type", "sdk-data-generator")
          .build();

      FileContainer fileContainer = FileContainer.newBuilder()
           .setFileMetadata(fileMetadata)
           .setFileBinary(FileBinary.newBuilder()
                .setBinaryUri(fileAOriginal.toUri().toString()))
           .build();
      files.add(fileContainer);

      List<FileMetadata> uploadFileResult =
                  client.files().upload(files, true);
        API Reference - Upload file
        Parameters:
        files - The files to upload.
        deleteTempFile - Set to true to remove the URI binary after upload. Set to false to keep the URI binary.
        Returns:
        The file metadata / headers for the uploaded files.
        Throws:
        Exception
        See Also:
        CogniteClient, CogniteClient.files()

      • download

        public List<FileContainer> download​(List<Item> files,
                                    URI downloadUri,
                                    boolean preferByteStream)
                             throws Exception
        Downloads file binaries. Downloads a set of file binaries based on externalId / id in the Item list. The file binaries can be downloaded as files or byte streams. In case the file is very large (> 200MB) it has to be streamed directly to the file system (i.e. downloaded as a file). Both the file header / metadata and the file binary will be returned. The complete information is encapsulated int the FileContainer returned from this method. The FileContainer will host the file binary stream if you set preferByteStream to true and the file size is < 200 MB. If preferByteStream is set to false or the file size is > 200MB the file binary will be stored on disk and the FileContainer will return the URI reference to the binary. Supported destination file stores for the file binary: - Local (network) disk. Specify the temp path as file://<host>/<my-path>/. Examples: file://localhost/home/files/, file:///home/files/, file:///c:/temp/ - Google Cloud Storage. Specify the temp path as gs://<my-storage-bucket>/<my-path>/. - S3 object storage. Specify the temp path as s3://<my-storage-bucket>/<my-path>/. If the file binary download URI points to the local file system we will try to name the file binary based on the file metadata/header fileName attribute. For remote storage we will use a randomly generated filename.

        Example:

         
      List<Item> files = List.of(Item.newBuilder().setId(10).build());
      List<FileContainer> downloadFilesResults =
                  client.files().download(files, Paths.get("").toUri(), false);
        API Reference - Download files
        Parameters:
        files - The list of files to download.
        downloadUri - The URI to the download storage
        preferByteStream - Set to true to return byte streams when possible, set to false to always store binary as file.
        Returns:
        File containers with file headers and references/byte streams of the binary.
        Throws:
        Exception
        See Also:
        downloadFileBinaries(List,URI,boolean), CogniteClient, CogniteClient.files()

      • downloadToPath

        public List<FileContainer> downloadToPath​(List<Item> files,
                                          Path downloadPath)
                                   throws Exception
        Downloads file binaries to a local / network path. Downloads a set of file binaries based on externalId / id in the Item list. Both the file header / metadata and the file binary will be returned. The complete information is encapsulated int the FileContainer returned from this method. The FileContainer will host the URI reference to the binary.

        Example:

         
      List<Item> files = List.of(Item.newBuilder().setId(10).build());
      List<FileContainer> downloadFilesResults =
                  client.files().downloadToPath(files, Paths.get(""));
        API Reference - Download files
        Parameters:
        files - The list of files to download.
        downloadPath - The Path to the download storage.
        Returns:
        File containers with file headers and references/byte streams of the binary.
        Throws:
        Exception
        See Also:
        download(List,URI,boolean), CogniteClient, CogniteClient.files()

      • downloadFileBinaries

        public List<FileBinary> downloadFileBinaries​(List<Item> fileItems,
                                             @Nullable
                                             URI tempStoragePath,
                                             boolean forceTempStorage)
                                      throws Exception
        Downloads file binaries. This method is intended for advanced use cases, for example when using this SDK as a part of a distributed system. Downloads a set of file binaries based on externalId / id in the Item list. The file binaries can be downloaded as files or byte streams. In case the file is very large (> 200MB) it has to be streamed directly to the file system (to the temp storage area). Supported temp storage for the file binary: - Local (network) disk. Specify the temp path as file://<host>/<my-path>/. Examples: file://localhost/home/files/, file:///home/files/, file:///c:/temp/ - Google Cloud Storage. Specify the temp path as gs://<my-storage-bucket>/<my-path>/. - S3 object storage. Specify the temp path as s3://<my-storage-bucket>/<my-path>/.

        Example:

         
      List<Item> fileItems = List.of(Item.newBuilder().setId(10).build());
      List<FileBinary> fileBinaries =
                     client.files()
                           .downloadFileBinaries(fileItems, URI.create("s3://testbucket"), true);
        API Reference - Download files
        Parameters:
        fileItems - The list of files to download.
        tempStoragePath - The URI to the download storage. Set to null to only perform in-memory download.
        forceTempStorage - Set to true to always download the binary to temp storage
        Returns:
        The file binary.
        Throws:
        Exception
        See Also:
        CogniteClient, CogniteClient.files()

      • delete

        public List<Item> delete​(List<Item> files)
                  throws Exception
        Deletes a set of files. The files to delete are identified via their externalId / id by submitting a list of Item.

        Example:

         
     List<Item> files = List.of(Item.newBuilder().setExternalId("1").build());
     List<Item> deletedItemsResults = client.files().delete(files);
        API Reference - Delete files
        Parameters:
        files - a list of Item representing the events (externalId / id) to be deleted
        Returns:
        The deleted events via Item
        Throws:
        Exception
        See Also:
        ApiBase.DeleteItems.deleteItems(List), CogniteClient, CogniteClient.files()

      • buildPartitionsList

        protected List<String> buildPartitionsList​(int noPartitions)
        Builds an array of partition specifications for parallel retrieval from the Cognite api. This specification is used as a parameter together with the filter / list endpoints. The number of partitions indicate the number of parallel read streams. Employ one partition specification per read stream.

        Example:

         
      List<String> partitions = buildPartitionsList(getClient().getClientConfig().getNoListPartitions());
        Parameters:
        noPartitions - The total number of partitions
        Returns:
        a List of partition specifications

      • listJson

        protected Iterator<List<String>> listJson​(ResourceType resourceType,
                                          Request requestParameters,
                                          String... partitions)
                                   throws Exception
        Will return the results from a list / filter api endpoint. For example, the filter assets endpoint. The results are paged through / iterated over via an Iterator--the entire results set is not buffered in memory, but streamed in "pages" from the Cognite api. If you need to buffer the entire results set, then you have to stream these results into your own data structure. This method support parallel retrieval via a set of partition specifications. The specified partitions will be collected and merged together before being returned via the Iterator.

        Example:

         
      Iterator<List<String>> result = listJson(resourceType, requestParameters, partitions);
        Parameters:
        resourceType - The resource type to query / filter / list. Ex. event, asset, time series.
        requestParameters - The query / filter specification. Follows the Cognite api request parameters.
        partitions - An optional set of partitions to read via.
        Returns:
        an Iterator over the results set.
        Throws:
        Exception
        See Also:
        listJson(ResourceType,Request,String,String...)

      • listJson

        protected Iterator<List<String>> listJson​(ResourceType resourceType,
                                          Request requestParameters,
                                          String partitionKey,
                                          String... partitions)
                                   throws Exception
        Will return the results from a list / filter api endpoint. For example, the filter assets endpoint. The results are paged through / iterated over via an Iterator--the entire results set is not buffered in memory, but streamed in "pages" from the Cognite api. If you need to buffer the entire results set, then you have to stream these results into your own data structure. This method support parallel retrieval via a set of partition specifications. The specified partitions will be collected and merged together before being returned via the Iterator.

        Example:

         
      Iterator<List<String>> result = listJson(resourceType, requestParameters, partitionKey, partitions);
        Parameters:
        resourceType - The resource type to query / filter / list. Ex. event, asset, time series.
        requestParameters - The query / filter specification. Follows the Cognite api request parameters.
        partitionKey - The key to use for the partitions in the read request. For example partition or cursor.
        partitions - An optional set of partitions to read via.
        Returns:
        an Iterator over the results set.
        Throws:
        Exception

      • retrieveJson

        protected List<String> retrieveJson​(ResourceType resourceType,
                                    Collection<Item> items)
                             throws Exception
        Retrieve items by id. Will ignore unknown ids by default.

        Example:

         
      Collection<Item> items = //Collection of items with ids;
      List<String> result = retrieveJson(resourceType, items);
        Parameters:
        resourceType - The item resource type (Event, Asset, etc.) to retrieve.
        items - The item(s) externalId / id to retrieve.
        Returns:
        The items in Json representation.
        Throws:
        Exception
        See Also:
        retrieveJson(ResourceType,Collection,Map)

      • retrieveJson

        protected List<String> retrieveJson​(ResourceType resourceType,
                                    Collection<Item> items,
                                    Map<String,​Object> parameters)
                             throws Exception
        Retrieve items by id. This version allows you to explicitly set additional parameters for the retrieve request. For example: <"ignoreUnknownIds", true> and <"fetchResources", true>.

        Example:

         
      Collection<Item> items = //Collection of items with ids;
      Map<String, Object> parameters = //Parameters;
      List<String> result = retrieveJson(resourceType, items, parameters);
        Parameters:
        resourceType - The item resource type (Event, Asset, etc.) to retrieve.
        items - The item(s) externalId / id to retrieve.
        parameters - Additional parameters for the request. For example <"ignoreUnknownIds", true>
        Returns:
        The items in Json representation.
        Throws:
        Exception

      • aggregate

        protected Aggregate aggregate​(ResourceType resourceType,
                              Request requestParameters)
                       throws Exception
        Performs an item aggregation request to Cognite Data Fusion. The default aggregation is a total item count based on the (optional) filters in the request. Some resource types, for example Event, supports multiple types of aggregation.

        Example:

         
      Aggregate aggregateResult = aggregate(resourceType,requestParameters);
        Parameters:
        resourceType - The resource type to perform aggregation of.
        requestParameters - The request containing filters.
        Returns:
        The aggregation result.
        Throws:
        Exception
        See Also:
        Cognite API v1 specification

      • addAuthInfo

        protected Request addAuthInfo​(Request request)
                       throws Exception
        Adds the required authentication information into the request object. If the request object already have complete auth info nothing will be added. The following authentication schemes are supported: 1) API key. When using an api key, this service will look up the corresponding project/tenant to issue requests to.

        Example:

         
      Request requestParams = addAuthInfo(request);
        Parameters:
        request - The request to enrich with auth information.
        Returns:
        The request parameters with auth info added to it.
        Throws:
        Exception

      • parseItems

        protected List<Item> parseItems​(List<String> input)
                         throws Exception
        Parses a list of item object in json representation to typed objects.

        Example:

         
      List<String> input = //List of json;
      List<Item> resultList = parseItems(input);
        Parameters:
        input - the item list in Json string representation
        Returns:
        the parsed item objects
        Throws:
        Exception

      • toRequestItems

        protected List<Map<String,​Object>> toRequestItems​(Collection<Item> itemList)
        Converts a list of Item to a request object structure (that can later be parsed to Json).

        Example:

         
      Collection<Item> itemList = //Collection of items;
      List<Map<String, Object>> result = toRequestItems(itemList);
        Parameters:
        itemList - The items to parse.
        Returns:
        The items in request item object form.

      • deDuplicate

        protected List<Item> deDuplicate​(Collection<Item> itemList)
        De-duplicates a collection of Item.

        Example:

         
      Collection<Item> itemList = //Collection of items;
      List<Item> result = deDuplicate(itemList);
        Parameters:
        itemList -
        Returns:

      • itemsHaveId

        protected boolean itemsHaveId​(Collection<Item> items)
        Returns true if all items contain either an externalId or id.

        Example:

         
      Collection<Item> items = //Collection of items;
      boolean result = itemsHaveId(items);
        Parameters:
        items -
        Returns:

      • mapItemToId

        protected Map<String,​Item> mapItemToId​(Collection<Item> items)
        Maps all items to their externalId (primary) or id (secondary). If the id function does not return any identity, the item will be mapped to the empty string. Via the identity mapping, this function will also perform deduplication of the input items.

        Example:

         
      Collection<Item> items = //Collection of items;
      Map<String, Item> result = mapItemToId(items);
        Parameters:
        items - the items to map to externalId / id.
        Returns:
        the Map with all items mapped to externalId / id.

      • parseString

        protected String parseString​(String itemJson,
                             String fieldName)
        Try parsing the specified Json path as a String.

        Example:

         
      String json = //String of json object
      String result = parseString(json, "name");
        Parameters:
        itemJson - The Json string
        fieldName - The Json path to parse
        Returns:
        The Json path as a String.

      • parseName

        protected String parseName​(String json)
        Returns the name attribute value from a json input.

        Example:

         
      String json = //String of json object
      String result = parseName(json);
        Parameters:
        json - the json to parse
        Returns:
        The name value