com.google.cloud.dataflow.sdk.io

## Class DatastoreIO

- java.lang.Object
- - com.google.cloud.dataflow.sdk.io.DatastoreIO

Deprecated.
replaced by `DatastoreIO`

    @Deprecated
     @Experimental(value=SOURCE_SINK)
    public class DatastoreIO
    extends Object

`DatastoreIO` provides an API to Read and Write `PCollections` of Google Cloud Datastore `DatastoreV1.Entity` objects.

Google Cloud Datastore is a fully managed NoSQL data storage service. An `Entity` is an object in Datastore, analogous to a row in traditional database table.

This API currently requires an authentication workaround. To use `DatastoreIO`, users must use the `gcloud` command line tool to get credentials for Datastore:

     $ gcloud auth login

To read a `PCollection` from a query to Datastore, use `source()` and its methods `DatastoreIO.Source.withDataset(java.lang.String)` and `DatastoreIO.Source.withQuery(com.google.api.services.datastore.DatastoreV1.Query)` to specify the dataset to query and the query to read from. You can optionally provide a namespace to query within using `DatastoreIO.Source.withNamespace(java.lang.String)` or a Datastore host using `DatastoreIO.Source.withHost(java.lang.String)`.

For example:

     // Read a query from Datastore
     PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
     Query query = ...;
     String dataset = "...";

     Pipeline p = Pipeline.create(options);
     PCollection<Entity> entities = p.apply(
         Read.from(DatastoreIO.source()
             .withDataset(datasetId)
             .withQuery(query)
             .withHost(host)));

or:

     // Read a query from Datastore using the default namespace and host
     PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
     Query query = ...;
     String dataset = "...";

     Pipeline p = Pipeline.create(options);
     PCollection<Entity> entities = p.apply(DatastoreIO.readFrom(datasetId, query));
     p.run();

Note: Normally, a Cloud Dataflow job will read from Cloud Datastore in parallel across many workers. However, when the `DatastoreV1.Query` is configured with a limit using `DatastoreV1.Query.Builder.setLimit(int)`, then all returned results will be read by a single Dataflow worker in order to ensure correct data.

To write a `PCollection` to a Datastore, use `writeTo(java.lang.String)`, specifying the datastore to write to:

     PCollection<Entity> entities = ...;
     entities.apply(DatastoreIO.writeTo(dataset));
     p.run();

To optionally change the host that is used to write to the Datastore, use `sink()` to build a `DatastoreIO.Sink` and write to it using the `Write` transform:

     PCollection<Entity> entities = ...;
     entities.apply(Write.to(DatastoreIO.sink().withDataset(dataset).withHost(host)));

`Entities` in the `PCollection` to be written must have complete `Keys`. Complete `Keys` specify the `name` and `id` of the `Entity`, where incomplete `Keys` do not. A `namespace` other than the project default may be written to by specifying it in the `Entity` `Keys`.

     Key.Builder keyBuilder = DatastoreHelper.makeKey(...);
     keyBuilder.getPartitionIdBuilder().setNamespace(namespace);

`Entities` will be committed as upsert (update or insert) mutations. Please read Entities, Properties, and Keys for more information about `Entity` keys.

### Permissions

Permission requirements depend on the `PipelineRunner` that is used to execute the Dataflow job. Please refer to the documentation of corresponding `PipelineRunner`s for more details.

Please see Cloud Datastore Sign Up for security and permission related information specific to Datastore.

See Also:
  `PipelineRunner`

### Nested Class Summary

Nested Classes
`static class` `DatastoreIO.DatastoreReader` Deprecated. A `Source.Reader` over the records from a query of the datastore.
`static class` `DatastoreIO.Sink` Deprecated. A `Sink` that writes a `PCollection` containing `Entities` to a Datastore kind.
`static class` `DatastoreIO.Source` Deprecated. A `Source` that reads the result rows of a Datastore query as `Entity` objects.

### Field Summary

Fields
`static int` `DATASTORE_BATCH_UPDATE_LIMIT` Deprecated. Datastore has a limit of 500 mutations per batch operation, so we flush changes to Datastore every 500 entities.
`static String` `DEFAULT_HOST` Deprecated.

### Constructor Summary

Constructors
Constructor and Description
`DatastoreIO()` Deprecated.

### Method Summary

All Methods Static Methods Concrete Methods Deprecated Methods
`static DatastoreIO.Source` `read()` Deprecated. the name and return type do not match. Use `source()`.
`static Read.Bounded<DatastoreV1.Entity>` `readFrom(String datasetId, DatastoreV1.Query query)` Deprecated. Returns a `PTransform` that reads Datastore entities from the query against the given dataset.
`static Read.Bounded<DatastoreV1.Entity>` `readFrom(String host, String datasetId, DatastoreV1.Query query)` Deprecated. prefer `source()` with `DatastoreIO.Source.withHost(java.lang.String)`, `DatastoreIO.Source.withDataset(java.lang.String)`, `DatastoreIO.Source.withQuery(com.google.api.services.datastore.DatastoreV1.Query)`s.
`static DatastoreIO.Sink` `sink()` Deprecated. Returns a new `DatastoreIO.Sink` builder using the default host.
`static DatastoreIO.Source` `source()` Deprecated. Returns an empty `DatastoreIO.Source` builder with the default `host`.
`static Write.Bound<DatastoreV1.Entity>` `writeTo(String datasetId)` Deprecated. Returns a new `Write` transform that will write to a `Sink`.

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Field Detail

#### DEFAULT_HOST

    public static final String DEFAULT_HOST

Deprecated.

See Also:
  Constant Field Values

#### DATASTORE_BATCH_UPDATE_LIMIT

    public static final int DATASTORE_BATCH_UPDATE_LIMIT

Deprecated.
Datastore has a limit of 500 mutations per batch operation, so we flush changes to Datastore every 500 entities.

See Also:
  Constant Field Values

### Constructor Detail

#### DatastoreIO

    public DatastoreIO()

Deprecated.

### Method Detail

#### read

    @Deprecated
    public static DatastoreIO.Source read()

Deprecated. the name and return type do not match. Use `source()`.
Returns an empty `DatastoreIO.Source` builder with the default `host`. Configure the `dataset`, `query`, and `namespace` using `DatastoreIO.Source.withDataset(java.lang.String)`, `DatastoreIO.Source.withQuery(com.google.api.services.datastore.DatastoreV1.Query)`, and `DatastoreIO.Source.withNamespace(java.lang.String)`.

#### source

    public static DatastoreIO.Source source()

Deprecated.
Returns an empty `DatastoreIO.Source` builder with the default `host`. Configure the `dataset`, `query`, and `namespace` using `DatastoreIO.Source.withDataset(java.lang.String)`, `DatastoreIO.Source.withQuery(com.google.api.services.datastore.DatastoreV1.Query)`, and `DatastoreIO.Source.withNamespace(java.lang.String)`.

The resulting `DatastoreIO.Source` object can be passed to `Read` to create a `PTransform` that will read from Datastore.

#### readFrom

    public static Read.Bounded<DatastoreV1.Entity> readFrom(String datasetId,
                                                            DatastoreV1.Query query)

Deprecated.
Returns a `PTransform` that reads Datastore entities from the query against the given dataset.

#### readFrom

    @Deprecated
    public static Read.Bounded<DatastoreV1.Entity> readFrom(String host,
                                                                        String datasetId,
                                                                        DatastoreV1.Query query)

Deprecated. prefer `source()` with `DatastoreIO.Source.withHost(java.lang.String)`, `DatastoreIO.Source.withDataset(java.lang.String)`, `DatastoreIO.Source.withQuery(com.google.api.services.datastore.DatastoreV1.Query)`s.
Returns a `PTransform` that reads Datastore entities from the query against the given dataset and host.

#### sink

    public static DatastoreIO.Sink sink()

Deprecated.
Returns a new `DatastoreIO.Sink` builder using the default host. You need to further configure it using `DatastoreIO.Sink.withDataset(java.lang.String)`, and optionally `DatastoreIO.Sink.withHost(java.lang.String)` before using it in a `Write` transform.

For example: `p.apply(Write.to(DatastoreIO.sink().withDataset(dataset)));`

#### writeTo

    public static Write.Bound<DatastoreV1.Entity> writeTo(String datasetId)

Deprecated.
Returns a new `Write` transform that will write to a `Sink`.

For example: `p.apply(DatastoreIO.writeTo(dataset));`