com.google.cloud.dataflow.sdk.transforms

## Class RemoveDuplicates<T>

- java.lang.Object
- - com.google.cloud.dataflow.sdk.transforms.PTransform<PCollection<T>,PCollection<T>>
  - - com.google.cloud.dataflow.sdk.transforms.RemoveDuplicates<T>

Type Parameters:
  `T` - the type of the elements of the input and output `PCollection`s

All Implemented Interfaces:
  HasDisplayData, Serializable

    public class RemoveDuplicates<T>
    extends PTransform<PCollection<T>,PCollection<T>>

`RemoveDuplicates<T>` takes a `PCollection<T>` and returns a `PCollection<T>` that has all the elements of the input but with duplicate elements removed such that each element is unique within each window.

Two values of type `T` are compared for equality not by regular Java `Object.equals(java.lang.Object)`, but instead by first encoding each of the elements using the `PCollection`'s `Coder`, and then comparing the encoded bytes. This admits efficient parallel evaluation.

Optionally, a function may be provided that maps each element to a representative value. In this case, two elements will be considered duplicates if they have equal representative values, with equality being determined as above.

By default, the `Coder` of the output `PCollection` is the same as the `Coder` of the input `PCollection`.

Each output element is in the same window as its corresponding input element, and has the timestamp of the end of that window. The output `PCollection` has the same `WindowFn` as the input.

Does not preserve any order the input PCollection might have had.

Example of use:

     PCollection<String> words = ...;
     PCollection<String> uniqueWords =
         words.apply(RemoveDuplicates.<String>create());

See Also:
  Serialized Form

### Nested Class Summary

Nested Classes
`static class` `RemoveDuplicates.WithRepresentativeValues<T,IdT>` A `RemoveDuplicates` `PTransform` that uses a `SerializableFunction` to obtain a representative value for each input element.

### Field Summary

### Fields inherited from class com.google.cloud.dataflow.sdk.transforms.PTransform

`name`

### Constructor Summary

Constructors
Constructor and Description
`RemoveDuplicates()`

### Method Summary

All Methods Static Methods Instance Methods Concrete Methods
`PCollection<T>` `apply(PCollection<T> in)` Applies this `PTransform` on the given `InputT`, and returns its `Output`.
`static <T> RemoveDuplicates<T>` `create()` Returns a `RemoveDuplicates<T>` `PTransform`.
`static <T,IdT> RemoveDuplicates.WithRepresentativeValues<T,IdT>` `withRepresentativeValueFn(SerializableFunction<T,IdT> fn)` Returns a `RemoveDuplicates<T, IdT>` `PTransform`.

### Methods inherited from class com.google.cloud.dataflow.sdk.transforms.PTransform

`getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, populateDisplayData, toString, validate`

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

### Constructor Detail

#### RemoveDuplicates

    public RemoveDuplicates()

### Method Detail

#### create

    public static <T> RemoveDuplicates<T> create()

Returns a `RemoveDuplicates<T>` `PTransform`.

Type Parameters:
  `T` - the type of the elements of the input and output `PCollection`s

#### withRepresentativeValueFn

    public static <T,IdT> RemoveDuplicates.WithRepresentativeValues<T,IdT> withRepresentativeValueFn(SerializableFunction<T,IdT> fn)

Returns a `RemoveDuplicates<T, IdT>` `PTransform`.

Type Parameters:
  `T` - the type of the elements of the input and output `PCollection`s
  `IdT` - the type of the representative value used to dedup

#### apply

    public PCollection<T> apply(PCollection<T> in)

Description copied from class: `PTransform`
Applies this `PTransform` on the given `InputT`, and returns its `Output`.

Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

The default implementation throws an exception. A derived class must either implement apply, or else each runner must supply a custom implementation via `PipelineRunner.apply(com.google.cloud.dataflow.sdk.transforms.PTransform<InputT, OutputT>, InputT)`.

Overrides:
  `apply` in class `PTransform<PCollection<T>,PCollection<T>>`