com.google.cloud.dataflow.sdk.transforms

Class DoFnTester<I,O>

java.lang.Object

com.google.cloud.dataflow.sdk.transforms.DoFnTester<I,O>

Type Parameters:

I - the type of the DoFn's (main) input elements

O - the type of the DoFn's (main) output elements

public class DoFnTester<I,O>
extends java.lang.Object

A harness for unit-testing a DoFn.

For example:

 DoFn<Input, Output> fn = ...;

 DoFnTester<Input, Output> fnTester = DoFnTester.of(fn);

 // Set arguments shared across all batches:
 fnTester.setSideInputs(...);      // If fn takes side inputs.
 fnTester.setSideOutputTags(...);  // If fn writes to side outputs.

 // Process a batch containing a single input element:
 Input testInput = ...;
 List<Output> testOutputs = fnTester.processBatch(testInput);
 Assert.assertThat(testOutputs,
                   JUnitMatchers.hasItems(...));

 // Process a bigger batch:
 Assert.assertThat(fnTester.processBatch(i1, i2, ...),
                   JUnitMatchers.hasItems(...));
  

Method Summary

Methods

Modifier and Type	Method and Description
void	clearOutputElements() Clears the record of the elements output so far to the main output.
<T> void	clearSideOutputElements(TupleTag<T> tag) Clears the record of the elements output so far to the side output with the given tag.
void	finishBundle() Calls DoFn.finishBundle(com.google.cloud.dataflow.sdk.transforms.DoFn<I, O>.Context) of the DoFn under test.
static <I,O> DoFnTester<I,O>	of(DoFn<I,O> fn) Returns a DoFnTester supporting unit-testing of the given DoFn.
java.util.List<O>	peekOutputElements() Returns the elements output so far to the main output.
<T> java.util.List<T>	peekSideOutputElements(TupleTag<T> tag) Returns the elements output so far to the side output with the given tag.
java.util.List<O>	processBatch(I... inputElements) A convenience operation that first calls startBundle(), then calls processElement(I) on each of the arguments, then calls finishBundle(), then returns the result of takeOutputElements().
void	processElement(I element) Calls DoFn.processElement(com.google.cloud.dataflow.sdk.transforms.DoFn<I, O>.ProcessContext) on the DoFn under test, in a context where DoFn.ProcessContext.element() returns the given element.
void	setSideInput(PCollectionView<?> sideInput, java.lang.Iterable<com.google.cloud.dataflow.sdk.util.WindowedValue<?>> value) Registers the values of a side input PCollectionView to pass to the DoFn under test.
void	setSideInputInGlobalWindow(PCollectionView<?> sideInput, java.lang.Iterable<?> value) Registers the values for a side input PCollectionView to pass to the DoFn under test.
void	setSideInputs(java.util.Map<PCollectionView<?>,java.lang.Iterable<com.google.cloud.dataflow.sdk.util.WindowedValue<?>>> sideInputs) Registers the tuple of values of the side input PCollectionViews to pass to the DoFn under test.
void	setSideOutputTags(TupleTagList sideOutputTags) Registers the list of TupleTags that can be used by the DoFn under test to output to side output PCollections.
void	startBundle() Calls DoFn.startBundle(com.google.cloud.dataflow.sdk.transforms.DoFn<I, O>.Context) on the DoFn under test.
java.util.List<O>	takeOutputElements() Returns the elements output so far to the main output.
<T> java.util.List<T>	takeSideOutputElements(TupleTag<T> tag) Returns the elements output so far to the side output with the given tag.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

of

public static <I,O> DoFnTester<I,O> of(DoFn<I,O> fn)

Returns a DoFnTester supporting unit-testing of the given DoFn.

setSideInputs

public void setSideInputs(java.util.Map<PCollectionView<?>,java.lang.Iterable<com.google.cloud.dataflow.sdk.util.WindowedValue<?>>> sideInputs)

Registers the tuple of values of the side input PCollectionViews to pass to the DoFn under test.

If needed, first creates a fresh instance of the DoFn under test.

If this isn't called, DoFnTester assumes the DoFn takes no side inputs.

setSideInput

public void setSideInput(PCollectionView<?> sideInput,
                java.lang.Iterable<com.google.cloud.dataflow.sdk.util.WindowedValue<?>> value)

Registers the values of a side input PCollectionView to pass to the DoFn under test.

If needed, first creates a fresh instance of the DoFn under test.

If this isn't called, DoFnTester assumes the DoFn takes no side inputs.

setSideInputInGlobalWindow

public void setSideInputInGlobalWindow(PCollectionView<?> sideInput,
                              java.lang.Iterable<?> value)

Registers the values for a side input PCollectionView to pass to the DoFn under test. All values are placed in the global window.

setSideOutputTags

public void setSideOutputTags(TupleTagList sideOutputTags)

Registers the list of TupleTags that can be used by the DoFn under test to output to side output PCollections.

If needed, first creates a fresh instance of the DoFn under test.

If this isn't called, DoFnTester assumes the DoFn doesn't emit to any side outputs.

processBatch

public java.util.List<O> processBatch(I... inputElements)

A convenience operation that first calls startBundle(), then calls processElement(I) on each of the arguments, then calls finishBundle(), then returns the result of takeOutputElements().

startBundle

public void startBundle()

Calls DoFn.startBundle(com.google.cloud.dataflow.sdk.transforms.DoFn<I, O>.Context) on the DoFn under test.

If needed, first creates a fresh instance of the DoFn under test.

processElement

public void processElement(I element)

Calls DoFn.processElement(com.google.cloud.dataflow.sdk.transforms.DoFn<I, O>.ProcessContext) on the DoFn under test, in a context where DoFn.ProcessContext.element() returns the given element.

Will call startBundle() automatically, if it hasn't already been called.

Throws:

java.lang.IllegalStateException - if the DoFn under test has already been finished

finishBundle

public void finishBundle()

Calls DoFn.finishBundle(com.google.cloud.dataflow.sdk.transforms.DoFn<I, O>.Context) of the DoFn under test.

Will call startBundle() automatically, if it hasn't already been called.

Throws:

java.lang.IllegalStateException - if the DoFn under test has already been finished

peekOutputElements

public java.util.List<O> peekOutputElements()

Returns the elements output so far to the main output. Does not clear them, so subsequent calls will continue to include these elements.

See Also:

takeOutputElements(), TODO: provide accessors that take and return {@code WindowedValue}s in order to test timestamp- and window-sensitive DoFns.

clearOutputElements

public void clearOutputElements()

Clears the record of the elements output so far to the main output.

See Also:

peekOutputElements()

takeOutputElements

public java.util.List<O> takeOutputElements()

Returns the elements output so far to the main output. Clears the list so these elements don't appear in future calls.

See Also:

peekOutputElements()

peekSideOutputElements

public <T> java.util.List<T> peekSideOutputElements(TupleTag<T> tag)

Returns the elements output so far to the side output with the given tag. Does not clear them, so subsequent calls will continue to include these elements.

See Also:

takeSideOutputElements(com.google.cloud.dataflow.sdk.values.TupleTag<T>), clearSideOutputElements(com.google.cloud.dataflow.sdk.values.TupleTag<T>)

clearSideOutputElements

public <T> void clearSideOutputElements(TupleTag<T> tag)

Clears the record of the elements output so far to the side output with the given tag.

See Also:

peekSideOutputElements(com.google.cloud.dataflow.sdk.values.TupleTag<T>)

takeSideOutputElements

public <T> java.util.List<T> takeSideOutputElements(TupleTag<T> tag)

Returns the elements output so far to the side output with the given tag. Clears the list so these elements don't appear in future calls.

See Also:

peekSideOutputElements(com.google.cloud.dataflow.sdk.values.TupleTag<T>)