com.google.cloud.dataflow.examples.common

Class PubsubFileInjector

java.lang.Object

com.google.cloud.dataflow.examples.common.PubsubFileInjector

public class PubsubFileInjector
extends Object

A batch Dataflow pipeline for injecting a set of GCS files into a PubSub topic line by line. Empty lines are skipped.

This is useful for testing streaming pipelines. Note that since batch pipelines might retry chunks, this does _not_ guarantee exactly-once injection of file data. Some lines may be published multiple times.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	PubsubFileInjector.Bound A DoFn that publishes non-empty lines to Google Cloud PubSub.
static class	PubsubFileInjector.Unbound An incomplete PubsubFileInjector transform with unbound output topic.

Constructor Summary

Constructors

Constructor and Description
PubsubFileInjector()

Method Summary

Modifier and Type	Method and Description
static void	main(String[] args) Sets up and starts streaming pipeline.
static PubsubFileInjector.Bound	publish(String outputTopic) Creates a PubsubFileInjector transform that publishes to the given output topic.
static PubsubFileInjector.Unbound	withTimestampLabelKey(String timestampLabelKey) Creates a PubsubFileInjector transform with the given timestamp label key.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PubsubFileInjector

public PubsubFileInjector()

Method Detail

withTimestampLabelKey

public static PubsubFileInjector.Unbound withTimestampLabelKey(String timestampLabelKey)

Creates a PubsubFileInjector transform with the given timestamp label key.

publish

public static PubsubFileInjector.Bound publish(String outputTopic)

Creates a PubsubFileInjector transform that publishes to the given output topic.

main

public static void main(String[] args)

Sets up and starts streaming pipeline.