Merge pull request #568 from davorbonaci/merge_v2_into_master

Reverse-integrate: merge `v2` into `master`

Author: davorbonaci

.gitattributes

@@ -1,3 +1,17 @@
+# Copyright (C) 2017 Google Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may not
+# use this file except in compliance with the License. You may obtain a copy of
+# the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations under
+# the License.
+
 # The default behavior, which overrides 'core.autocrlf', is to use Git's
 # built-in heuristics to determine whether a particular file is text or binary.
 # Text files are automatically normalized to the user's platforms.

.gitignore

@@ -1,3 +1,17 @@
+# Copyright (C) 2017 Google Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may not
+# use this file except in compliance with the License. You may obtain a copy of
+# the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations under
+# the License.
+
 target/
 
 # Ignore IntelliJ files.

.travis.yml

@@ -1,3 +1,17 @@
+# Copyright (C) 2017 Google Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may not
+# use this file except in compliance with the License. You may obtain a copy of
+# the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations under
+# the License.
+
 language: java
 
 sudo: false
@@ -15,24 +29,26 @@ matrix:
   include:
     # On OSX, run with default JDK only.
     - os: osx
-      env: MAVEN_OVERRIDE=""
     # On Linux, run with specific JDKs only.
     - os: linux
-      env: CUSTOM_JDK="oraclejdk8" MAVEN_OVERRIDE="-Ddataflow.surefire_argline='-Xmx512m'"
-    - os: linux
-      env: CUSTOM_JDK="oraclejdk7" MAVEN_OVERRIDE="-Ddataflow.surefire_argline='-Xmx512m'"
-    - os: linux
-      env: CUSTOM_JDK="openjdk7" MAVEN_OVERRIDE="-Ddataflow.surefire_argline='-Xmx512m'"
+      env: CUSTOM_JDK="oraclejdk8"
+    # The distribution does not build with Java 7 by design. We need to rewrite these tests
+    # to, for example, build and install with Java 8 and then test examples with Java 7.
+    # - os: linux
+    #   env: CUSTOM_JDK="oraclejdk7"
+    # - os: linux
+    #  env: CUSTOM_JDK="openjdk7"
 
 before_install:
   - if [ "$TRAVIS_OS_NAME" == "osx" ]; then export JAVA_HOME=$(/usr/libexec/java_home); fi
   - if [ "$TRAVIS_OS_NAME" == "linux" ]; then jdk_switcher use "$CUSTOM_JDK"; fi
 
 install:
   - travis_retry mvn install clean -U -DskipTests=true
-  - travis_retry mvn -f contrib/kafka/pom.xml install clean -U -DskipTests=true
 
 script:
-  - travis_retry mvn $MAVEN_OVERRIDE install -U
-  - travis_retry mvn -f contrib/kafka/pom.xml $MAVEN_OVERRIDE install -U
-  - travis_retry travis/test_wordcount.sh
+  # Verify that the project can be built and installed.
+  - mvn install
+  # Verify that starter and examples archetypes have the correct version of the NOTICE file.
+  - diff -q NOTICE maven-archetypes/starter/src/main/resources/NOTICE
+  - diff -q NOTICE maven-archetypes/examples/src/main/resources/NOTICE

CONTRIBUTING.md

@@ -1,6 +1,31 @@
+<!--
+  Copyright (C) 2017 Google Inc.
+
+  Licensed under the Apache License, Version 2.0 (the "License"); you may not
+  use this file except in compliance with the License. You may obtain a copy of
+  the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+  WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+  License for the specific language governing permissions and limitations under
+  the License.
+-->
+
 Want to contribute? Great! First, read this page (including the small print at
 the end).
 
+Google Cloud Dataflow SDK is a distribution of Apache Beam. If you'd like to
+change anything under the `org.apache.beam.*` namespace, please submit that
+change directly to the [Apache Beam](https://github.com/apache/beam) project.
+
+This repository contains code to build the Dataflow distribution of Beam, and
+some Dataflow-specific code. Only changes to how the distribution is built, or
+the Dataflow-specific code under the `com.google.cloud.dataflow.*` namespace,
+can be merged here.
+
 ### Before you contribute
 Before we can use your code, you must sign the
 [Google Individual Contributor License Agreement](https://developers.google.com/open-source/cla/individual?csw=1)
@@ -21,11 +46,6 @@ frustration later on.
 All submissions, including submissions by project members, require review. We
 use GitHub pull requests for this purpose.
 
-### Organization
-During our review and triage of incoming pull requests, we'll advise whether to
-include your contribution into the mainline SDK, or to maintain it within the
-separate group of [community-contributed modules](https://github.com/GoogleCloudPlatform/DataflowJavaSDK/tree/master/contrib).
-
 ### The small print
 Contributions made by corporations are covered by a different agreement than
 the one above, the Software Grant and Corporate Contributor License Agreement.

NOTICE

@@ -0,0 +1,5 @@
+Google Cloud Dataflow SDK for Java
+Copyright 2017, Google Inc.
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).

README.md

@@ -1,10 +1,30 @@
+<!--
+  Copyright (C) 2017 Google Inc.
+
+  Licensed under the Apache License, Version 2.0 (the "License"); you may not
+  use this file except in compliance with the License. You may obtain a copy of
+  the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+  WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+  License for the specific language governing permissions and limitations under
+  the License.
+-->
+
 # Google Cloud Dataflow SDK for Java
 
 [Google Cloud Dataflow](https://cloud.google.com/dataflow/) provides a simple,
 powerful programming model for building both batch and streaming parallel data
-processing pipelines. This repository hosts the open-sourced Cloud Dataflow SDK
-for Java, which can be used to run pipelines against the Google Cloud Dataflow
-Service.
+processing pipelines.
+
+Dataflow SDK for Java is a distribution of a portion of the
+[Apache Beam](https://beam.apache.org) project. This repository hosts the
+code to build this distribution and any Dataflow-specific code/modules. The
+underlying source code is hosted in the
+[Apache Beam repository](https://github.com/apache/beam).
 
 [General usage](https://cloud.google.com/dataflow/getting-started) of Google
 Cloud Dataflow does **not** require use of this repository. Instead:
@@ -25,13 +45,19 @@ environments like Eclipse or Apache Maven:
 [DataflowJavaSDK-examples](https://github.com/GoogleCloudPlatform/DataflowJavaSDK-examples)
 repository.
 
-However, if you'd like to contribute to the SDK, write your own PipelineRunner,
-or just dig in for the fun of it, please stay with us here!
+<!-- 1. If you are using [Eclipse](https://eclipse.org/) integrated development
+environment (IDE), the
+[Cloud Dataflow Plugin for Eclipse](https://cloud.google.com/dataflow/getting-started-eclipse)
+provides tools to create and execute Dataflow pipelines locally and on the
+Dataflow Service. -->
 
-## Status [![Build Status](https://travis-ci.org/GoogleCloudPlatform/DataflowJavaSDK.svg?branch=master)](https://travis-ci.org/GoogleCloudPlatform/DataflowJavaSDK)
+## Status [![Build Status](https://travis-ci.org/GoogleCloudPlatform/DataflowJavaSDK.svg?branch=v2)](https://travis-ci.org/GoogleCloudPlatform/DataflowJavaSDK)
 
-Both the SDK and the Dataflow Service are generally available, open to all
-developers, and considered stable and fully qualified for production use.
+This branch is a work-in-progress for the Dataflow SDK for Java, version 2.0.0.
+It is currently supported on the Cloud Dataflow service in Beta.
+
+<!--Both the SDK and the Dataflow Service are generally available, open to all
+developers, and considered stable and fully qualified for production use.-->
 
 ## Overview
 
@@ -48,25 +74,17 @@ for execution.
 * [`PipelineRunner`](https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/PipelineRunner.java):
 specifies where and how the pipeline should execute.
 
-We provide three PipelineRunners:
+We provide two runners:
 
-  1. The [`DirectPipelineRunner`](https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/DirectPipelineRunner.java)
+  1. The [`DirectRunner`](https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/DirectPipelineRunner.java)
 runs the pipeline on your local machine.
-  2. The [`DataflowPipelineRunner`](https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/DataflowPipelineRunner.java)
+  1. The [`DataflowRunner`](https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/DataflowPipelineRunner.java)
 submits the pipeline to the Dataflow Service, where it runs using managed
 resources in the [Google Cloud Platform](https://cloud.google.com) (GCP).
-  3. The [`BlockingDataflowPipelineRunner`](https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/BlockingDataflowPipelineRunner.java)
-submits the pipeline to the Dataflow Service via the `DataflowPipelineRunner`
-and then prints messages about the job status until the execution is complete.
 
 The SDK is built to be extensible and support additional execution environments
-beyond local execution and the Google Cloud Dataflow Service. In partnership
-with [Cloudera](https://www.cloudera.com/), you can run Dataflow pipelines on
-an [Apache Spark](https://spark.apache.org/) backend using the
-[`SparkPipelineRunner`](https://github.com/cloudera/spark-dataflow).
-Additionally, you can run Dataflow pipelines on an
-[Apache Flink](https://flink.apache.org/) backend using the
-[`FlinkPipelineRunner`](https://github.com/dataArtisans/flink-dataflow).
+beyond local execution and the Google Cloud Dataflow Service. Apache Beam
+contains additional SDKs, runners, IO connectors, etc.
 
 ## Getting Started
 
@@ -77,35 +95,12 @@ module provides a set of basic Java APIs to program against.
 * The [`examples`](https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/examples)
 module provides a few samples to get started. We recommend starting with the
 `WordCount` example.
-* The [`contrib`](https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/contrib)
-directory hosts community-contributed Dataflow modules.
 
 The following command will build both the `sdk` and `example` modules and
 install them in your local Maven repository:
 
     mvn clean install
 
-You can speed up the build and install process by using the following options:
-
-  1. To skip execution of the unit tests, run:
-
-        mvn install -DskipTests
-
-  2. While iterating on a specific module, use the following command to compile
-  and reinstall it. For example, to reinstall the `examples` module, run:
-
-        mvn install -pl examples
-
-  Be careful, however, as this command will use the most recently installed SDK
-  from the local repository (or Maven Central) even if you have changed it
-  locally.
-
-If you are using [Eclipse](https://eclipse.org/) integrated development
-environment (IDE), the
-[Cloud Dataflow Plugin for Eclipse](https://cloud.google.com/dataflow/getting-started-eclipse)
-provides tools to create and execute Dataflow pipelines locally and on the
-Dataflow Service.
-
 After building and installing, you can execute the `WordCount` and other
 example pipelines by following the instructions in this
 [README](https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/examples/README.md).
@@ -121,5 +116,6 @@ on GitHub to report any bugs, comments or questions regarding SDK development.
 ## More Information
 
 * [Google Cloud Dataflow](https://cloud.google.com/dataflow/)
+* [Apache Beam](https://beam.apache.org/)
 * [Dataflow Concepts and Programming Model](https://cloud.google.com/dataflow/model/programming-model)
 * [Java API Reference](https://cloud.google.com/dataflow/java-sdk/JavaDoc/index)