Apache Hadoop Connectors

Libraries and tools for interoperability between Apache Hadoop related
open-source software and Google Cloud Platform.

Google Cloud Storage connector for Apache Hadoop (HCFS)

The Google Cloud Storage connector for Hadoop enables running MapReduce jobs
directly on data in Google Cloud Storage by implementing the Hadoop FileSystem
interface. For details, see the README.

Google BigQuery connector for Apache Hadoop MapReduce

The Google BigQuery connector for Hadoop MapReduce enables running MapReduce
jobs on data in BigQuery by implementing the InputFormat & OutputFormat
interfaces. For more details see
the documentation

Building the Cloud Storage and BigQuery connectors

Note that build requires Java 11+ and fails with older Java versions.

To build the connector for specific Hadoop version, run the following commands
from the main directory:

./mvnw clean package

In order to verify test coverage for specific Hadoop version, run the following
commands from the main directory:

./mvnw -P coverage clean verify

The Cloud Storage connector JAR can be found in gcs/target/. The BigQuery
connector JAR can be found in bigquery/target/.

Adding the Cloud Storage and BigQuery connectors to your build

Maven group ID is com.google.cloud.bigdataoss and artifact ID for Cloud
Storage connector is gcs-connector and for BigQuery connectors is
bigquery-connector.

To add a dependency on one of the connectors using Maven, use the following:

• Cloud Storage connector:

  <dependency>
    <groupId>com.google.cloud.bigdataoss</groupId>
    <artifactId>gcs-connector</artifactId>
    <version>hadoop2-2.2.2</version>
  </dependency>

• BigQuery connector:

  <dependency>
    <groupId>com.google.cloud.bigdataoss</groupId>
    <artifactId>bigquery-connector</artifactId>
    <version>hadoop2-1.2.0</version>
  </dependency>

Resources

On Stack Overflow, use the tag
google-cloud-dataproc
for questions about the connectors in this repository. This tag receives
responses from the Stack Overflow community and Google engineers, who monitor
the tag and offer unofficial support.