Skip to content

Commit 709387d

Browse files
s1ckDarthMaxdongjoon-hyun
committed
[SPARK-27300][GRAPH] Add Spark Graph modules and dependencies
## What changes were proposed in this pull request? This PR introduces the necessary Maven modules for the new [Spark Graph](https://issues.apache.org/jira/browse/SPARK-25994) feature for Spark 3.0. * `spark-graph` is a parent module that users depend on to get all graph functionalities (Cypher and Graph Algorithms) * `spark-graph-api` defines the [Property Graph API](https://docs.google.com/document/d/1Wxzghj0PvpOVu7XD1iA8uonRYhexwn18utdcTxtkxlI) that is being shared between Cypher and Algorithms * `spark-cypher` contains a Cypher query engine implementation Both, `spark-graph-api` and `spark-cypher` depend on Spark SQL. Note, that the Maven module for Graph Algorithms is not part of this PR and will be introduced in https://issues.apache.org/jira/browse/SPARK-27302 A PoC for a running Cypher implementation can be found in this WIP PR apache#24297 ## How was this patch tested? Pass the Jenkins with all profiles and manually build and check the followings. ``` $ ls assembly/target/scala-2.12/jars/spark-cypher* assembly/target/scala-2.12/jars/spark-cypher_2.12-3.0.0-SNAPSHOT.jar $ ls assembly/target/scala-2.12/jars/spark-graph* | grep -v graphx assembly/target/scala-2.12/jars/spark-graph-api_2.12-3.0.0-SNAPSHOT.jar assembly/target/scala-2.12/jars/spark-graph_2.12-3.0.0-SNAPSHOT.jar ``` Closes apache#24490 from s1ck/SPARK-27300. Lead-authored-by: Martin Junghanns <[email protected]> Co-authored-by: Max Kießling <[email protected]> Co-authored-by: Martin Junghanns <[email protected]> Co-authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 6dcf09b commit 709387d

File tree

9 files changed

+222
-2
lines changed

9 files changed

+222
-2
lines changed

‎LICENSE-binary

+1
Original file line numberDiff line numberDiff line change
@@ -369,6 +369,7 @@ org.eclipse.jetty:jetty-util
369369
org.eclipse.jetty:jetty-webapp
370370
org.eclipse.jetty:jetty-xml
371371
org.scala-lang.modules:scala-xml_2.12
372+
org.opencypher:okapi-shade
372373

373374
core/src/main/java/org/apache/spark/util/collection/TimSort.java
374375
core/src/main/resources/org/apache/spark/ui/static/bootstrap*

‎assembly/pom.xml

+5
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,11 @@
6464
<artifactId>spark-graphx_${scala.binary.version}</artifactId>
6565
<version>${project.version}</version>
6666
</dependency>
67+
<dependency>
68+
<groupId>org.apache.spark</groupId>
69+
<artifactId>spark-graph_${scala.binary.version}</artifactId>
70+
<version>${project.version}</version>
71+
</dependency>
6772
<dependency>
6873
<groupId>org.apache.spark</groupId>
6974
<artifactId>spark-sql_${scala.binary.version}</artifactId>

‎dev/deps/spark-deps-hadoop-2.7

+1
Original file line numberDiff line numberDiff line change
@@ -151,6 +151,7 @@ minlog-1.3.0.jar
151151
netty-3.9.9.Final.jar
152152
netty-all-4.1.30.Final.jar
153153
objenesis-2.5.1.jar
154+
okapi-shade-0.4.2.jar
154155
okhttp-3.8.1.jar
155156
okio-1.13.0.jar
156157
opencsv-2.3.jar

‎dev/deps/spark-deps-hadoop-3.2

+1
Original file line numberDiff line numberDiff line change
@@ -170,6 +170,7 @@ netty-3.9.9.Final.jar
170170
netty-all-4.1.30.Final.jar
171171
nimbus-jose-jwt-4.41.1.jar
172172
objenesis-2.5.1.jar
173+
okapi-shade-0.4.2.jar
173174
okhttp-2.7.5.jar
174175
okhttp-3.8.1.jar
175176
okio-1.13.0.jar

‎graph/api/pom.xml

+55
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!--
3+
~ Licensed to the Apache Software Foundation (ASF) under one or more
4+
~ contributor license agreements. See the NOTICE file distributed with
5+
~ this work for additional information regarding copyright ownership.
6+
~ The ASF licenses this file to You under the Apache License, Version 2.0
7+
~ (the "License"); you may not use this file except in compliance with
8+
~ the License. You may obtain a copy of the License at
9+
~
10+
~ http://www.apache.org/licenses/LICENSE-2.0
11+
~
12+
~ Unless required by applicable law or agreed to in writing, software
13+
~ distributed under the License is distributed on an "AS IS" BASIS,
14+
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
~ See the License for the specific language governing permissions and
16+
~ limitations under the License.
17+
-->
18+
19+
<project xmlns="http://maven.apache.org/POM/4.0.0"
20+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
21+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
22+
<modelVersion>4.0.0</modelVersion>
23+
<parent>
24+
<groupId>org.apache.spark</groupId>
25+
<artifactId>spark-parent_2.12</artifactId>
26+
<version>3.0.0-SNAPSHOT</version>
27+
<relativePath>../../pom.xml</relativePath>
28+
</parent>
29+
30+
<artifactId>spark-graph-api_2.12</artifactId>
31+
<packaging>jar</packaging>
32+
<name>Spark Project Graph API</name>
33+
<properties>
34+
<sbt.project.name>graph-api</sbt.project.name>
35+
</properties>
36+
<url>http://spark.apache.org/</url>
37+
38+
<dependencies>
39+
<dependency>
40+
<groupId>org.apache.spark</groupId>
41+
<artifactId>spark-core_${scala.binary.version}</artifactId>
42+
<version>${project.version}</version>
43+
</dependency>
44+
<dependency>
45+
<groupId>org.apache.spark</groupId>
46+
<artifactId>spark-sql_${scala.binary.version}</artifactId>
47+
<version>${project.version}</version>
48+
</dependency>
49+
</dependencies>
50+
51+
<build>
52+
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
53+
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
54+
</build>
55+
</project>

‎graph/cypher/pom.xml

+100
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!--
3+
~ Licensed to the Apache Software Foundation (ASF) under one or more
4+
~ contributor license agreements. See the NOTICE file distributed with
5+
~ this work for additional information regarding copyright ownership.
6+
~ The ASF licenses this file to You under the Apache License, Version 2.0
7+
~ (the "License"); you may not use this file except in compliance with
8+
~ the License. You may obtain a copy of the License at
9+
~
10+
~ http://www.apache.org/licenses/LICENSE-2.0
11+
~
12+
~ Unless required by applicable law or agreed to in writing, software
13+
~ distributed under the License is distributed on an "AS IS" BASIS,
14+
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
~ See the License for the specific language governing permissions and
16+
~ limitations under the License.
17+
-->
18+
19+
<project xmlns="http://maven.apache.org/POM/4.0.0"
20+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
21+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
22+
<modelVersion>4.0.0</modelVersion>
23+
<parent>
24+
<groupId>org.apache.spark</groupId>
25+
<artifactId>spark-parent_2.12</artifactId>
26+
<version>3.0.0-SNAPSHOT</version>
27+
<relativePath>../../pom.xml</relativePath>
28+
</parent>
29+
30+
<artifactId>spark-cypher_2.12</artifactId>
31+
<packaging>jar</packaging>
32+
<name>Spark Project Cypher</name>
33+
<properties>
34+
<sbt.project.name>cypher</sbt.project.name>
35+
</properties>
36+
<url>http://spark.apache.org/</url>
37+
38+
<dependencies>
39+
<dependency>
40+
<groupId>org.apache.spark</groupId>
41+
<artifactId>spark-core_${scala.binary.version}</artifactId>
42+
<version>${project.version}</version>
43+
</dependency>
44+
<dependency>
45+
<groupId>org.apache.spark</groupId>
46+
<artifactId>spark-sql_${scala.binary.version}</artifactId>
47+
<version>${project.version}</version>
48+
</dependency>
49+
<dependency>
50+
<groupId>org.apache.spark</groupId>
51+
<artifactId>spark-catalyst_${scala.binary.version}</artifactId>
52+
<version>${project.version}</version>
53+
</dependency>
54+
<dependency>
55+
<groupId>org.apache.spark</groupId>
56+
<artifactId>spark-graph-api_${scala.binary.version}</artifactId>
57+
<version>${project.version}</version>
58+
</dependency>
59+
<dependency>
60+
<groupId>org.opencypher</groupId>
61+
<artifactId>okapi-shade</artifactId>
62+
<version>${okapi.version}</version>
63+
</dependency>
64+
65+
<!-- Test dependencies -->
66+
67+
<dependency>
68+
<groupId>org.apache.spark</groupId>
69+
<artifactId>spark-core_${scala.binary.version}</artifactId>
70+
<version>${project.version}</version>
71+
<type>test-jar</type>
72+
<scope>test</scope>
73+
</dependency>
74+
<dependency>
75+
<groupId>org.apache.spark</groupId>
76+
<artifactId>spark-sql_${scala.binary.version}</artifactId>
77+
<version>${project.version}</version>
78+
<type>test-jar</type>
79+
<scope>test</scope>
80+
</dependency>
81+
<dependency>
82+
<groupId>org.apache.spark</groupId>
83+
<artifactId>spark-catalyst_${scala.binary.version}</artifactId>
84+
<version>${project.version}</version>
85+
<type>test-jar</type>
86+
<scope>test</scope>
87+
</dependency>
88+
<dependency>
89+
<groupId>org.opencypher</groupId>
90+
<artifactId>okapi-tck</artifactId>
91+
<version>${okapi.version}</version>
92+
<scope>test</scope>
93+
</dependency>
94+
</dependencies>
95+
96+
<build>
97+
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
98+
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
99+
</build>
100+
</project>

‎graph/graph/pom.xml

+50
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!--
3+
~ Licensed to the Apache Software Foundation (ASF) under one or more
4+
~ contributor license agreements. See the NOTICE file distributed with
5+
~ this work for additional information regarding copyright ownership.
6+
~ The ASF licenses this file to You under the Apache License, Version 2.0
7+
~ (the "License"); you may not use this file except in compliance with
8+
~ the License. You may obtain a copy of the License at
9+
~
10+
~ http://www.apache.org/licenses/LICENSE-2.0
11+
~
12+
~ Unless required by applicable law or agreed to in writing, software
13+
~ distributed under the License is distributed on an "AS IS" BASIS,
14+
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
~ See the License for the specific language governing permissions and
16+
~ limitations under the License.
17+
-->
18+
19+
<project xmlns="http://maven.apache.org/POM/4.0.0"
20+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
21+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
22+
<modelVersion>4.0.0</modelVersion>
23+
<parent>
24+
<groupId>org.apache.spark</groupId>
25+
<artifactId>spark-parent_2.12</artifactId>
26+
<version>3.0.0-SNAPSHOT</version>
27+
<relativePath>../../pom.xml</relativePath>
28+
</parent>
29+
30+
<artifactId>spark-graph_2.12</artifactId>
31+
<packaging>jar</packaging>
32+
<name>Spark Project Graph</name>
33+
<properties>
34+
<sbt.project.name>graph</sbt.project.name>
35+
</properties>
36+
<url>http://spark.apache.org/</url>
37+
38+
<dependencies>
39+
<dependency>
40+
<groupId>org.apache.spark</groupId>
41+
<artifactId>spark-cypher_${scala.binary.version}</artifactId>
42+
<version>${project.version}</version>
43+
</dependency>
44+
</dependencies>
45+
46+
<build>
47+
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
48+
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
49+
</build>
50+
</project>

‎pom.xml

+4
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,9 @@
106106
<module>external/kafka-0-10-assembly</module>
107107
<module>external/kafka-0-10-sql</module>
108108
<module>external/avro</module>
109+
<module>graph/api</module>
110+
<module>graph/cypher</module>
111+
<module>graph/graph</module>
109112
<!-- See additional modules enabled by profiles below -->
110113
</modules>
111114

@@ -124,6 +127,7 @@
124127
<yarn.version>${hadoop.version}</yarn.version>
125128
<zookeeper.version>3.4.6</zookeeper.version>
126129
<curator.version>2.7.1</curator.version>
130+
<okapi.version>0.4.2</okapi.version>
127131
<hive.group>org.spark-project.hive</hive.group>
128132
<hive.classifier></hive.classifier>
129133
<!-- Version used in Maven Hive dependency -->

‎project/SparkBuild.scala

+5-2
Original file line numberDiff line numberDiff line change
@@ -48,12 +48,15 @@ object BuildCommons {
4848
val streamingProjects@Seq(streaming, streamingKafka010) =
4949
Seq("streaming", "streaming-kafka-0-10").map(ProjectRef(buildLocation, _))
5050

51+
val graphProjects@Seq(graph, graphApi, cypher) =
52+
Seq("graph", "graph-api", "cypher").map(ProjectRef(buildLocation, _))
53+
5154
val allProjects@Seq(
5255
core, graphx, mllib, mllibLocal, repl, networkCommon, networkShuffle, launcher, unsafe, tags, sketch, kvstore, _*
5356
) = Seq(
5457
"core", "graphx", "mllib", "mllib-local", "repl", "network-common", "network-shuffle", "launcher", "unsafe",
5558
"tags", "sketch", "kvstore"
56-
).map(ProjectRef(buildLocation, _)) ++ sqlProjects ++ streamingProjects
59+
).map(ProjectRef(buildLocation, _)) ++ sqlProjects ++ streamingProjects ++ graphProjects
5760

5861
val optionallyEnabledProjects@Seq(kubernetes, mesos, yarn,
5962
sparkGangliaLgpl, streamingKinesisAsl,
@@ -332,7 +335,7 @@ object SparkBuild extends PomBuild {
332335
val mimaProjects = allProjects.filterNot { x =>
333336
Seq(
334337
spark, hive, hiveThriftServer, catalyst, repl, networkCommon, networkShuffle, networkYarn,
335-
unsafe, tags, tokenProviderKafka010, sqlKafka010, kvstore, avro
338+
unsafe, tags, tokenProviderKafka010, sqlKafka010, kvstore, avro, graph, graphApi, cypher
336339
).contains(x)
337340
}
338341

0 commit comments

Comments
 (0)