Richard Searle


Survey of Akka integration with Spark

21 Jan 2015

Apache Spark now contains a mechanism to integrate with Akka Actors, with further documentation. That material is fairly limited and does not provide much guidance.

There is one example, which is rather more complicated than might be first expected.

Typesafe provide an activator with source. This implementation directly references an internal Spark component SparkEnv, whose documentation indicates that external usage is discouraged and it may be made private in a future release. The placement of SparkEnv within the Java API is a further discouragement.

Spark contains an official zeromqReceiver that uses ActorHelper, without all the issues mentioned above.

SPARK-5267 calls for a standard Camel implementation, without providing any details or source code.

The camel-spark project illustrates a Camel integration via akka-camel.

DataStax provide a Cassandra Connector Demo