Skip to main content

Spark Installation

Ensure that you have Apache Spark installed to use the Spark components. Once complete, simply pip install the required Python libraries.

Currently Xircuits Spark components are tested and working on

  • Windows 10 Spark 3.1.2 and Hadoop 3.2

  • Centos 8 [Vector Engine] Spark 3.1.1 and Hadoop 3.3

Spark Installation on Windows

Refer to this documentation for Windows installation:

Once complete, try:

# cd into your spark dir and run spark-shell

PS C:\apps\opt\spark-3.1.2-bin-hadoop3.2\bin> spark-shell
Using Spark's default log4j profile: org/apache/spark/
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
21/11/02 13:15:07 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
Spark context Web UI available at http://LAPTOP-71A1ED6U:4041
Spark context available as 'sc' (master = local[*], app id = local-1635830108092).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.1.2

Using Scala version 2.12.10 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_261)
Type in expressions to have them evaluated.
Type :help for more information.


You can also see the Spark web UI by accessing http://localhost:4040/



# Could not locate Hadoop executable: C:\apps\opt\spark-3.1.2-bin-hadoop3.2\bin\winutils.exe -see

Download corresponding Hadoop winutils from

Spark Installation for Centos8 [Vector Engine]

Please refer to this guide for Centos8 [Vector Engine].



Modified from

Pinguin.csv //