The remote run is a special Xircuits feature which allows user to execute tasks remotely. While the main usage is for Spark Submit, it can also be used for submitting jobs to servers.
Configuring the Remote Run
The remote run configurations can be viewed at
.xircuits/config.ini. You will have 3 levels of configuration depth:
REMOTE_EXECUTION: The main remote run category. Example is
RUN_TYPES: The subcategories under the main remote runs options. Each of the run types will have their own
CONFIGURATION. Example for remote execution category
CONFIGURATION: The configurations for each
RUN_TYPES. There are 4 parameters that should be specified:
name: The name of the configuration. You may use the same name as the
command: The command that is to be executed. Multiple lines should be appended with
msg: The message to be displayed when users execute the configuration.
url: The url to be displayed. Useful when an external dashboard exists. For example the Spark dashboard is
How it Works
Under the hood, the execute configuration is performed using using the subprocess module. A new process is spawned when Remote Run is selected and executed.
CONFIGURATION will be appended to the compiled Python Xircuits file, such that what will be executed is
For example, if your
$SPARK_HOME/bin/spark-submit and you've compiled
SparkLinePlot.py, the process that will be executed is:
The output is displayed in the Xircuits Output Terminal.
__ __ ___ _ _
\ \ \ \/ (_)_ __ ___ _ _(_) |_ ___
\ \ \ /| | '__/ __| | | | | __/ __|
/ / / \| | | | (__| |_| | | |_\__ \
/_/ /_/\_\_|_| \___|\__,_|_|\__|___/
Xircuits is running...
Remote Execution in process...
Please go to http://localhost:8088/ for more details
Running Spark Submit using local mode
Whenever the config.ini file is modified, Xircuits will recognize the changes only after switching the run type via the toolbar.