sparklanes
latest
Contents:
Writing Data Processing Tasks
Creating a Task
Sharing Resources between Tasks
Accessing the Pyspark API from within Tasks
Defining Processing Lanes
YAML definition files
Using the API
Branching & Running Tasks in Parallel
Submitting lanes to Spark
Console script
Packaging
Extra Data
Spark Configuration
Custom main
API reference
sparklanes
Internals
sparklanes._framework package
sparklanes._framework.env module
sparklanes._framework.errors module
sparklanes._framework.lane module
sparklanes._framework.log module
sparklanes._framework.spark module
sparklanes._framework.task module
sparklanes._framework.validation module
sparklanes._submit package
sparklanes._submit._main module
sparklanes._submit.submit module
Example: Simple ETL lane
sparklanes
Docs
»
Index
Edit on GitHub
Index
A
|
B
|
C
|
G
|
I
|
J
|
L
|
M
|
P
|
R
|
S
|
T
|
U
|
V
A
add() (sparklanes.Lane method)
arg_spec() (in module sparklanes._framework.validation)
B
Branch (class in sparklanes)
build_lane_from_yaml() (in module sparklanes)
C
cache() (sparklanes._framework.task.LaneTask method)
cached (sparklanes._framework.task.TaskCache attribute)
CacheError
clear_cache() (sparklanes._framework.task.LaneTask method)
code (sparklanes._framework.errors.LaneSchemaError attribute)
conn (in module sparklanes)
G
get() (sparklanes._framework.task.TaskCache static method)
I
init_default() (sparklanes.SparkContextAndSessionContainer class method)
INIT_SPARK_ON_IMPORT (in module sparklanes._framework.env)
INTERNAL_LOGGER_NAME (in module sparklanes._framework.env)
J
join() (sparklanes._framework.task.LaneTaskThread method)
L
Lane (class in sparklanes)
LaneExecutionError
LaneImportError
LaneSchemaError
LaneTask (class in sparklanes._framework.task)
LaneTaskThread (class in sparklanes._framework.task)
M
main() (in module sparklanes._submit._main)
make_default_logger() (in module sparklanes._framework.log)
P
parse_args() (in module sparklanes._submit._main)
R
run() (sparklanes._framework.task.LaneTaskThread method)
(sparklanes.Lane method)
S
sc (sparklanes.SparkContextAndSessionContainer attribute)
set_sc() (sparklanes.SparkContextAndSessionContainer class method)
set_spark() (sparklanes.SparkContextAndSessionContainer class method)
spark (sparklanes.SparkContextAndSessionContainer attribute)
SPARK_APP_NAME (in module sparklanes._framework.env)
SparkContextAndSessionContainer (class in sparklanes)
sparklanes (module)
sparklanes._framework.env (module)
sparklanes._framework.errors (module)
sparklanes._framework.lane (module)
sparklanes._framework.log (module)
sparklanes._framework.spark (module)
sparklanes._framework.task (module)
sparklanes._framework.validation (module)
sparklanes._submit._main (module)
sparklanes._submit.submit (module)
submit_to_spark() (in module sparklanes._submit.submit)
T
Task() (in module sparklanes)
TaskCache (class in sparklanes._framework.task)
TaskInitializationError
U
uncache() (sparklanes._framework.task.LaneTask method)
UNNAMED_BRANCH_NAME (in module sparklanes._framework.env)
UNNAMED_LANE_NAME (in module sparklanes._framework.env)
V
validate_params() (in module sparklanes._framework.validation)
validate_schema() (in module sparklanes._framework.validation)
VERBOSE_TESTING (in module sparklanes._framework.env)
Read the Docs
v: latest
Versions
latest
Downloads
pdf
html
epub
On Read the Docs
Project Home
Builds
Free document hosting provided by
Read the Docs
.