sparklanes
latest

Contents:

  • Writing Data Processing Tasks
    • Creating a Task
    • Sharing Resources between Tasks
    • Accessing the Pyspark API from within Tasks
  • Defining Processing Lanes
    • YAML definition files
    • Using the API
    • Branching & Running Tasks in Parallel
  • Submitting lanes to Spark
    • Console script
    • Packaging
    • Extra Data
    • Spark Configuration
    • Custom main
  • API reference
    • sparklanes
    • Internals
      • sparklanes._framework package
        • sparklanes._framework.env module
        • sparklanes._framework.errors module
        • sparklanes._framework.lane module
        • sparklanes._framework.log module
        • sparklanes._framework.spark module
        • sparklanes._framework.task module
        • sparklanes._framework.validation module
      • sparklanes._submit package
        • sparklanes._submit._main module
        • sparklanes._submit.submit module
  • Example: Simple ETL lane
sparklanes
  • Docs »
  • Overview: module code

All modules for which code is available

  • sparklanes
    • sparklanes._framework.errors
    • sparklanes._framework.lane
    • sparklanes._framework.log
    • sparklanes._framework.spark
    • sparklanes._framework.task
    • sparklanes._framework.validation
    • sparklanes._submit._main
    • sparklanes._submit.submit

© Copyright 2019, Kevin Baumgarten Revision 0e5fc04a.

Built with Sphinx using a theme provided by Read the Docs.