Apache Airflow!

Apache Airflow alternatives

  • RunDeck

  • RunDeck is an open source automation service with a web console, command line tools and a WebAPI. It lets you easily run automation tasks across a set of nodes.

    tags: automation server-management system-administration deployment configuration
  • Apache Oozie

  • Oozie is a workflow scheduler system to manage Apache Hadoop jobs.

    tags: jobs hadoop workflow-tool
  • Azkaban

  • Azkaban is a batch workflow job scheduler created at LinkedIn to run Hadoop jobs. Azkaban resolves the ordering through job dependencies and provides an easy to use web user interface to maintain and track your workflows.

    tags: workflow jobs etl hadoop
  • Luigi

  • The purpose of Luigi is to address all the plumbing typically associated with long-running batch processes. You want to chain many tasks, automate them, and failures will happen. These tasks can be anything, but are typically long running things like Hadoop jobs, dumping data to/from databases, running machine learning algorithms, or anything else.

    tags: etl