Spark Jobs Management as a Service


Scroll Down


Mist Icon
Turn Analytics into Products

Increase the return on investment of big data analytics by building end user products on top of those.

Mist Icon
Expose data science and algorithms to enterprise applications

Mist makes it easier for developers to tap into big data algorithms from web apps by addressing those as APIs.

Mist Icon
Simplify the connection between business and data scientists

Deliver forecasting apps, smart alerts, and prescriptive insights that business can understand instead of statistical reports and raw algorithms.

Mist Icon
Let business interact with big data

Write a simple web app that lets business users type input into the model and provide feedback. Then the data scientist tunes the job based on that. Provides an easy interface that does not require coding.

Mist Icon
Hide complex big data details

Regular, non-big-data web programmers can run complicated big data aggregation jobs without having to understand distributed computing, machine learning, statistics, and advanced mathematics.

Mist Icon
Build productivity tools for data scientists

The advanced Mist user can use Mist to develop their own hosted notebooks, ETL builder, or write other productivity tools for data analysts.

Spark Applications Management: the Mist Technology

Scala, Python and SQL

Deploy Spark jobs implemented in Scala, Python, HiveQL and Spark SQL as pure functions available by REST API.

Synchronous and Asynchronous API

Supports either HTTP or Kafka/MQTT/AMQP for communication with Spark.

Orchestrates Spark Sessions

Mist instantiates and maintains the Spark context pool automatically for instant job execution and isolation.

Dynamic Spark clusters

Mist scales by creating on-demand EMR, ECS, DC/OS Spark clusters for new Spark contexts. It guarantees optimal resource utilization and SLA guarantees.

Docker per Spark Context

Mist manages multiple Spark sessions in multiple JVMs and/or Docker containers. This allows running jobs from multiple users and tenants in parallel.

Typesafe API

Mist automatically converts input requests into typed parameters for exposed Spark functions.

High availability and fault tolerance

Mist is built on the Akka resilient, distributed messaging framework. It can also be deployed on Mesos for high availability and efficient resource utilization.

Self healing

Mist persists job information so they can recover after program failure.

Data API Framework

(Pending) Mist functions accept and then return data frames. This allows building state safe data pipelines.

Other Hydrosphere.io Components

Open Source

Mist is open source softwares available under the Apache 2 License on GitHub