Friday, July 21, 2017

High Availability / Disaster Recovery Plan

Today let's discuss about high availability (HA) or more precisely disaster recovery plan (DRP).

As with any system, downtime can have major consequences for businesses. This quick article simply discuss two ways of achieving HA for ProActive.

Overall architecture

There are multiple ways for ProActive to be configured for High Availability (HA) / Disaster Recovery Plan (DRP).

  • ProActive stores its state within a database and includes an abstraction layer to configure the connection to a database. By default, the database is embedded within the ProActive folder. The objective is consequently to connect ProActive to a HA database (e.g. MariaDb can be configured this way, AWS RDS, ...)
  • The state being stored in an external database, it is important to monitor the behavior of ProActive. If it does not respond, it can be restarted which will then restart the scheduler, the diverse interfaces and connect to the database.

Below are two simple examples.

Monday, July 10, 2017

Introduction to Job Planner

Job Planner Methods

  • polling informations from other website,
  • updating regularly your data,
  • performing verifications and maintenance,
  • testing for new file in folder,
  • etc.

In this article, we will review these methods and go deeper into the latest one: the job planner.

Friday, June 23, 2017

Workflow Catalog through Examples

Introduction with an example : the workflow lifecycle management

Today let’s discover the new workflow catalog from ProActive. In a few words, the Workflow Catalog is a ProActive component that provides storage and versioning of Workflows through a REST API.

For a simplified explanation, we have here an example of ProActive utilization with three buckets. Each buckets represents a different stage of the workflow lifecycle. For instance, the workflow1, in the development bucket, was edited 3 times at the moment. Each edition corresponds to a revision. All the people who have access to the same bucket can read and write on all the workflows and their revisions.

A few use cases :

How would you handle sharing workflows ? Since buckets can be accessed by several users, transferring workflows between buckets simplifies the sharing process.

What about when you need a specific workflow within hundreds ? Don’t worry, use the search tool to narrow the list returned. Parameters such as owner can be used, other custom fields are also available thanks to generic information (e.g. infrastructure, language, etc.).

You found new bugs in the latest workflow revision ? The delete function can remove a selected revision to come back to another version.

Now let’s try some of these functions :



Tuesday, April 4, 2017

Legal & General Use Case

Resource consumption optimization and cloud leverage

Financial institutions are heavy consumers of computer resources for calculation of risks, opportunities, etc. They will take advantage of schedulers to ensure good distribution of workloads onto their existing infrastructure and minimize computing needs. Today let’s focus on Legal & General (L&G) case study and their transition to the new generation of open source scheduler.

Background and Specifications

Legal & General Group plc is a British multinational financial services company headquartered in London (UK). Its products include life insurance, general insurance, pensions and investments. It has operations in the United Kingdom, Egypt, France, Germany, the Gulf, India, the Netherlands and the United States. Their market capitalisation is at £13.5bn and they have £746bn assets under management.

Technologically, L&G used to base its Economic Capital and Solvency II simulation on IBM AlgoBatch. Their objective was to migrate from a private datacenter and Tibco DataSynapse to Azure Cloud and hybrid scheduling solution. Part of this migration the specifications were to handle Solvency II analysis on 2.5 million Monte Carlo scenarios, dynamically define and prioritize workloads and minimize time to delivery of results.

Friday, March 24, 2017

Orchestration and Meta-Scheduling for Big Data Architecture

MAPREDUCE, HADOOP, ETL, ELT, INFRASTRUCTURE, CLOUD, WORKFLOW, META-SCHEDULER AND COST OPTIMISATION

In today’s world, the adoption of Big Data is critical for most company survival. Storing, processing and extracting value from the data are becoming IT department's’ main focus. The huge amount of data, or as it is called Big Data, have four properties: Volume, Variety, Value and Velocity. Systems such as Hadoop, Spark, Storm, etc. are de facto the main building blocks for Big Data architectures (e.g. data lakes), but are fulfilling only part of the requirements. Moreover, in addition to this mix of features which represents a challenge for businesses, new opportunities will add even more complexity. Companies are now looking at integrating even more sources of data, at breaking silos (variety is increasing with structured and unstructured data), and at real-time and actionable data. All those are becoming key for decision makers.

Multiple solutions in the market have been supporting Big Data strategies, but none of them fits every company’s use cases. Consequently, each of these solutions will be responsible for extracting some meaning from the data. Although this mix of solutions adds complexity to infrastructure management, it also leverages the full information that can be extracted from the data. New questions are then raised like: How do I break company silos? How to make sense of this pool of unrelated and unstructured data? How to leverage supervised and unsupervised machine learning? How do I synchronize and orchestrate the different Big Data solutions? How can I allocate relevant resources? How do I ensure critical reports get prioritized? How do I enforce data locality rules and spread of the information? How do I monitor the whole data journey?

This article highlights two points on technical, operational and economic challenges around orchestration solutions. To leverage Big Data, companies will address those in order to optimize its infrastructure, extract faster and deeper insight into the data, and thus get a competitive edge. For a more detail and complete document, do not hesitate to download the full white paper.

Wednesday, January 18, 2017

Build Your Employee Self-Service Portal With ProActive

 In the past few years, people are consuming more and more services to perform their jobs properly. Managing them is the role of the IT department. However some employees might want to bypass this department to be faster which can create what is called “shadow IT”. Some others might be frustrated at being queued and waiting for additional checks to be performed.

 A solution for this situation needs to balance IT and non-IT needs by providing a user friendly interface which is fast for non-IT (from another department) users and provides governance for IT users. More precisely, this could be achieved by joining all possible applications and services into a single platform. This way, application and service lifecycles can be easily managed and custom templates can be made available for all to use (and create). This would allow for faster and more agile deployment as well as improve governance by giving visibility over the current services and by using a common standard.

 ProActive Cloud Automation offers a solution through a self-service portal to monitor and manage application and service lifecycles. The IT department can easily create templates which follow business policies and could be made available to user groups.

Wednesday, January 11, 2017

Network State Selection for Data Transfer

 Many applications such as online meetings and video streaming need to transmit large amount of data as quickly as possible. Good signal is consequently required for those transmissions, otherwise you might experience web conferences cutting or video pausing at unexpected time. Other applications need to send content with appropriate quality according to signal strength.

Resource selection based on network properties

 However, the quality of said signal can vary due to external parameters. Fortunately, networks are redundant which allow multiple paths with different properties to lead to the same device. This is why there is a use for flows to pass by carefully chosen resources. This choice may be made according to values such as ping and/or bandwidth.