Designing an Orchestration System

Are you a computer engineer or data engineer facing a situation where you have to deal with various applications and services and manage them manually?

Or, like John, a Data Engineer, do you have to store the data from API in a database regularly or run the scripts on ec2 instances periodically?

You must know orchestral music in which various musical instruments play coordinated so that the desired music effect is produced.

Credit: Pinterest

Likewise, in software computing, orchestration means configuring the tasks or workflows in an automated way to coordinate the systems. Before deep-diving into the details, let’s see the goals of orchestrating the workflows.

Goals of orchestration:

a) Process streamlining

b) Redundancy reduction

Difference between Orchestration and Workflow Automation:

Though automation and orchestration are related concepts, there is a significant difference.

Workflow automation is an approach to designing the task flow running independently following the business requirements. Here we talk about a single task.

Orchestration is about streamlining the different tasks to eliminate manual processes where monitoring the workflow becomes extremely easy, reducing errors and increasing the accuracy of the tasks.

When to use orchestration

1. Scheduling the tasks

Many tools are available like AWS Step Functions, Airflow, Kubernetes, and different container orchestration tools like AWS EKS, etc.

It is pretty easy to work with AWS Step Functions since it is a serverless service, and we, as developers, can focus on applications rather than managing the servers.

In this blog, we will see the hands-on orchestrating of the lambda functions with AWS Step Functions.

Understanding AWS Step Functions:

Step Functions allows you to create workflows that automatically scale to meet the needs of your business without requiring manual intervention.

Difference between Standard and Express Workflow

States:
You define your workflows in the Amazon States Language in AWS Step Functions.

The Step Functions console provides a graphical representation of the state machine to help visualize your application logic.

States are elements in your state machine.

Understanding the requirements: How to orchestrate the workflow?

1. Hands-on Example:

Problem Statement:

Let’s have the example where you want to fetch the data from API and store it in the database.

In our example, we will store the API response in the AWS S3 bucket as a JSON file.

This is the API: https://jsonplaceholder.typicode.com/todos/

The end goal of the problem statement is to dump the API response into the S3 bucket as a JSON file.

Step-by-step breakdown:

1. Understand the API: parameters required, data generated by the API endpoint
2. Check who are the data consumers and how should the data look like.
3. What database will be required to store the data and why?
4. Design the database tables accordingly
5. Designing a rough flow of how the elements of the system will be structured
6. Implementation of the orchestration system

This is what the first flow looks like.

Designing the Flowchart for the Problem Statement

Now let’s create the Lambda Function where we will write the program. Lambda Function is a serverless computing service provided by AWS. It offers the developer to write the code in different programming languages like Python, Java, and Node.JS.

Creating the Lambda Function

This is the link to check if the python library is available or if you need to install it.

One way to install the python library is to use AWS Cloud9.

Then, you can add the python library as a layer in the Lambda function.

Lambda function to collect the data from API endpoint

Now, let’s create a State machine to invoke our Lambda Function by clicking on ‘Create State Machine’. I like the most about making the state machines because we can drag and drop the services we want to include as part of the flow.

Create a state machine interface

The following diagram shows what the final state machine looks like.

State machine to invoke the Lambda Function

Testing the state machine:

You can start the execution to check whether the state machine executes successfully. The following diagram shows the graph inspector.

Test the State Machine

After the state machine is created successfully, it’s time to schedule it to run daily or as per the requirements. To do this, we need to create an EventBridge rule, another AWS service.

Firstly, we need to add rule details.

In the second step, we define the schedule pattern.

CRON expression is a string of 6 fields (minutes, hours, day of month, month, day of week, and year) describing the schedule details. (ref 1, ref 2)

Define the schedule pattern

Since we are creating this rule to run the step function state machine, we have to add the target AWS service.

The rule is created and is visible under the Rules on Amazon EventBridge UI.

AWS EventBridge Rule

Let’s check the next day if the step function state machine is executed successfully or not.

State machine Execution

🎉🎉Congratulations!!! Your Lambda function is now orchestrated with the step functions.

Advantages & Importance of Orchestration

Credit: ExpertHub

1. Step functions & Lambda functions are serverless: We, as developers, need to focus on the application instead of managing the servers. Cloud services are not really “serverless” because the servers are not user-controlled. They are abstracted away from the end-users. Another advantage is that it is a usage-based payment model.
2. Step functions State Machine Workflow: One can view the workflow like a flowchart and understand how the workflow will be executed.
3. Increased efficiency and accuracy of the tasks
4. Reliable processes
5. Linking and automating the provisioning of different services is achieved in an easy way
6. The workflows are auto-scalable and they have an inbuilt error handling mechanism

Conclusion

In this blog, we have seen what orchestration is, the difference between automation and orchestration, a demo orchestrating lambda functions with step functions, and the importance of orchestration.

Thanks for reading! 🙂