API Data Services vs Routes
API Data Services 01 Background
02 Decision Tree
vs
03 Scenario 1 – Routes
Routes 04 Scenario 2 – Data Services
05 Example API Data Services
06 Example Routes
07 Other Considerations
Background
• Data processing in general can be split into major type - Real Time and Batch Processing.
• Real time is when you want to process the data as and when it comes through.
• Batch Processing is when the data is collected and processed when we have the required volumes or at specific
intervals
• Data Services ad Route Jobs are likely to be used when you are dealing with Real Time scenarios.
• In this MBO/ICO we want to learn when to use API Data Services vs Routes.
3
Decision Tree
It is not clear when to use which type of job but depends on scenarios.
• Please read the best practice document to start with.
• Both the styles can be pretty much used to achieve your goal, it may be easier with one style than the other
• There are lot more components available when using Data Services
• Use Routes when
• dealing with one or multiple applications (Salesforce, Workday) applications
• passing data between different modules within or outside
• ftp the data from one to another
• Enhance the data – complex orchestration
• Gathering data from multiple sources
• Sending data to multiple recipients
• Its event driven and there is a need of constant listening to see when data arrives or is changed
• You want to use existing DI jobs
• Scenario required usage of HashMap
• Smaller sets of data
4
Decision Tree (continued)
• Enterprise Integration Patterns are required
(https://www.enterpriseintegrationpatterns.com/patterns/messaging/)
• route a message or messages consecutively through a series of processing steps, with the sequence of steps
unknown at design time and variable for each message.
• Requirement is not to send the same message twice to an endpoint ( without much coding effort)
• Requirement is to distribute message across several endpoints ( using variety of load balancing strategies in
the route job)
• Requirement is to route one or more messages to several endpoints at one go and process them in different
ways.
• Requirement is to route messages to a separate endpoint while forwarded to the ultimate destination.
• Requirement is to limit the number of messages flowing to a specific endpoint in order to prevent it from
getting overloaded.
• Use API Data Services when
• Its more data oriented
• Shifting big data chunks from source to target
• No complex data manipulation required
• More data volume required 5
• Scenario does not require HashMap ( as its is not thread Safe), only globalMap will do as its thread safe
Scenario 1 – Routes
A retail company needs to be able to integrate and pass data within its departments, idea is that when orders are
placed then immediately required parts need to be ordered from multiple suppliers to make that product,
payments need to be raised , product manufactured and shipping the product to the destination.
- Following questions can be asked to decide if Route jobs are the right choice here
• Is tight integration required between different departments
• Is this a batch or real time process
• Is complex data manipulation required
• Routes can be ideal in the above scenario as routes can offer tight integration between applications, event triggering can be
used along with routes to meet the goals for the scenario
6
Scenario 2 – API Data Services
In this example scenario an organization is moving from one application to another and will want to transfer
all data from one to another ( say from Sales force to Workday)
This case is mostly migration of data which requires movement of data from one place to another also there
will be a time frame when there might be some parallel running to synchronize the systems.
API Data Services are ideal in this scenario where data needs to be moved from one place to another, no
complex conversion are required and APIs from both the systems can be effectively use to transfer data and
track responses to achieve the goal.
7
Example API Data Services
Requirement is to simply migrate data from one place to another, No complexity involved, it’s a simple 1:1 transfer
Here we have created a Talend Api Data Services job which listens to a customer table and returns the data as per
the request and pass it to the other end, the data can be enhanced and passed to a different target system as
required.
8
Example API Data Services (continued)
9
Example Routes
• Use s3 buckets (or any event driven application like salesforce), SQS que, Route
job
• Route job constantly listens the SQS que
• As soon as a file is passed to the S3 bucket , SQS que will get populated and the
route job will know of some activity has taken place
( or any event driven system like salesforce , workday etc) route will get the
information and than can be split, enhanced, throttled or do all the things a route
job can perform and pass it to another application or target system). In our case
depending the name off the file we will pass the file to another S3 bucket
10
Example Routes (continued)
AWSSQS components reads data from the SQS que which gets populated when a file is added to S3 bucket
Talend job on next page
11
Example Routes (continued)
Talend job called from the routes job using cTalend job
12
Other Considerations
Choosing between a Route job vs a Data Services job may not be straight forward,
always make sure what the requirement is and are there Talend components in
Routes which will make life easier and better as compared to using Data Services.
13
