One of the first thing one tries when evaluating an integration platform, is a simple file to file integration. This Managed File Transfer is the moral equivalent of an “Hello World” program for integration and middleware platforms.
In this post, I will walk through how one does implement a simple FILE to FILE scenario with Crosscut. In the following post, I will outline the technical challenges and talk about the Azure components used under the hood to build such a scenario.
In the following weeks, I will use this dual post approach to explain our technical journey in building Crosscut. One post will outline how the scenario appears to customers and integration partners, while the next will delve into technical details.
So, let’s see how our integration partners use Crosscut to build a simple integration.
Building Publication and Subscription Ports
Like many integration platforms, Crosscut relies on a publish/subscribe integration pattern to transport messages from a source system to a target system. Of course, it also relies on the Message Endpoint pattern, known in Crosscut as Connectors to acquire data from a source system and transmit data to a target system.
So the first thing to do is to build a couple of ports, one that will detect files on the source system and another that will transmit this file to a target system.
So first, connect to Crosscut:
Jump to the “Developer” mode in the portal. That’s where a developer will design data integrations. The other two modes are “Monitoring”, where a user can gain insights into how the integrations behave – such as Message Interchanges, Event Logs, etc. – and “Management”, where an administrator create users and grant permissions to various parts of the Platform.
Select the “Publication Ports” menu on the left, and click on the “New” button to create a new publication port.
Give a meaningful name to the port and click on the “Select a Connector” box.
In this case, select the FILE inbound connector.
By default, only the required properties are shown when configuring a connector. Type in the location of the folder from which files will be taken on the source system, and click “Save”.
Back on the publication port configuration screen, notice that a “Request Pipeline” box is shown. This indicates that the publication port is a One-Way port and that an optional sequence of processing can be applied to the message before being submitted to Crosscut.
In this case, leave the box empty, and click “Save”.
In order to configure the subscription port, select the “Subscription Ports” menu on the left, and follow the same steps. Just select the outbound FILE connector and choose an appropriate location for a folder in which files will be transmitted on the target system.
Creating and Activating a Data Integration Flow
In order to build the data integration, one must link the publication port to the subscription port. Select the “Flows” menu on the left, and click the “New Flow” button to create a new flow.
Select the publication and subscription ports created in the previous steps, and use drag and drop to create a link from the publication port to the subscription port. This instructs Crosscut that whenever a message is published by that particular publication port, a copy should be delivered to the specified subscription port.
It is also possible to click on the link to set Subscription Filters in order to select only a subset of all the messages published by the publication port. But for this post, the subscription filter is a catch-all filter.
A flow must be activated before messages can be delivered to Crosscut.
Click the hamburger menu from the flow tile and select “Activate”. This operation is asynchronous, because it creates all the necessary infrastructure on Azure to allow for message delivery.
Once the flow is activated, the publication and subscription ports must be hosted in some sort of executable process, in order for the data integration to effectively take place.
Creating a Group for Hosting Publication and Subscription Ports
The executable process in which publication and subscription ports are hosted is part of a logical Group that is created and managed centrally in Crosscut. Each group can contain one or more instances of an executable, each of which hosts the publication and subscription ports that are associated with it.
Crosscut comes with a builtin Group, named Cloud Connect to host publication ports and subscription ports that connect to online systems, such as Salesforce.com, or an FTP server, for instance. In order to host publication and subscription ports that connect to a LOB system, one must create one or more dedicated groups.
From the left side, select the “Group” menu and click “New” to create a new Group.
Once a group is created, one or more ports must be associated with it. In a typical scenario, the publication and subscription ports would be associated with differents groups. Each group would map to an instance of a process that runs on a different machine. In this post, I choose to have a single group for demonstration purposes.
The ports associated with a group can be Started or Stopped. Obviously, for messaging to flow from one port to another, the ports must be started…
Running the Data Integration
All is now configured to run the data integration. The remaining steps involve downloading and running an instance of the executable process that hosts the publication and subscription ports associated with the group configured in the previous section of this post.
This executable process is named OnPrem-Connect because it is typically deployed on a machine running on-premises. Click on the Cloud icon on the right hand side of the group caption to download a setup program that install a pre-configured instance of the executable process.
Complete the installation wizard and wait for the service to start and register itself to the Crosscut portal.
Once registered, an instance of the OnPrem-Connect executable is identified by a green indicator on the group caption bar. There can be other instances of the same group, running on other machines for redundancy and performance purposes.
To complete this post, when a file is dropped in the publication folder…
… it is taken by the OnPrem-Connect instance that hosts the publication port and published to Crosscut as a message.
Once subscribed to, the message is then transferred to the OnPrem-Connect instance that hosts the subscription port, which delivers its payload to the target folder.
Here was a quick overview of a simple FILE to FILE integration scenario with Crosscut.