If you’re like most organisations, you have plenty of apps in place, but need a way to share data between them via integrations in order to generate business insight and drive decision-making.
Connecting multiple data sources makes it possible to get the ‘aha’ moments you need to remain competitive — in fact, you risk falling behind if you aren’t leveraging insights from your integrations to the fullest extent possible.
That said, tying apps together isn’t always as easy as it should be. In this article, we’ll explore a few solutions for common integrations problems — including why some of the tools you’re likely most familiar with aren’t always the best choice.
The Limitations with Tools Like Mulesoft and Boomi
Broadly speaking, there are two approaches to integration platforms in the marketplace today. The first is to make use of a single vendor/single product integration platform, such as Mulesoft or Boomi.
These platforms present themselves as a panacea for integration issues. However, the reality is that enterprise data is extremely heterogeneous. While these platforms might present effective demos for mainstream and contrived use cases (i.e. FTP/CSV to SQL Server), they often require extensive custom implementation work to meet custom enterprise requirements.
If you’re a customer with non-mainstream use cases, then out-of-the-box connectors for integrations probably won’t exist with these platforms. In this way, they’re essentially proprietary runtime environments with extremely high entry price points. They’re also likely to present difficulty when it comes to obtaining expertise and resources, due to their limited marketplaces.
How Azure Manages Integrations
The second of the two approaches to integration platforms is managing integrations through Azure.
Not only does Azure offer the ability to generate those ‘aha’ moments mentioned earlier in a systematic, repeatable way, it’s a broadly adopted cloud platform that’s backed by the world’s largest software vendor.
A couple of other key differentiators include Azure’s clear consumption-based fees and charges, its predictable pricing (which scales with the workload in question), and the competitive pressure on its pricing from other public cloud vendors such as AWS and GCP.
Also, consider that Azure’s product offerings have matured significantly in recent releases. Of particular note are its on-demand features, which present a true, serverless pay-as-you-go model. This is particularly true of Azure Data Factory and Azure Synapse’s SQL-on-demand.
Azure Architecture Elements That Support Integrations
Within the Azure platform, there are several specific architecture elements that can support your integration needs.
Azure Purview
Azure Purview is a unified platform for data governance that maps on-premises and cloud-based data sources.
Azure Purview Data Maps capture metadata about enterprise data present in analytics and operation systems on-premises and cloud, and are automatically kept up-to-date with built-in automated scanning and classification systems.
In addition, the Azure Purview Data Catalog allows business and technical users alike to quickly and easily find relevant data using a search experience with filters based on various lenses like glossary terms, classifications, sensitivity labels and more. For subject matter experts, the Purview Data Catalog provides data curation features like business glossary management and ability to automate tagging of data assets with glossary terms.
Further, data consumers and producers can visually trace the lineage of data assets, starting from the operational systems on-premises; through to movement, transformation, and enrichment with various data storage and processing systems in the cloud; to consumption in an analytics system (such as Power BI).
Azure Data Factory
Next up, Azure Data Factory is a serverless data integration service that allows organisations to integrate data across a variety of sources. Data Factory comes with a large number of in-built connectors, as well as the ability to create your own to address requirements such as Hydstra.
Even better, Azure Data Factory fully supports modern DevOps practices with pipeline definitions committed to Azure DevOps Git repositories with full source code control.
Azure Storage Gen 2
Azure Storage Gen 2 is a cost-effective solution for supporting integrations as a data lake technology, especially compared to more difficult to manage SQL server data tiers. This is because Azure Storage Gen 2 integrates with Synapse SQL on-demand to facilitate highly scalable and serverless SQL operations against Azure storage.
Compared to the enormous computing footprint required for a platform like SQL Server Analysis Services, Azure Storage Gen 2’s SQL on-demand operations are very cost-effective. And when data is at rest, the cost is minimal, and the scalability of the underlying storage is, for practical purposes, unlimited.
Azure Synapse
Finally, consider Azure Synapse, Azure’s enterprise analytics service. One key feature of Azure Synapse is its serverless SQL model, which allows burstable processing of data natively in Azure Storage Gen 2, without the committed compute resources that are normally associated with BI environments.
Notably, Azure Synapse also allows for integration with Apache Spark for any predictive or machine learning requirements organisations may have.
Leveraging the Cloud for Integrations
Regardless of the approach you take to managing your integrations and deriving business insight from them, make sure you’re fully aware of all your options. If you’ve only ever considered solutions like Mulesoft or Boomi, know that there may be better options available that allow you to leverage the scalable, cost-effective nature of the cloud.
If you have questions about this approach or want to get in touch with us about your cloud offerings, reach out to connect with Codify’s cloud specialists.