Hybrid architectures

Cloud Computing

Hybrid architectures

At a time when data variety and generation, as well as data transfer, are constantly increasing, the following questions arise: How can data be kept secure and at the same time in an easily scalable system? How can security regulations and governance requirements be complied with and the benefits of public clouds utilised? The answer: a hybrid cloud architecture. Hybrid cloud architecture is a mixture of traditional IT, private and public clouds, as well as internal and external infrastructure. This allows the potential of cloud computing to be utilised while keeping the potential risks and security regulations under control. But what does such a hybrid cloud architecture look like in practice and what are the advantages of such an architecture?

By Lukas Heusser, Analytics Consultant

AWS S3

AWS S3 is an object storage service from Amazon AWS. In the area of hybrid storage, S3 offers seamless integration between local applications and S3 storage. Amazon AWS offers various services for this, such as AWS Storage Gateway and AWS DataSync. The advantages of AWS S3 include scalability and data availability.

Memory Scaling

There are several options in the area of scalable storage. On the one hand, there are products such as AWS Storage Gateway. This tool is used to create and store an entire network drive in the S3 storage of Amazon AWS, which can be integrated locally and scaled very quickly. On the other hand, there is the option of regularly transferring the data to a cloud storage, such as AWS S3, using an ETL tool such as NiFi.

AWS Storage Gateway

AWS Storage Gateway is a hybrid storage service from Amazon AWS that enables the use of cloud storage locally. The service connects the local applications to the AWS services using a virtual machine or a hardware gateway appliance. The Storage Gateway enables the use of cloud resources with local data and applications.

The advantages of the Storage Gateway at a glance:

  • Easy to put into operation
  • Simple integration of resources
  • Storage can be integrated like a network drive via iSCSI, among other things
  • Data is stored in S3
  • Data can be easily restored
  • Network drives can be easily integrated into overlying applications such as CDP
  • Storage up to 100 TB per network drive

The disadvantages of the storage gateway are as follows:

  • Additional network component in the form of a VM or hardware
  • Requires around 300 GB of local memory for the upload buffer and the cache on the component

Conclusion on the Storage Gateway:

The AWS Storage Gateway is a simple but effective solution to extend on-premises storage into the cloud and to use cloud resources for local data and applications.

AWS Intelligent Tiering

Previously, there were four storage classes for AWS Object Storage (S3):

  • Standard - for frequently requested data
  • Standard IA - for rarely requested data
  • One Zone-IA - for non-critical and rarely requested data
  • Glacier - for rare and archived critical data

Previously, the storage class into which the data was to be categorised had to be specified during the data upload. If the requirements for the storage class changed, a manual adjustment was necessary. Now there is a new storage class for S3 called "AWS Intelligent Tiering". Data in this storage class is automatically categorised between the "Standard" and "Standard IA" storage classes, depending on the access pattern. The file accesses are analysed by AWS and the ideal storage class is determined for each file.

ETL-Tools

ETL tools can also be used to transport data from a local system to the cloud, such as Apache NiFi - an open source ETL tool from the Apache stack. The big difference between ETL tools and direct cloud connections is that there is only a connection to the cloud when a transport is executed. In addition, the data to be transported can be modified directly in the ETL tool or enriched with additional information or logic before it is extracted from the on-premises system. This means, for example, that critical and security-relevant data can already be anonymised on-premises to ensure compliance.

Apache Ozone

Another option is Apache Ozone. Apache Ozone is a distributed key-value object store that can manage large and small files, in contrast to HDFS, which was developed for larger files. The following are the advantages of Ozone:

  • Highly consistent
  • Layered architecture, the namespace management is shared by the node and block management layers.
  • Painless recovery, similar to HDFS, the HDDS can be effortlessly and effectively recovered from catastrophic events without losing data
  • Open source and therefore a large and helpful community
  • Integration with the Hadoop ecosystem

Although Apache Ozone has many advantages, productive use would be premature as Ozone is only available in an alpha version. It is not yet known when a first stable release will be published.

Computing

The public cloud also offers many opportunities in the area of computing. There are two ways to utilise the potential of the public cloud. Either everything is developed cloud-native or computing-intensive applications are developed in the cloud and applications with critical and security-relevant data continue to be developed on-premises.

Cloud Native

One way to utilise the advantages of a cloud is to develop all applications in the cloud. The advantages are as follows:

  • Full utilisation of the elasticity and scalability of the cloud
  • Pricing adapts to usage
  • No additional infrastructure that needs to be maintained and operated

The providers that offer services for application development in the cloud include AWS, Snowflake, Azure and Databricks.

Separate jobs

Previously, the storage class into which the data was to be categorised had to be specified during the data upload. If the requirements for the storage class changed, a manual adjustment was necessary. Now there is a new storage class for S3 called "AWS Intelligent Tiering". Data in this storage class is automatically categorised between the "Standard" and "Standard IA" storage classes, depending on the access pattern. The file accesses are analysed by AWS and the ideal storage class is determined for each file:

  • Compliance with legal and other regulations
  • Utilising the scalability and elasticity of the cloud

Such an architecture also has the following disadvantages:

  • Various systems must be operated and maintained
  • Uploading data to the cloud can be a bottleneck

More Clouds

In addition to AWS, the public cloud solution from Amazon, there are other cloud solutions. These include Microsoft Azure, a public cloud solution from Microsoft, and the Enterprise Service Cloud, a cloud solution from Swisscom with data storage in Switzerland.

Enterprise Service Cloud

The Swisscom Enterprise Service Cloud is a cloud solution from Swisscom and is primarily aimed at companies that want access to local experts and value keeping their data in Switzerland.

On the technical side, the Swisscom Enterprise Service Cloud is aimed at customers with complex IT infrastructures.

Swisscom and the Cloud

Swisscom is one of the technology leaders in the cloud sector in Switzerland. In addition to various major projects in the public cloud and certified experts in the areas of Azure and AWS, Swisscom operates its own cloud infrastructure with the "Enterprise Swisscom Cloud". This makes Swisscom the ideal companion, regardless of your cloud requirements.

If you are interested, please do not hesitate to contact us:
Lukas Heusser, Analytics Consultant

Lukas Heusser

Lukas Heusser

Analytics Consultant

Mehr getIT-Beiträge

Bereit für Swisscom

Finde deinen Job oder die Karrierewelt, die zu dir passt. In der du mitgestalten und dich weiterentwickeln willst.

Was du draus machst, ist was uns ausmacht.

Zu den Karrierewelten

Zu den offenen Security Stellen