@CloudExpo Authors: Liz McMillan, Yeshim Deniz, Elizabeth White, Zakia Bouachraoui, Pat Romanski

Related Topics: @CloudExpo, Microservices Expo

@CloudExpo: Article

Information Life Cycle Management (ILM) Using the Amazon Cloud Platform

Archival and retrieval solutions using EC2 and EBS

Information Life Cycle Management (ILM)
Information life cycle management (ILM) is a process for managing information through its life cycle, from conception until disposal.

As part of the ILM policy for the enterprise, some major decisions need to be made:

  • When to archive the transaction data from the transactional or live databases to archived databases
  • Up to what period will data will be stored in the archival databases before they can be purged permanently

Successful  ILM Policies about information archival, retention and purging are important for the following:

  • Redundant or Less Important information on Live production databases, can needlessly increase demands on high-performance disk systems, slow application performance, and increase the time required to back up databases and upgrade applications.
  • There are a number of factors which can influence how long your information should be retained for. In broad terms, they can be divided into two main camps: internal and external factors.
  • o Internal factors will primarily be determined by operational considerations, for example how long the information will be of use to the organization in its business transactions
  • o External factors will be largely governed by legal and regulatory requirements. Many pieces of legislation will have statutes of limitations stated within them which helps define the minimum amount of time information covered by that legislation should be kept .

So we see that Enterprises need to Archive , Retain  and Purge large volumes  of data over their life cycle to  enable  Efficient Processing of New Business Capabilities, while meeting the legal needs of the past data.

Current Issues in Archival, Retention & Retrieval

  • Batch Jobs generally transfer the information from Transactional databases to Archival databases at frequent intervals
  • There are some predictability in the Growth of Disk Space for the Archival database however the Disk Spaces and Servers are pre allocated at least to 75% of maximum capacity
  • Archival Servers are kept relatively less utilized till the Retention Period for random requests for old data
  • No predictability in the archival retrieval requests and hence the systems are over-engineered for the spike conditions
  • When there is an sudden unprecedented need for archived data retrieval like the ‘External Audit' situation, the systems may fail to scale up as there is no dynamic scalability built in.

Amazon Electronic Block Store (EBS) As Archival Medium

  • Amazon EBS allows you to create storage volumes from 1 GB to 1 TB that can be mounted as devices by Amazon EC2 instances. Multiple volumes can be mounted to the same instance.
  • As it is Persistent and yet a Dynamically allocated storage mechanism, it naturally provides a ideal medium for Archival needs of the enterprise
  • Each storage volume is automatically replicated within the same Availability Zone. This prevents data loss due to failure of any single hardware component.
  • With Amazon Elastic Block Store, you only pay for what you use. Volume storage is charged by the amount you allocate until you release it, and is priced at a rate of $0.10 per allocated GB per month AmazonEBSalso charges $0.10 per 1 million I/O requests you make to your volume. This factor makes the Archival Storage cost relatively less when compared to traditional server hosting solutions.

Amazon EBS Backed EC2  as Retrieval Medium

  • As the EBS storage is considered as persistent storage for the Amazon EC2 instances, together EC2 & EBS can function as a elastic Retrieval medium for the archived data in a dynamic and elastic scalable manner
  • The EC2 Instances from AMI that are backed by EBS have several characteristics to play a effective Retrieval platform in an ILM life cycle
  • o Size Limit of 1 TB
  • o Data Persistence to support the retention policies
  • o Dynamic Scalability to take care spikes in need like an external Audit Cycle
  • With AMIs backed by Amazon EBS, you're charged for volume storage and usage in addition to the AMI and instance usage charges.
  • When an Amazon EBS-backed instance is stopped, you're not charged for instance usage; however, you're still charged for volume storage
  • The below diagram explains the ILM aspects of Archival, Retention & Retrieval in a most efficient and dynamic way, so that the costs of ILM is lower than doing it using traditional data center means. The below life cycle aspects are covered in the below diagram,
      • o Create / Modify Data
      • o Archival & Retrieval
      • o Protect & Recover
      • o Remove

As we see it, the  dynamic elasticity nature of Amazon Cloud  Platform (EC2, EBS)  supports these attributes of ILM and can function as a effective ILM platform if planned properly.

More Stories By Srinivasan Sundara Rajan

Highly passionate about utilizing Digital Technologies to enable next generation enterprise. Believes in enterprise transformation through the Natives (Cloud Native & Mobile Native).

CloudEXPO Stories
SYS-CON Events announced today that CrowdReviews.com has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5–7, 2018, at the Javits Center in New York City, NY. CrowdReviews.com is a transparent online platform for determining which products and services are the best based on the opinion of the crowd. The crowd consists of Internet users that have experienced products and services first-hand and have an interest in letting other potential buyers learn their thoughts on their experience.
In this presentation, you will learn first hand what works and what doesn't while architecting and deploying OpenStack. Some of the topics will include:- best practices for creating repeatable deployments of OpenStack- multi-site considerations- how to customize OpenStack to integrate with your existing systems and security best practices.
In his general session at 19th Cloud Expo, Manish Dixit, VP of Product and Engineering at Dice, discussed how Dice leverages data insights and tools to help both tech professionals and recruiters better understand how skills relate to each other and which skills are in high demand using interactive visualizations and salary indicator tools to maximize earning potential. Manish Dixit is VP of Product and Engineering at Dice. As the leader of the Product, Engineering and Data Sciences team at Dice, he takes a metrics-driven approach to management. His experience in building and managing high performance teams was built throughout his experience at Oracle, Sun Microsystems and SocialEkwity.
Transformation Abstract Encryption and privacy in the cloud is a daunting yet essential task for both security practitioners and application developers, especially as applications continue moving to the cloud at an exponential rate. What are some best practices and processes for enterprises to follow that balance both security and ease of use requirements? What technologies are available to empower enterprises with code, data and key protection from cloud providers, system administrators, insiders, government compulsion, and network hackers? Join Ambuj Kumar (CEO, Fortanix) to discuss best practices and technologies for enterprises to securely transition to a multi-cloud hybrid world.
Modern software design has fundamentally changed how we manage applications, causing many to turn to containers as the new virtual machine for resource management. As container adoption grows beyond stateless applications to stateful workloads, the need for persistent storage is foundational - something customers routinely cite as a top pain point. In his session at @DevOpsSummit at 21st Cloud Expo, Bill Borsari, Head of Systems Engineering at Datera, explored how organizations can reap the benefits of the cloud without losing performance as containers become the new paradigm.