Welcome!

@CloudExpo Authors: Elizabeth White, Liz McMillan, Shelly Palmer, Pat Romanski, Yeshim Deniz

Related Topics: @CloudExpo, Microservices Expo

@CloudExpo: Article

Information Life Cycle Management (ILM) Using the Amazon Cloud Platform

Archival and retrieval solutions using EC2 and EBS

Information Life Cycle Management (ILM)
Information life cycle management (ILM) is a process for managing information through its life cycle, from conception until disposal.

As part of the ILM policy for the enterprise, some major decisions need to be made:

  • When to archive the transaction data from the transactional or live databases to archived databases
  • Up to what period will data will be stored in the archival databases before they can be purged permanently

Successful  ILM Policies about information archival, retention and purging are important for the following:

  • Redundant or Less Important information on Live production databases, can needlessly increase demands on high-performance disk systems, slow application performance, and increase the time required to back up databases and upgrade applications.
  • There are a number of factors which can influence how long your information should be retained for. In broad terms, they can be divided into two main camps: internal and external factors.
  • o Internal factors will primarily be determined by operational considerations, for example how long the information will be of use to the organization in its business transactions
  • o External factors will be largely governed by legal and regulatory requirements. Many pieces of legislation will have statutes of limitations stated within them which helps define the minimum amount of time information covered by that legislation should be kept .

So we see that Enterprises need to Archive , Retain  and Purge large volumes  of data over their life cycle to  enable  Efficient Processing of New Business Capabilities, while meeting the legal needs of the past data.

Current Issues in Archival, Retention & Retrieval

  • Batch Jobs generally transfer the information from Transactional databases to Archival databases at frequent intervals
  • There are some predictability in the Growth of Disk Space for the Archival database however the Disk Spaces and Servers are pre allocated at least to 75% of maximum capacity
  • Archival Servers are kept relatively less utilized till the Retention Period for random requests for old data
  • No predictability in the archival retrieval requests and hence the systems are over-engineered for the spike conditions
  • When there is an sudden unprecedented need for archived data retrieval like the ‘External Audit' situation, the systems may fail to scale up as there is no dynamic scalability built in.

Amazon Electronic Block Store (EBS) As Archival Medium

  • Amazon EBS allows you to create storage volumes from 1 GB to 1 TB that can be mounted as devices by Amazon EC2 instances. Multiple volumes can be mounted to the same instance.
  • As it is Persistent and yet a Dynamically allocated storage mechanism, it naturally provides a ideal medium for Archival needs of the enterprise
  • Each storage volume is automatically replicated within the same Availability Zone. This prevents data loss due to failure of any single hardware component.
  • With Amazon Elastic Block Store, you only pay for what you use. Volume storage is charged by the amount you allocate until you release it, and is priced at a rate of $0.10 per allocated GB per month AmazonEBSalso charges $0.10 per 1 million I/O requests you make to your volume. This factor makes the Archival Storage cost relatively less when compared to traditional server hosting solutions.

Amazon EBS Backed EC2  as Retrieval Medium

  • As the EBS storage is considered as persistent storage for the Amazon EC2 instances, together EC2 & EBS can function as a elastic Retrieval medium for the archived data in a dynamic and elastic scalable manner
  • The EC2 Instances from AMI that are backed by EBS have several characteristics to play a effective Retrieval platform in an ILM life cycle
  • o Size Limit of 1 TB
  • o Data Persistence to support the retention policies
  • o Dynamic Scalability to take care spikes in need like an external Audit Cycle
  • With AMIs backed by Amazon EBS, you're charged for volume storage and usage in addition to the AMI and instance usage charges.
  • When an Amazon EBS-backed instance is stopped, you're not charged for instance usage; however, you're still charged for volume storage
  • The below diagram explains the ILM aspects of Archival, Retention & Retrieval in a most efficient and dynamic way, so that the costs of ILM is lower than doing it using traditional data center means. The below life cycle aspects are covered in the below diagram,
      • o Create / Modify Data
      • o Archival & Retrieval
      • o Protect & Recover
      • o Remove

As we see it, the  dynamic elasticity nature of Amazon Cloud  Platform (EC2, EBS)  supports these attributes of ILM and can function as a effective ILM platform if planned properly.

More Stories By Srinivasan Sundara Rajan

Highly passionate about utilizing Digital Technologies to enable next generation enterprise. Believes in enterprise transformation through the Natives (Cloud Native & Mobile Native).

@CloudExpo Stories
Cloud promises the agility required by today’s digital businesses. As organizations adopt cloud based infrastructures and services, their IT resources become increasingly dynamic and hybrid in nature. Managing these require modern IT operations and tools. In his session at 20th Cloud Expo, Raj Sundaram, Senior Principal Product Manager at CA Technologies, will discuss how to modernize your IT operations in order to proactively manage your hybrid cloud and IT environments. He will be sharing bes...
SYS-CON Events announced today that DivvyCloud will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. DivvyCloud software enables organizations to achieve their cloud computing goals by simplifying and automating security, compliance and cost optimization of public and private cloud infrastructure. Using DivvyCloud, customers can leverage programmatic Bots to identify and remediate common cloud problems in rea...
SYS-CON Events announced today that Tintri, Inc, a leading provider of enterprise cloud infrastructure, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Tintri offers an enterprise cloud platform built with public cloud-like web services and RESTful APIs. Organizations use Tintri all-flash storage with scale-out and automation as a foundation for their own clouds – to build agile development environments...
SYS-CON Events announced today that Progress, a global leader in application development, has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Enterprises today are rapidly adopting the cloud, while continuing to retain business-critical/sensitive data inside the firewall. This is creating two separate data silos – one inside the firewall and the other outside the firewall. Cloud ISVs ofte...
SYS-CON Events announced today that Tappest will exhibit MooseFS at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. MooseFS is a breakthrough concept in the storage industry. It allows you to secure stored data with either duplication or erasure coding using any server. The newest – 4.0 version of the software enables users to maintain the redundancy level with even 50% less hard drive space required. The software func...
SYS-CON Events announced today that Interoute has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Interoute is the owner operator of Europe's largest network and a global cloud services platform, which encompasses over 70,000 km of lit fiber, 15 data centers, 17 virtual data centers and 33 colocation centers, with connections to 195 additional partner data centers. Our full-service Unifie...
SYS-CON Events announced today that EARP will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. "We are a software house, so we perfectly understand challenges that other software houses face in their projects. We can augment a team, that will work with the same standards and processes as our partners' internal teams. Our teams will deliver the same quality within the required time and budget just as our partn...
SYS-CON Events announced today that Carbonite will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Carbonite protects your entire IT footprint with the right level of protection for each workload, ensuring lower costs and dependable solutions with DoubleTake and Evault.
SYS-CON Events announced today that Super Micro Computer, Inc., a global leader in compute, storage and networking technologies, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Supermicro (NASDAQ: SMCI), the leading innovator in high-performance, high-efficiency server technology, is a premier provider of advanced server Building Block Solutions® for Data Center, Cloud Computing, Enterprise IT, Hadoop/...
SYS-CON Events announced today that Outscale will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Outscale's technology makes an automated and adaptable Cloud available to businesses, supporting them in the most complex IT projects while controlling their operational aspects. You boost your IT infrastructure's reactivity, with request responses that only take a few seconds.
While some vendors scramble to create and sell you a fancy solution for monitoring your spanking new Amazon Lambdas, hear how you can do it on the cheap using just built-in Java APIs yourself. By exploiting a little-known fact that Lambdas aren’t exactly single threaded, you can effectively identify hot spots in your serverless code. In his session at 20th Cloud Expo, David Martin, Principal Product Owner at CA Technologies, will give a live demonstration and code walkthrough, showing how to ov...
SYS-CON Events announced today that Technologic Systems Inc., an embedded systems solutions company, will exhibit at SYS-CON's @ThingsExpo, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Technologic Systems is an embedded systems company with headquarters in Fountain Hills, Arizona. They have been in business for 32 years, helping more than 8,000 OEM customers and building over a hundred COTS products that have never been discontinued. Technologic Systems’ pr...
As cloud adoption continues to transform business, today's global enterprises are challenged with managing a growing amount of information living outside of the data center. The rapid adoption of IoT and increasingly mobile workforce are exacerbating the problem. Ensuring secure data sharing and efficient backup poses capacity and bandwidth considerations as well as policy and regulatory compliance issues.
SYS-CON Events announced today that Cloudistics, an on-premises cloud computing company, has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Cloudistics delivers a complete public cloud experience with composable on-premises infrastructures to medium and large enterprises. Its software-defined technology natively converges network, storage, compute, virtualization, and management into a ...
New competitors, disruptive technologies, and growing expectations are pushing every business to both adopt and deliver new digital services. This ‘Digital Transformation’ demands rapid delivery and continuous iteration of new competitive services via multiple channels, which in turn demands new service delivery techniques – including DevOps. In this power panel at @DevOpsSummit 20th Cloud Expo, moderated by DevOps Conference Co-Chair Andi Mann, panelists will examine how DevOps helps to meet th...
SYS-CON Events announced today that Progress, a global leader in application development, has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Enterprises today are rapidly adopting the cloud, while continuing to retain business-critical/sensitive data inside the firewall. This is creating two separate data silos – one inside the firewall and the other outside the firewall. Cloud ISVs oft...
As DevOps methodologies expand their reach across the enterprise, organizations face the daunting challenge of adapting related cloud strategies to ensure optimal alignment, from managing complexity to ensuring proper governance. How can culture, automation, legacy apps and even budget be reexamined to enable this ongoing shift within the modern software factory?
SYS-CON Events announced today that A&I Solutions has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Founded in 1999, A&I Solutions is a leading information technology (IT) software and services provider focusing on best-in-class enterprise solutions. By partnering with industry leaders in technology, A&I assures customers high performance levels across all IT environments including: mai...
SYS-CON Events announced today that Peak 10, Inc., a national IT infrastructure and cloud services provider, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Peak 10 provides reliable, tailored data center and network services, cloud and managed services. Its solutions are designed to scale and adapt to customers’ changing business needs, enabling them to lower costs, improve performance and focus intern...
SYS-CON Events announced today that Linux Academy, the foremost online Linux and cloud training platform and community, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Linux Academy was founded on the belief that providing high-quality, in-depth training should be available at an affordable price. Industry leaders in quality training, provided services, and student certification passes, its goal is to c...