Click here to close now.

Welcome!

Cloud Expo Authors: Jason Bloomberg, Clinton Wolfe, Dana Gardner, Kevin Jackson, Marco Meinardi

Related Topics: Cloud Expo, Java, Linux, Big Data Journal

Cloud Expo: Article

Monitoring and Analyzing AWS CloudTrail Data

Monitoring and Analyzing AWS CloudTrail data from multiple AWS regions

We recently released AWS CloudTrail integration with Logentries - and not surprisingly we've seen a significant uptick in adoption as one of our most popular integrations. My job as director of customer success is to make things as simple for our customers as possible. One question that consistently pops up, is how to collect AWS CloudTrail logs from multiple AWS regions.

We follow Amazon's best practices when it comes to integrating with, and receiving information from, CloudTrail. In short, this works as follows:

  • When configuring CloudTrail, it will write events to a S3 bucket.
  • You can configure Cloudtrail to send notifications to an Amazon SNS topic whenever new log events are recorded.
  • You can get updates sent to an Amazon Simple Queue Service (Amazon SQS) queue, which enables you to handle these notifications programmatically.
  • To configure Logentries to consume your Cloudtrail logs, simply add the URL of the SQS queue to the Logentries/Cloudtrail setup page.

Logentries speaks directly to the SQS queue inside of your AWS account, so an obvious question that presents itself is: If I'm running in multiple AWS regions, how do I get Logentries to pull from all of the regions?

The simple answer: you don't. Make AWS do the work for you!

Following the steps outlined below, you'll be able to monitor and analyze CloudTrail logs from any number of AWS regions all within one Logentries account.

Create an S3 Bucket
If you're new to the CloudTrail setup, the first requirement of CloudTrail logging is that the logs must go "somewhere." In AWS, this somewhere happens to be a S3 bucket which you should create. Simply navigate to the S3 service and select ‘Create Bucket'. By default, all permissions required are given to the bucket - i.e. there is no extra permissions/configuration necessary to configure CloudTrail logging with Logentries.
Screen Shot 2014-08-19 at 12.55.03 PM

Create an SQS Queue in a Primary Region
Next up, we need to create an SQS Queue to allow Logentries to consume your CloudTrail data. Create a new Queue and provide a ‘Queue Name' - default options are fine.
SQS_Management_Console_and_LogentriesCloudtrail_docx

Add permissions to the SQS Queue
Once the queue has been created, the correct permissions must be applied. When adding permissions to the SQS queue, you need to add your full account number/name (officially called the AWS User ARN).

To get the User ARN navigate to the IAM Service, select the user that you want to utilize and click ‘Summary'. When the user is created within the IAM section, make sure that the user has at least ‘Read-Only' access - so that the user has the relevant permissions to read the bucket. The string you need is available under User ARN in the ‘Summary' section and follows this format:

arn:aws:iam::<account_code>:user/

Next add ‘Receive', ‘Send', and ‘Delete' Actions to the SQS Queue(see below):
SQS_Management_Console
Enable CloudTrail in any region, and publish to an SNS topic

Once the above three steps have been complete, it's time to enable CloudTrail in the relevant regions. Navigate to the CloudTrail Service in your AWS Console and turn on CloudTrail. Do not create a new S3 bucket, but instead select the S3 Bucket created in step one above from the drop down menu. Once you've done this, click the Advanced link. For the first region you enable CloudTrail for, remember to include Global Services under Advanced options - this record API calls from any global AWS services such as IAM or AWS STS. Make sure that "SNS Notification for every log file delivery" is checked, and finally, specify a SNS Topic to publish to. A new SNS topic name should be given - and will be created by CloudTrail.

Follow these above steps for each region that you want to collect CloudTrail logs from. NOTE: when adding subsequent regions you will want to exclude Global Services to avoid duplicate log events being recorded for your Global Services.

Screen Shot 2014-08-19 at 2.48.02 PM
Subscribe the SQS Queue to the multiple SNS topics
Once each region has been setup, the last step in AWS is to subscribe your SQS Queue to each newly created SNS topic. Navigate to the SQS Service in your AWS Console and highlight the queue created in step 2 above. Under the ‘Queue Actions' menu at the top select ‘Subscribe Queue to SNS Topic'. Use the ‘Topic Region' drop down to select the region and the ‘Choose a Topic' drop down to select the topic created in the previous step. Hit the ‘Subscribe' button and wait for the confirmation that the queue has subscribed to that topic.

After selecting subscribe make sure to copy the SQS URL from the ‘Details' section on the page.
SQS_Management_Console1

Setup Logentries to Pull data from the SQS Queue
Login to your Logentries account and navigate to your AWS settings area (My Account -> AWS). Select enable CloudTrail, supply your IAM access key, secret key, and SQS URL that you have copied above. Hit Save! Note: your IAM access key and secret key are made available to your when you create a new IAM user and should be stored safely.

Log Data from CloudTrail will begin to stream in within approximately 15 minutes.

Account___Logentries
Sit back and let Logentries do it's magic!
Visit our CloudTrail documentation to see some of the other cool things you can do - in particular we provide out of the box tags and alerts for important CloudTrail events. Have questions or ideas how we can make our CloudTrail integration better? Reach out to me directly at [email protected].

 

More Stories By Trevor Parsons

Trevor Parsons is Chief Scientist and Co-founder of Logentries. Trevor has over 10 years experience in enterprise software and, in particular, has specialized in developing enterprise monitoring and performance tools for distributed systems. He is also a research fellow at the Performance Engineering Lab Research Group and was formerly a Scientist at the IBM Center for Advanced Studies. Trevor holds a PhD from University College Dublin, Ireland.

@CloudExpo Stories
Big Data is amazing, it's life changing and yes it is changing how we see our world. Big Data, however, can sometimes be too big. Organizations that are not amassing massive amounts of information and feeding into their decision buckets, smaller data that feeds in from customer buying patterns, buying decisions and buying influences can be more useful when used in the right way. In their session at Big Data Expo, Ermanno Bonifazi, CEO & Founder of Solgenia, and Ian Khan, Global Strategic Positi...
SYS-CON Media announced today that XebiaLabs launched a popular blog feed on DevOps Journal with close to 2,000 story reads in less than a day. DevOps Journal is focused on this critical enterprise IT topic in the world of cloud computing. DevOps Journal brings valuable information to DevOps professionals who are transforming the way enterprise IT is done.
Since 2008 and for the first time in history, more than half of humans live in urban areas, urging cities to become “smart.” Today, cities can leverage the wide availability of smartphones combined with new technologies such as Beacons or NFC to connect their urban furniture and environment to create citizen-first services that improve transportation, way-finding and information delivery. In her session at @ThingsExpo, Laetitia Gazel-Anthoine, CEO of Connecthings, will focus on successful use c...
Cloudian, Inc., the leading provider of hybrid cloud storage solutions, today announced availability of Cloudian HyperStore 5.1 software. HyperStore 5.1 is an enhanced Amazon S3-compliant, plug-and-play hybrid cloud software solution that now features full Apache Hadoop integration. Enterprises can now transform big data into smart data by running Hadoop analytics on HyperStore software and appliances. This in-place analytics, with no need to offload data to other systems for Hadoop analyses, en...
XebiaLabs has announced record growth and major highlights for 2014. These milestones include: Triple-digit worldwide revenue growth. Record number of new customers including 3M, Allianz, American Express, Credit Agricole, Digital Globe, Electronic Arts, EMC, Expedia, Fandango / NBC Universal, GATX, General Electric, ING, KPMG, Liberty Mutual, Natixis, Paychex, Providence Health, Societe General, Thomson Reuters TIIA-CREF and Umpqua Bank.
CodeFutures has announced Dan Lynn as its new CEO. Lynn assumes the role from Founder Cory Isaacson, who has joined RMS and will now serve as chairman of CodeFutures. Lynn brings more than 14 years of advanced technology and business success experience, and will help CodeFutures build on its industry leadership around its Agile Big Data initiatives. His technical expertise will be invaluable in advancing CodeFutures’ AgilData platform and new processes for streamlining and gaining value from gro...
Roberto Medrano, Executive Vice President at SOA Software, had reached 30,000 page views on his home page - http://RobertoMedrano.SYS-CON.com/ - on the SYS-CON family of online magazines, which includes Cloud Computing Journal, Internet of Things Journal, Big Data Journal, and SOA World Magazine. He is a recognized executive in the information technology fields of SOA, internet security, governance, and compliance. He has extensive experience with both start-ups and large companies, having been ...
“We just completed the roll out of our first public and private cloud offerings, which are a combination of public, hybrid, and private cloud,” stated Erik Levitt, CEO of Open Data Centers, in this SYS-CON.tv interview at the 14th International Cloud Expo®, held June 10-12, 2014, at the Javits Center in New York City. Cloud Expo® 2014 Silicon Valley, November 4–6, at the Santa Clara Convention Center in Santa Clara, CA, will feature technical sessions from a rock star conference faculty and the...
SYS-CON Events announced today that GENBAND, a leading developer of real time communications software solutions, has been named “Silver Sponsor” of SYS-CON's WebRTC Summit, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. The GENBAND team will be on hand to demonstrate their newest product, Kandy. Kandy is a communications Platform-as-a-Service (PaaS) that enables companies to seamlessly integrate more human communications into their Web and mobile applicatio...
VictorOps is making on-call suck less with the only collaborative alert management platform on the market. With easy on-call scheduling management, a real-time incident timeline that gives you contextual relevance around your alerts and powerful reporting features that make post-mortems more effective, VictorOps helps your IT/DevOps team solve problems faster.
Skeuomorphism usually means retaining existing design cues in something new that doesn’t actually need them. However, the concept of skeuomorphism can be thought of as relating more broadly to applying existing patterns to new technologies that, in fact, cry out for new approaches. In his session at DevOps Summit, Gordon Haff, Senior Cloud Strategy Marketing and Evangelism Manager at Red Hat, will discuss why containers should be paired with new architectural practices such as microservices ra...
Companies today struggle to manage the types and volume of data their customers and employees generate and use every day. With billions of requests daily, operational consistency can be elusive. In his session at Big Data Expo, Dave McCrory, CTO at Basho Technologies, will explore how a distributed systems solution, such as NoSQL, can give organizations the consistency and availability necessary to succeed with on-demand data, offering high availability at massive scale.
From telemedicine to smart cars, digital homes and industrial monitoring, the explosive growth of IoT has created exciting new business opportunities for real time calls and messaging. In his session at @ThingsExpo, Ivelin Ivanov, CEO and Co-Founder of Telestax, shared some of the new revenue sources that IoT created for Restcomm – the open source telephony platform from Telestax. Ivelin Ivanov is a technology entrepreneur who founded Mobicents, an Open Source VoIP Platform, to help create, de...
The industrial software market has treated data with the mentality of “collect everything now, worry about how to use it later.” We now find ourselves buried in data, with the pervasive connectivity of the (Industrial) Internet of Things only piling on more numbers. There’s too much data and not enough information. In his session at @ThingsExpo, Bob Gates, Global Marketing Director, GE’s Intelligent Platforms business, to discuss how realizing the power of IoT, software developers are now focu...
Security can create serious friction for DevOps processes. We've come up with an approach to alleviate the friction and provide security value to DevOps teams. In her session at DevOps Summit, Shannon Lietz, Senior Manager of DevSecOps at Intuit, will discuss how DevSecOps got started and how it has evolved. Shannon Lietz has over two decades of experience pursuing next generation security solutions. She is currently the DevSecOps Leader for Intuit where she is responsible for setting and driv...
Operational Hadoop and the Lambda Architecture for Streaming Data Apache Hadoop is emerging as a distributed platform for handling large and fast incoming streams of data. Predictive maintenance, supply chain optimization, and Internet-of-Things analysis are examples where Hadoop provides the scalable storage, processing, and analytics platform to gain meaningful insights from granular data that is typically only valuable from a large-scale, aggregate view. One architecture useful for capturing...
The 3rd International @ThingsExpo, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - is now accepting submissions to demo smart cars on the Expo Floor. Smart car sponsorship benefits include general brand exposure and increasing engagement with the developer ecosystem.
SYS-CON Events announced today that Vitria Technology, Inc. will exhibit at SYS-CON’s @ThingsExpo, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. Vitria will showcase the company’s new IoT Analytics Platform through live demonstrations at booth #330. Vitria’s IoT Analytics Platform, fully integrated and powered by an operational intelligence engine, enables customers to rapidly build and operationalize advanced analytics to deliver timely business outcomes ...
Thanks to Docker, it becomes very easy to leverage containers to build, ship, and run any Linux application on any kind of infrastructure. Docker is particularly helpful for microservice architectures because their successful implementation relies on a fast, efficient deployment mechanism – which is precisely one of the features of Docker. Microservice architectures are therefore becoming more popular, and are increasingly seen as an interesting option even for smaller projects, instead of bein...
DevOps is about increasing efficiency, but nothing is more inefficient than building the same application twice. However, this is a routine occurrence with enterprise applications that need both a rich desktop web interface and strong mobile support. With recent technological advances from Isomorphic Software and others, it is now feasible to create a rich desktop and tuned mobile experience with a single codebase, without compromising performance or usability.