Welcome!

@CloudExpo Authors: Carmen Gonzalez, Yeshim Deniz, Jason Bloomberg, Ed Featherston, Elizabeth White

Related Topics: @CloudExpo, Containers Expo Blog, @DevOpsSummit

@CloudExpo: Blog Feed Post

Docker Cloud Monitoring and Logging By @Sematext | @CloudExpo #Cloud #Containers

Docker Cloud is a container management service that supports multiple cloud providers

Docker Cloud Monitoring and Logging
By Stefan Thies

Docker Cloud is a hosted service for Docker Container Management, originally based on Tutum Cloud, which was acquired by Docker in October 2015. Sematext supported the deployment of Sematext Docker Agent on Tutum Cloud from the get-go, so naturally we were quick to add support for Docker Cloud as well.

What is Docker Cloud?
Docker Cloud is a container management service that supports multiple cloud providers such as Amazon, DigitalOcean, IBM Softlayer, MS Azure and Packet.net. This makes it much easier to switch Docker deployments to different cloud providers or use a mix of providers including on-premises nodes for hybrid cloud applications. The user interface in Docker Cloud makes it easy to manage nodes on all supported cloud platforms and is able to deploy application stacks in containers, defined in a "Stack YAML" file. This Stack files are very similar to Docker Compose files, but with additional options, e.g. to define deployment strategies for the containers. The graphical user interface helps to view and modify container configurations.

Docker Cloud Metrics & Logs
Once containers are deployed you can get a very basic real-time log stream view per container (see below). This is helpful for a quick glance at the most recent logs of a specific container.

Real-time log view in Docker Cloud

Real-time log view in Docker Cloud

There are currently no Docker metrics exposed anywhere in Docker Cloud, though that will surely be added with time. Docker Cloud does an excellent job for the "Build, Ship and Run" containers paradigm. But if you've ever built a production system you know there is more to it. There is this little wrinkle called Operations. So let's talk about the more realistic scenario - "Build, Ship, Run and Monitor"

Thanks to the Docker API, it is possible to add this functionality to Docker Cloud. Sematext Docker Agent is a small container that collects all Docker metrics, all app and Docker logs and all Docker events from Docker Cloud and together with SPM for Performance Monitoring and Logsene for Log Management and Analytics it provides advanced Performance Monitoring and Log Management functionality for stacks deployed in Docker Cloud:

  1. Detailed Metrics with a long retention time. Having detailed metrics helps optimize resource usage of applications. Detailed metrics let you set application-specific alerts for any critical resources your applications depends on. Metrics are aggregated for all hosts, images and containers and filterable by hosts, images, and containers. This lets you drill down from a cluster view down to a single container while troubleshooting or simply understand operations details. Long retention times for metrics make it possible to compare resources before and after different deployments and releases or troubleshoot problems that appear only when a service has been running over several days or weeks!
  2. Full-text search, filtering, and analytics across all containers. Logs are collected, parsed and shipped by Sematext Docker Agent. The integrated charting functions in Logsene and integrations for Kibana and Grafana make it easy to analyze logs collected in Docker Cloud. In short, you can use Logsene as a "super grep" for your Docker and application logs, but also as amuch more affordable Splunk or any other BI tool, or a managed Elastic stack (aka ELK).
  3. Long retention time for logs, metrics and events. Comparing metrics and logs during deployments or watching the performance under different workloads requires one to store logs and metrics for a reasonable time. We have seen cases where memory leaks started to get serious after a few weeks of stable operations, although initially they were not detected. In such a case all context information like logs, events and metrics could be very valuable in identifying the root cause of such problems.
  4. Tracking of all Docker Events. Tracking of all Docker Events gives you a clear view of your containers' life cycle. For example, by collecting Events you gain insight into what happens with your containers during (re)deployments or the re-scheduling of containers to different nodes. Some containers might be configured for automatic restarts and the events could indicate if container processes crash frequently. In case of out-of-memory events, it might be wise to modify the memory limits or check with the developers, why this event happened.
  5. Anomaly detection and alerts for all logs and metrics. Who wants to watch metrics and logs all day long? Not me! Let the monitoring system watch outliers for metrics or query your logs! Anomaly detection can help reduce the noise and alert fatigue often caused by classic threshold-based alerts. Even log-alerting is possible with Logsene e.g. to detect anomalies in the log frequency of a specific query. For example, a search for "error" in the system might normally return a dozen non-critical errors, which could be ignored. A growth in the log frequency of error messages indicates that something might be going wrong. Another type of alerts is the Heartbeat alert for all cluster nodes. Disk Space alerts are very useful for Docker nodes, because Docker images might consume a lot of disk space. Docker Cloud runs some cleanup agents to remove unused containers and images; nevertheless the default disk-space alert created by SPM gives you an early warning before the capacity limit is reached.

Here's a short video about Log Management and Monitoring for Docker. It gives a general overview of Monitoring and Log Management in Docker context, plus shows you how to use SPM and Logsene as a single pane of glass for your Docker metrics and logs.

Having all this operational insights, and having it in a single pane of glass makes everyone's work (and that means life, too) simpler. We all want that, no? With that in mind, we've made sure the Sematext Docker Agent setup is super quick and easy:

  1. Get a free account at apps.sematext.com, if you don't have one already
  2. Create an SPM App of type "Docker" to obtain the SPM Application Token and/or
    Create a Logsene App to obtain the Logsene Application Token
  3. Click the "Deploy to Cloud" button in Sematext UI and copy the generated token into the Stackfile text field in Docker Cloud

    Create SPM app and deploy to Docker Cloud

    Create SPM app and deploy to Docker Cloud

  4. As soon you click "Create and deploy" in Docker Cloud the Sematext Docker Agent will be pulled from Docker Hub and it will start on all nodes managed by Docker Cloud. A few seconds later you should see Events, Logs & Metrics in SPM & Logsene.

Docker Cloud Metrics Overview in SPM

Docker Events and Metrics in SPM

Structured Docker Logs in Logsene / Kibana discover view

Everything mentioned above could take you just 10-15 minutes, meaning that in 10-15 you could be looking at charts with all your Docker operations data in one place, accessible by your whole team!

If you have feedback for monitoring and logging on Docker Cloud get in touch with us via @sematext or email us at [email protected] - we love to talk about monitoring and logging and appreciate user feedback that helps us improve our services and make them better serve your needs. If you want to try SPM or Logsene, start here.

Read the original blog entry...

More Stories By Sematext Blog

Sematext is a globally distributed organization that builds innovative Cloud and On Premises solutions for performance monitoring, alerting and anomaly detection (SPM), log management and analytics (Logsene), and search analytics (SSA). We also provide Search and Big Data consulting services and offer 24/7 production support for Solr and Elasticsearch.

@CloudExpo Stories
As pervasive as cloud technology is -- and as persuasive as the arguments are for using it -- the cloud has its limits. Some companies will always have security concerns about storing data in the cloud and certain high-transaction applications will always be better suited for on-premises storage. Those statements were among the bottom-line takeaways delivered at Cloud Expo this week, a three day, bi-annual event focused on cloud technologies, adoption and associated challenges.
SYS-CON Events announced today that Hitachi, the leading provider the Internet of Things and Digital Transformation, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Hitachi Data Systems, a wholly owned subsidiary of Hitachi, Ltd., offers an integrated portfolio of services and solutions that enable digital transformation through enhanced data management, governance, mobility and analytics. We help globa...
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm.
SYS-CON Events announced today that Hitachi Data Systems, a wholly owned subsidiary of Hitachi LTD., will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City. Hitachi Data Systems (HDS) will be featuring the Hitachi Content Platform (HCP) portfolio. This is the industry’s only offering that allows organizations to bring together object storage, file sync and share, cloud storage gateways, and sophisticated search an...
While some vendors scramble to create and sell you a fancy solution for monitoring your spanking new Amazon Lambdas, hear how you can do it on the cheap using just built-in Java APIs yourself. By exploiting a little-known fact that Lambdas aren’t exactly single threaded, you can effectively identify hot spots in your serverless code. In his session at 20th Cloud Expo, David Martin, Principal Product Owner at CA Technologies, will give a live demonstration and code walkthrough, showing how to o...
SYS-CON Events announced today that CollabNet, a global leader in enterprise software development, release automation and DevOps solutions, will be a Bronze Sponsor of SYS-CON's 20th International Cloud Expo®, taking place from June 6-8, 2017, at the Javits Center in New York City, NY. CollabNet offers a broad range of solutions with the mission of helping modern organizations deliver quality software at speed. The company’s latest innovation, the DevOps Lifecycle Manager (DLM), supports Value S...
"Peak 10 is a national cloud data center solutions managed services provider, and part of that is disaster recovery. We see a growing trend in the industry where companies are coming to us looking for assistance in their DR strategy," stated Andrew Cole, Director of Solutions Engineering at Peak 10, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
@GonzalezCarmen has been ranked the Number One Influencer and @ThingsExpo has been named the Number One Brand in the “M2M 2016: Top 100 Influencers and Brands” by Analytic. Onalytica analyzed tweets over the last 6 months mentioning the keywords M2M OR “Machine to Machine.” They then identified the top 100 most influential brands and individuals leading the discussion on Twitter.
SYS-CON Events announced today that Juniper Networks (NYSE: JNPR), an industry leader in automated, scalable and secure networks, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Juniper Networks challenges the status quo with products, solutions and services that transform the economics of networking. The company co-innovates with customers and partners to deliver automated, scalable and secure network...
SYS-CON Events announced today that Super Micro Computer, Inc., a global leader in compute, storage and networking technologies, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Supermicro (NASDAQ: SMCI), the leading innovator in high-performance, high-efficiency server technology, is a premier provider of advanced server Building Block Solutions® for Data Center, Cloud Computing, Enterprise IT, Hadoop/...
Amazon has gradually rolled out parts of its IoT offerings in the last year, but these are just the tip of the iceberg. In addition to optimizing their back-end AWS offerings, Amazon is laying the ground work to be a major force in IoT – especially in the connected home and office. Amazon is extending its reach by building on its dominant Cloud IoT platform, its Dash Button strategy, recently announced Replenishment Services, the Echo/Alexa voice recognition control platform, the 6-7 strategic...
The explosion of new web/cloud/IoT-based applications and the data they generate are transforming our world right before our eyes. In this rush to adopt these new technologies, organizations are often ignoring fundamental questions concerning who owns the data and failing to ask for permission to conduct invasive surveillance of their customers. Organizations that are not transparent about how their systems gather data telemetry without offering shared data ownership risk product rejection, regu...
Judith Hurwitz is president and CEO of Hurwitz & Associates, a Needham, Mass., research and consulting firm focused on emerging technology, including big data, cognitive computing and governance. She is co-author of the book Cognitive Computing and Big Data Analytics, published in 2015. Her Cloud Expo session, "What Is the Business Imperative for Cognitive Computing?" is scheduled for Wednesday, June 8, at 8:40 a.m. In it, she puts cognitive computing into perspective with its value to the busin...
In his keynote at 19th Cloud Expo, Sheng Liang, co-founder and CEO of Rancher Labs, discussed the technological advances and new business opportunities created by the rapid adoption of containers. With the success of Amazon Web Services (AWS) and various open source technologies used to build private clouds, cloud computing has become an essential component of IT strategy. However, users continue to face challenges in implementing clouds, as older technologies evolve and newer ones like Docker c...
In his general session at 19th Cloud Expo, Manish Dixit, VP of Product and Engineering at Dice, discussed how Dice leverages data insights and tools to help both tech professionals and recruiters better understand how skills relate to each other and which skills are in high demand using interactive visualizations and salary indicator tools to maximize earning potential. Manish Dixit is VP of Product and Engineering at Dice. As the leader of the Product, Engineering and Data Sciences team at D...
SYS-CON Events announced today that T-Mobile will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. As America's Un-carrier, T-Mobile US, Inc., is redefining the way consumers and businesses buy wireless services through leading product and service innovation. The Company's advanced nationwide 4G LTE network delivers outstanding wireless experiences to 67.4 million customers who are unwilling to compromise on ...
Bert Loomis was a visionary. This general session will highlight how Bert Loomis and people like him inspire us to build great things with small inventions. In their general session at 19th Cloud Expo, Harold Hannon, Architect at IBM Bluemix, and Michael O'Neill, Strategic Business Development at Nvidia, discussed the accelerating pace of AI development and how IBM Cloud and NVIDIA are partnering to bring AI capabilities to "every day," on-demand. They also reviewed two "free infrastructure" pr...
With billions of sensors deployed worldwide, the amount of machine-generated data will soon exceed what our networks can handle. But consumers and businesses will expect seamless experiences and real-time responsiveness. What does this mean for IoT devices and the infrastructure that supports them? More of the data will need to be handled at - or closer to - the devices themselves.
New competitors, disruptive technologies, and growing expectations are pushing every business to both adopt and deliver new digital services. This ‘Digital Transformation’ demands rapid delivery and continuous iteration of new competitive services via multiple channels, which in turn demands new service delivery techniques – including DevOps. In this power panel at @DevOpsSummit 20th Cloud Expo, moderated by DevOps Conference Co-Chair Andi Mann, panelists will examine how DevOps helps to meet th...
NHK, Japan Broadcasting, will feature the upcoming @ThingsExpo Silicon Valley in a special 'Internet of Things' and smart technology documentary that will be filmed on the expo floor between November 3 to 5, 2015, in Santa Clara. NHK is the sole public TV network in Japan equivalent to the BBC in the UK and the largest in Asia with many award-winning science and technology programs. Japanese TV is producing a documentary about IoT and Smart technology and will be covering @ThingsExpo Silicon Val...