Cloudkeeper
Housekeeping for Clouds!
Table of contents
Overview
Cloudkeeper is “housekeeping for clouds” - find leaky resources, manage quota limits, detect drift and clean up.
Cloudkeeper indexes resources, captures dependencies and maps out your infrastructure in a graph so that it’s understandable for a human. The graph contains metrics for each resource. Developers and SREs can search the graph with a query language, and create alerting and clean-up workflows. Metrics can be aggregated and exported to a time series database like Prometheus.
If you ever
- had a standstill in your CI pipeline because a broken job leaked cloud resources which triggered a quota limit
- wanted to find all expired certificate
- had to change the tags of thousands of EC2 instances at once
- needed to delete all unused EBS volumes that had no I/O in the past month
- wished for a god view that lets you explore all cloud usage across all clouds
- reported the cost of a project across different accounts or even across clouds
- cleaned up orphaned load balancers that had no active backends
- wanted to automate any of the above
Those are the kinds of situations Cloudkeeper was built for.
Currently it can collect AWS, Google Cloud, VMWare Vsphere, OneLogin and Slack. The later can also be used for notification of resource cleanups. If the cloud you are using is not listed it is easy to write your own collectors. An example can be found here.
Cloudkeeper consists of multiple components described in the component list below
The latest Docker image is: ghcr.io/someengineering/cloudkeeper:2.0.0a6
The latest Documentation can be found on https://docs.some.engineering
Docker based quick start
In this quick start guide, we’re showing you three things, how to:
1. install Cloudkeeper for AWS with docker
2. use the Cloudkeeper CLI to run your first collect process
3. query the results of the collect process
The docker set-up takes 2-5 minutes. The duration of the first collect process depends on the size of your environment - usually 5-10 minutes.
Examples and data in this guide are based on a small AWS Cloud9 environment.
To start exploring you need AWS credentials and a working Docker environment with access to AWS APIs.
We assume you are familiar with basic Docker operations and how to operate a Linux shell.
Continue reading the Quick Start Guide
--> https://docs.some.engineering/getting_started/quick_start.html
Cloning this repository
This Git repo uses Git Large File Storage (LFS).
If you would like to work on the UI ckui, before cloning the repo make sure to have git-lfs installed!
One time setup:
$ git clone https://github.com/someengineering/cloudkeeper.git
$ cd cloudkeeper/
$ git lfs install # installs git-lfs hooks
Once set up you can interact with the repo like any other Git project. All large UI assets will be retrived from Github's LFS servers.
If you have no need for the UI assets git-lfs is optional.
Component list
ckcorethe platform maintaining the MultiDiGraph.ckshthe Cloudkeeper shell to interact with the core.ckuia UI prototype that can load ckcore exported data but has no backend connection yet.ckworkerprovides workers that load plugins to perform collect and cleanup operations.ckmetricsis a Prometheus exporter.pluginsare a collection of worker plugins like AWS
Contact
If you have any questions feel free to join our Discord or open a GitHub issue.
License
Copyright 2021 Some Engineering Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
![[plugins/aws][feat] Collect tags and enable tagging methods for Route53 Hosted Zones](https://avatars.githubusercontent.com/u/66877958?v=4)
![[plugins/k8s][feat] Add argument to collect all contexts in config file](https://avatars.githubusercontent.com/u/19917125?v=4)
![[resotocore] `Update process interrupted` with large graph updates](https://avatars.githubusercontent.com/u/100136?v=4)
![[resoto][feat] CDK construct](https://avatars.githubusercontent.com/u/1043441?v=4)
![[resotocore][feat] Allow configuration of workflow schedules](https://avatars.githubusercontent.com/u/330485?v=4)
![[resotocore] Help text is hard to read in Ubuntu Terminal](https://avatars.githubusercontent.com/u/5024311?v=4)
37 Aug 13, 2022
2 Nov 21, 2021
9 Feb 20, 2022
837 Jan 07, 2023
4 Feb 19, 2022
18 Oct 21, 2022
7 Dec 14, 2022
2 Dec 23, 2021
2.6k Dec 27, 2022
18 Dec 21, 2022
2.6k Jan 06, 2023
1 Nov 02, 2021
8 Oct 21, 2022
2 Aug 30, 2022
1 Dec 04, 2021
1 Nov 18, 2021
15 Oct 08, 2022
28 Dec 16, 2022
1 Jan 05, 2022
10 Mar 05, 2022