The Kubernetes ecosystem is a candy store
Database in Kubernetes: Is that a good idea?
Welcome to the 53rd edition of CloudPro! Today, we’ll talk about:
⭐Masterclass:
🔍Secret Knowledge:
⚡Techwave:
At Kubernetes 10th Anniversary in Mountain View: History Remembered
Kali Linux 2024.2 Release (t64, GNOME 46 & Community Packages)
🛠️HackHub: Best Tools for the Cloud
Cheers,
Editor-in-Chief
Forwarded this Email? Signup Here
⭐MasterClass: Tutorials & Guides
⭐Kubernetes webhook admission controller
When providing Kubernetes as a cloud service, managing authentication and authorization is crucial. Kubernetes API server handles authentication through multiple methods. To implement custom authorization logic, you can use Kubernetes admission controllers. These controllers intercept requests to the Kubernetes API server and direct them to a custom service running in a pod, which checks user access levels.
To implement this, you can use Kind to create a Kubernetes cluster on your laptop. Develop an authorization service in Golang, secure it with OpenSSL, and push its image to a Docker registry. The authorization service intercepts and evaluates requests, allowing or denying them based on custom logic.
Secure communication between the API server and the webhook requires a TLS certificate, which you generate using OpenSSL and store as a Kubernetes secret. Deploy your service and configure the Kubernetes API to use it for validating incoming requests. Test the setup by creating a pod and observing the logs to confirm that unauthorized requests are blocked by your custom admission controller.
⭐The Kubernetes ecosystem is a candy store
Think of the Kubernetes ecosystem as a candy store. Just like a candy store has a wide variety of sweets to choose from, the Kubernetes ecosystem offers a vast array of tools and solutions for developers. These tools are designed to make working with Kubernetes easier and more efficient, no matter where you're using it—on public clouds, private clouds, or even at the edge.
Because Kubernetes is so flexible and widely used, many developers and vendors create tools that work with it. These tools can often be used in different environments without needing to be changed for each one, unlike tools made for specific cloud providers. This creates a big market for Kubernetes tools, making it easier and cheaper to develop them.
Open source contributions have been a big part of this ecosystem. Many tools are open source, allowing developers to share their solutions with the community. This sharing makes Kubernetes more useful and popular, attracting even more users and developers.
⭐Why is Building Pipelines Different from Software Development?
Building CI/CD pipelines differs from traditional software development primarily due to their focus on automation and reproducibility. Unlike typical software projects that may involve complex business logic and user interfaces, CI/CD pipelines are primarily about automating the process of building, testing, and deploying software. They often rely on configuration files (commonly written in YAML) that instruct pipeline runners on how to execute each step. These steps include checking out code, setting up environments, running tests, and deploying artifacts.
One major challenge with CI/CD pipelines is ensuring that they run consistently across different environments. Commands that work perfectly on a developer's local machine may fail in a CI/CD runner due to differences in the environment. This lack of reproducibility makes debugging pipelines difficult, often requiring developers to rely on logging and trial-and-error. To address these issues, some modern tools and frameworks allow pipelines to be treated more like traditional code, using familiar programming languages and testing frameworks, which can significantly improve debugging and maintainability.
⭐How to handle long-running tasks in AWS AppSync
AWS AppSync's new feature allows Lambda functions to be invoked asynchronously. Previously, all requests in AppSync were processed synchronously, which could be limiting for tasks requiring more than 30 seconds to complete. Before this update, developers had to offload long-running tasks to another Lambda function via SQS or direct invocation, returning a generic response initially and managing the final response through subscriptions.
Now, by specifying the "Event" invocation type, tasks can be executed asynchronously directly from the resolver, eliminating the need for intermediate steps. This simplifies the handling of long-running tasks, although payload size constraints remain—256KB for the asynchronous request and 5MB for the synchronous response. Subscriptions can deliver the final result, provided it doesn't exceed 240KB. This enhancement makes handling complex, long-duration tasks more straightforward within AppSync, while still adhering to AWS payload limits.
⭐Automate PostgreSQL Partitioning
In this article, we integrate monitoring and alerting within the database using pg_cron and AWS Lambda, along with Amazon SNS for notifications.
The solution involves scheduling a maintenance job with pg_cron and using AWS Lambda to handle failures. The maintenance job checks for failures in the cron.job_run_details table and triggers a Lambda function if any issues are found. This function then sends alerts via Amazon SNS. By integrating these tools, we can effectively monitor and manage PostgreSQL partition maintenance directly from the database, ensuring timely alerts and proactive maintenance.
🔍Secret Knowledge: Learning Resources
🔍How to transparently use a proxy with any application (Docker) using Iptables and RedSocks
Using internet proxies is common for imposing restrictions on how applications or machines connect to the internet, often for security purposes. Many applications support proxies for protocols like HTTP, HTTPS, and FTP by setting environment variables. However, there are exceptions where this method doesn't work, such as with the Scrapy framework's FTP support. Instead of modifying Scrapy, a generic solution involves using IPTables and RedSocks to transparently forward FTP traffic through a proxy.
To implement this solution, you can create a Docker container that uses IPTables to redirect outgoing traffic to a local RedSocks service, which then forwards it to the proxy. This setup allows applications to use the proxy without being explicitly configured for it. RedSocks handles the proxy connection transparently, overcoming limitations with protocols like HTTPS and FTP that IPTables alone cannot manage. Using IPTables in conjunction with RedSocks provides a flexible and generic way to enforce proxy usage for various applications.
🔍Database in Kubernetes: Is that a good idea?
Running databases in Kubernetes (K8s) is a contentious topic due to the inherent differences between managing stateless and stateful applications. Kubernetes excels at managing stateless services but struggles with the demands of stateful services like databases, which require high reliability, security, and performance. Although Kubernetes offers features like StatefulSet and Persistent Volumes to support databases, these are often insufficient for the stringent requirements of production-level databases.
Placing databases in Kubernetes can lead to complications and performance issues, as network storage is not as reliable or fast as local storage. Using high-performance local storage ties the database to specific nodes, defeating Kubernetes' flexibility. This creates a lose-lose scenario: Kubernetes loses its efficiency in managing stateless applications, while databases suffer from reduced reliability and performance. Therefore, for most users, traditional deployment methods or managed database services like RDS remain more practical and reliable solutions for handling critical database operations.
🔍Why serverless is trending again
Serverless computing has seen a resurgence due to several key developments. Initially, the term "serverless" was misunderstood, leading to confusion about its actual meaning. Over time, a solid definition emerged, clarifying that serverless involves developers writing functions to handle individual events without needing to manage the underlying server infrastructure. This approach has significantly increased productivity, as developers can focus on code rather than server management. Additionally, success stories from companies that have drastically reduced maintenance costs and accelerated application deployment have fueled renewed interest.
The major breakthrough driving the current serverless trend is the advent of WebAssembly. Traditional serverless frameworks suffered from issues like slow cold starts and cloud vendor lock-in. WebAssembly addresses these problems by providing a high-performance, portable runtime that can run anywhere, from small devices to large servers. With tools like Fermyon's Spin and SpinKube, serverless functions can now be seamlessly integrated into Kubernetes environments, offering unprecedented speed and flexibility. This new compute layer has effectively resolved the limitations of earlier serverless technologies, making serverless a more viable and attractive option for modern application development.
HTTP logs are vital for web development and server management, providing insights into traffic, errors, and application issues. However, standard HTTP logs often miss crucial details like request and response bodies, making debugging difficult. Without full visibility, understanding application behavior and ensuring robust security becomes challenging.
Our article introduces Gleam and Quickwit as solutions to these limitations. Gleam, a language compiling to Erlang, allows us to build an efficient proxy that captures comprehensive HTTP logs, including request and response bodies and headers. Quickwit then offers a scalable and efficient storage solution for these detailed logs, ensuring you have all the information needed for effective debugging and monitoring.
🔍Easy AWS permissions for your EKS workloads
EKS Pod Identity simplifies granting AWS permissions to Kubernetes applications on Amazon EKS. Traditionally, configuring these permissions could be complex and time-consuming. With EKS Pod Identity, you can directly associate an IAM role with a Kubernetes service account using EKS APIs. This integration allows your applications to securely access AWS services like S3 or DynamoDB by automatically obtaining temporary AWS credentials, streamlining the setup process.
By leveraging EKS Pod Identity, Kubernetes cluster administrators can manage AWS permissions more efficiently. This feature eliminates the need for manual IAM role configuration per application or cluster, offering a more straightforward approach tailored for EKS environments. It enhances security and reduces administrative overhead, making it easier to scale applications that require seamless access to AWS resources within Kubernetes deployments on EKS.
⚡ TechWave: Cloud News & Analysis
⚡The Future of Electric Eel and Apps
TrueNAS is evolving with a focus on enhancing its application ecosystem, particularly with the upcoming Electric Eel release. Recognizing community feedback for simpler and more flexible application deployment methods, TrueNAS will introduce native Docker and Docker Compose support. This move aims to streamline the management and portability of apps, aligning more closely with industry standards and user preferences.
Electric Eel, scheduled for release in Q4 this year, marks a significant update for TrueNAS SCALE. It promises a range of enhancements including OpenZFS improvements, web UI upgrades, integrated cloud backups, and enhanced security features. The introduction of Docker Compose support will allow users to easily migrate existing apps and deploy new ones directly from TrueNAS catalogs or through standard YAML configuration files.
⚡At Kubernetes 10th Anniversary in Mountain View: History Remembered
The 10th anniversary celebration of Kubernetes was a nostalgic and heartfelt event, marking a decade of innovation and community collaboration. Held at Google's Mountain View campus, the event gathered key figures who recounted the journey from Kubernetes' inception to its current status as a cornerstone of modern computing infrastructure.
Eric Brewer, reflecting emotionally on the journey, highlighted how Kubernetes fulfilled a vision spanning decades — from early conceptualizations of scalable network services to the cloud-native ecosystem we see today. The community's role was pivotal, with Brewer emphasizing that Kubernetes' success was not just about technology but about the collective effort of developers worldwide.
The event also honored pivotal moments and individuals, from the scrappy beginnings of the original team to pivotal contributions like Docker's influence on container technology. It underscored how community, collaboration, and perseverance were fundamental in shaping Kubernetes into what it is today — a testament to the power of open source and shared innovation.
⚡Kali Linux 2024.2 Release (t64, GNOME 46 & Community Packages)
Kali Linux 2024.2, the latest release, arrives slightly delayed but packed with significant updates. The focus has been on under-the-hood improvements and community contributions, resulting in new packages, bug fixes, and enhanced stability. Key changes include the introduction of the t64 transition for 32-bit platforms to address the Year 2038 problem, ensuring future compatibility for ARM architectures like Raspberry Pi.
Desktop enhancements feature GNOME 46 and improved Xfce support, offering a more polished user experience. Additionally, Kali introduces 17 new tools for network reconnaissance, exploit searching, and more, reflecting ongoing community-driven innovation. Users are encouraged to update using `apt full-upgrade` to seamlessly transition to these latest features and ensure optimal system performance.
⚡Introducing Amazon EMR on EKS with Apache Flink
Amazon recently launched Apache Flink for Amazon EMR on Amazon EKS, combining the scalability of Apache Flink with the flexibility and management capabilities of Kubernetes. Apache Flink is known for its real-time data processing capabilities, making it ideal for streaming workloads. By integrating with EMR on EKS, customers benefit from a managed environment that handles infrastructure complexities, such as installation, scaling, and security. This setup allows data engineers and scientists to focus on application development while leveraging Kubernetes' robust features like multi-tenancy and resource management.
EMR on EKS enhances operational efficiency by automating tasks like configuration management and scaling, optimizing both performance and cost. It also supports integration with existing AWS services like S3 and IAM for enhanced data management and access control. For users, this means easier deployment, better resource utilization, and the ability to scale applications seamlessly based on workload demands.
🛠️HackHub: Best Tools for Cloud
🛠️typesense/showcase-linux-commits-search
The Linux Commit History Search demo uses Typesense to search 1 million commit messages from the Linux kernel repository, providing a fast, typo-tolerant search experience.
🛠️trek10inc/cfn-changeset-viewer
The `cfn-changeset-viewer` CLI helps you view changes in an AWS CloudFormation ChangeSet in a more readable format. It shows detailed differences between resource configurations, including nested stacks. Changes are indicated by icons (+ for add, ~ for modify, - for remove, ↓ for import, ? for dynamic changes). It also warns about changes that might require resource replacement. Example usage includes specifying the ChangeSet ARN or stack name to view changes.
Beta9 is a platform that simplifies running serverless functions on cloud GPUs. It supports:
- Running Python functions on thousands of GPUs, scaling resources automatically.
- Flexibility to use public cloud or private hardware.
- Optimized for AI with fast model deployments and cold starts.
Use-cases include serverless inference endpoints and parallelized workloads across multiple containers. Beta9 also supports enqueuing async jobs and is designed for quick deployment using custom container technologies and scheduling engines.
🛠️dayjournal/qgis-amazonlocationservice-plugin
The Amazon Location Service Plugin for QGIS integrates Amazon's Location Service into QGIS, allowing you to use its features directly within the QGIS environment. You can manage resources like maps, geocoding, and routing using AWS CDK or CloudFormation to configure and deploy these services. After installing the plugin, you set up resource names and API keys in the plugin's configuration menu. Functions include displaying maps, geocoding places, and calculating routes, all accessible through QGIS's menus once configured.
🛠️LumoSQL
LumoSQL is an enhanced version of SQLite, not a fork, designed to add features like security, privacy, and performance improvements. It integrates with different key-value store engines such as LMDB and Berkeley DB alongside SQLite's default Btree storage. LumoSQL supports modern encryption and per-row checksums for data integrity. It's maintained under the MIT license and supported by the NLNet Foundation, running on various architectures and operating systems including x86, ARM-32, and RISC-V with Linux and BSD distributions.
📢 If your company is interested in reaching an audience of developers and, technical professionals, and decision makers, you may want to advertise with us.
If you have any comments or feedback, just reply back to this email.
Thanks for reading and have a great day!