State of AI Engineering

What The State of AI Engineering Report Reveals About AI in Production

AI engineering has moved past experimentation, but are your observability practices keeping up? Datadog analyzed LLM telemetry from 1,000+ customers to reveal what's actually happening at scale: model adoption shifts, hidden token costs, and what the rise of agentic frameworks means for reliability. Download the report to benchmark your AI stack against production reality.

IBM + ReadHat save OpenSource! - IBM and Red Hat have fixed more than 400 previously unknown vulnerabilities in open-source Java libraries through their Lightwell initiative. The project uses AI-assisted tools and human review to identify security flaws and develop tested fixes for existing software. Read more.

npm package that can Wipe your machine - A compromised Tensorlake npm package steals developer credentials and spreads malicious code to other packages and GitHub repositories. The dangerous twist? It installs a background monitor that can delete your home directory if you revoke the stolen GitHub token. The attack targets developer machines and can also expose secrets used by AI coding tools. Read more.

Coudflare marries Deno - Deno’s team is joining Cloudflare to make it easier for developers to self-host applications using Cloudflare Workers and Durable Objects. Cloudflare plans to combine Deno’s work with its own workerd runtime and celld infrastructure, while Deno Deploy is scheduled to shut down after a six-month transition period. Deno will remain open source, but its team plans to stop developing the runtime after one year of security and bug-fix updates. Read more.

The Evolution of Automation at Google

Why Automation Matters

Automation helps engineers manage systems faster, more reliably, and with fewer mistakes. In its discussion of automation, Google explains that repeating tasks manually can lead to errors because people may perform the same task differently each time. Automation makes these tasks more consistent and allows engineers to focus on more important work. It can also detect common problems and fix them quickly, reducing downtime and improving system reliability. However, automation is not a perfect solution. If it is poorly designed, it can create bigger problems instead of solving them.

How Google Improved Automation

Google's experience shows that automation works best when it grows beyond simple scripts. In the beginning, engineers used scripts to manage machines, restart services, and perform routine tasks. As Google's infrastructure grew, these scripts became harder to maintain and could fail when systems changed. Google gradually moved towards building platforms and systems that could manage themselves. One example was Decider, an automated system created in 2009 to handle MySQL database failovers. It could complete failovers in less than 30 seconds in 95% of cases. This reduced the team's time spent on routine maintenance by 95%, allowing engineers to focus on improving other parts of the system.

When Automation Goes Wrong

Automation can also cause serious failures if it does not include proper safety checks. Google once experienced an incident in which a machine decommissioning process accidentally triggered disk erasure across almost all its machines in third-party data centers. The mistake happened because the system treated an empty list of machines as a command to erase everything. Although Google's own data centers continued serving users, engineers spent days restoring affected machines and weeks improving safety checks. This incident shows why automation needs clear limits, careful testing, and ways to prevent unexpected actions.

From Automation to Autonomous Systems

Google's larger goal is to build autonomous systems that can detect problems and recover without human intervention. Instead of relying on separate scripts, engineers can design systems with built-in recovery, clear APIs, and fewer manual steps. The key lesson is that automation should be part of system design from the beginning. Good automation saves time, but well-designed autonomous systems go further by improving reliability as infrastructure grows. Human oversight still matters because engineers need to understand how systems work and be ready to respond when automation fails.

Read the official Google Blog.

OpenAI kills Mathematicians - OpenAI has released mathematical results generated by an internal AI model, including work on previously unsolved problems across different areas of mathematics. It has published research details, reasoning summaries, and computer-checkable Lean proof formalizations on GitHub to help researchers examine the results. Read more.

Anthropic defends the Internet - Anthropic is launching free, regular AI-powered security scans for open-source projects and a new program that helps protect power grids, water systems, transport networks, and government systems. Its OSS Scanner reports vulnerabilities with explanations and suggested fixes, while the infrastructure program provides advanced Claude models, security research, and engineering support. Read more.

Buzz of the Week!

Rendezvous Hashing

Rendezvous hashing, also called highest random weight hashing, is a technique for deciding which server should store or handle a particular key in a distributed system. For each key, the system calculates a score for every available server using the key and server identity, then assigns the key to the server with the highest score. When a server joins or leaves, only the keys whose winning server changes need to move, rather than redistributing everything. This makes it useful for distributed caches, sharded databases, and request routing across changing server pools. Unlike consistent hashing, it does not require a ring structure, and its scoring approach can make server selection straightforward to implement.

Things that launched. Things that went viral. Things you'll pretend to try.

Unison

Unison Unison is a file-synchronization tool for POSIX-compliant systems (e.g. *BSD, GNU/Linux, macOS) and Windows.

gasket

gasket is a framework for building Node.js applications with a focus on production conventions and extensibility.

Mizu

Mizu is a terminal-based API traffic viewer that captures and displays HTTP traffic from Kubernetes environments.

Build Braincells, Not Just Features

This weekend’s read: Decisions

This week’s watch: How did Humans invent Computer?

Meanwhile…