I’ve been experimenting with Hermes Agent, an open-source AI agent that can execute commands, browse the web, manage scheduled tasks and interact with external services.

The idea is pretty simple: have an assistant running 24/7 that can handle some of the boring administrative tasks I would otherwise have to remember to do myself.

The problem is that an AI agent with shell access and the ability to execute arbitrary code isn’t something I’d be particularly comfortable running alongside my personal files, banking credentials or other services.

Even if you trust the model, you can’t necessarily trust everything it reads. A malicious webpage or email could contain instructions intended to convince the agent to do something it shouldn’t.

So instead of trying to make Hermes perfectly secure, I decided to build the infrastructure around the assumption that the agent could eventually be compromised.

The Threat Model

My main concern is prompt injection: untrusted content persuading Hermes to use its legitimate tools in ways I didn’t intend.

Imagine asking Hermes to research something online. One of the pages it visits contains instructions telling it to read a local configuration file and send its contents to an external endpoint.

That doesn’t require a container escape. If Hermes can read the file and reach the destination, its normal shell or web tools may be enough.

A software flaw could also give an attacker direct control of the Hermes process. Because the agent already has shell access, both cases need the same basic containment: limits on the files, services and network connections available to that process. I want those limits to hold even when the model makes the wrong decision.

My approach is based on three principles:

  • Assume anything the agent reads could be malicious.
  • Give it the minimum permissions necessary.
  • Enforce sensitive operations and business rules in separate services.

The objective isn’t to prevent the agent from making mistakes. It’s to limit what those mistakes can actually accomplish.

Isolating Hermes on a VPS

I started with a dedicated OVH VPS running Ubuntu, with 4 vCPUs and 8 GB of RAM.

There’s nothing especially interesting about the hardware. It gives the agent its own machine, away from my personal files and home network.

Hermes runs inside Docker. These are the resource and privilege settings from its Compose service:

mem_limit: 5g
cpus: 3.0
pids_limit: 512

security_opt:
  - no-new-privileges:true

The CPU, memory and process limits leave room for the OS and supporting services. Disk usage and outbound traffic need separate limits.

The no-new-privileges flag blocks privilege gains when executing programs, such as elevation through setuid executables. It does not remove privileges the process already has, so choosing a non-root container user and restricting capabilities are separate settings.

I don’t mount the Docker socket or my SSH credentials into the container, and it runs without privileged mode or host networking.

On the host, I use a separate administrator account for Docker management. The Hermes runtime account isn’t a member of the Docker group, which is important because access to the Docker socket is effectively equivalent to root access.

I use a versioned Docker image rather than blindly pulling latest. Tags can be reassigned, so pinning the exact image contents requires a digest. Updates still need to be deliberate so that version pinning doesn’t leave known vulnerabilities in place.

These controls help contain the process. They cannot decide whether an otherwise permitted API call matches what I actually wanted.

Narrow APIs Instead of Upstream Credentials

I wanted Hermes to perform useful actions such as creating invoices, checking calendar availability and sending notifications. But I didn’t want to give it the underlying credentials for those systems.

For example, I built a small Go service for generating Mexican electronic invoices (CFDIs).

Generating these invoices involves tax information, signing certificates and an external invoicing provider. Giving an LLM direct access to all of that seemed unnecessary.

Instead, the invoicing service owns the credentials and exposes a very small API.

Rather than letting Hermes construct an arbitrary invoice, it can request something like:

{
  "client": "client_a",
  "period": "2026-10"
}

The service validates the request, looks up the configured client, applies the appropriate invoice rules and handles the external API calls.

Hermes doesn’t need to know the client’s tax information, the signing certificate password or the invoicing provider’s API key.

It also doesn’t need to receive the generated PDF or XML. Those can be delivered directly to my email by the trusted service.

That keeps the underlying credentials and documents out of Hermes’ normal workflow. Triggering an invoice does not need to grant access to either the signing key or the resulting files.

Hermes still has authority to request an invoice, though. Authenticating a request establishes which credential was used; it cannot establish that I wanted that invoice. If a token is readable by Hermes’ shell, it may also be copied by a compromised agent.

For this boundary to hold, the service must enforce its own rules: allowed clients and periods, fixed invoice details, limits on request frequency and idempotency tied to the intended invoice so that retries do not issue duplicates. Actions outside those rules need to be rejected or approved through a mechanism Hermes cannot approve for itself. Recording issued invoices also makes misuse easier to investigate.

Even with those checks, a malicious prompt might trigger an unwanted invoice within the permitted scope. Keeping signing credentials private limits one kind of damage; it does not make invoice creation harmless.

Email Is a Bigger Security Problem Than It Looks

Giving an agent access to your primary inbox is convenient, but it also creates a fairly large attack surface. An inbox contains far more than ordinary correspondence: account recovery links, invoices, financial information, personal documents and potentially enough information to impersonate you.

There’s also the obvious prompt injection problem. Emails are untrusted content, and an agent that reads them shouldn’t treat their contents as instructions.

I decided not to connect Hermes to my main Outlook account at all.

Instead, I use separate mailboxes for different purposes. A dedicated Hermes mailbox can receive low-risk information, while more sensitive material goes through restricted ingestion services.

As an example of restricted ingestion, consider a bank statement. A process with sole access to a statement mailbox could extract selected transaction fields, remove account numbers and pass a structured summary to Hermes.

That would give Hermes enough information to analyse spending without exposing the original statement or mailbox. It would still reveal spending information, and retained merchant names or payment references would remain untrusted text. Redaction reduces disclosure; it does not eliminate confidentiality or prompt injection risks.

For sending, I built a small mail service that Hermes can call to send plain-text notifications to my personal email address. It doesn’t expose general-purpose SMTP functionality or allow Hermes to choose arbitrary recipients.

Conceptually, the API is closer to:

{
  "subject": "Weekly Summary",
  "body": "Three items need attention."
}

The recipient and SMTP credentials are configured inside the service. Its API exposes no operation to retrieve the password or change the destination, so Hermes cannot use this API to send mail to an arbitrary address.

The content remains agent output, even when it arrives through my mail service. A compromised Hermes could send misleading messages or flood my inbox. Limits on message frequency and size reduce that abuse; restricting recipients does not make the messages trustworthy.

For confidential documents, such as invoices, I prefer the originating trusted service to email them directly. That way the documents never need to pass through Hermes in the first place.

Service and Network Boundaries

The services also run in separate containers with their own persistent storage.

Hermes can communicate with the APIs it needs through private Docker networks, but it doesn’t get their volumes mounted into its filesystem.

The intended data flow looks like this:

Hermes container ---> web and model providers (outbound)
       |
       | private container connections
       +---> CFDI API ------> Enlace API
       +---> Calendar API --> calendar data
       +---> Mail API ------> email provider

Each trusted service manages its own authentication and decides what information it will return. Those services are part of the security boundary: their code must treat every request from Hermes as untrusted.

Keeping internal API ports unpublished avoids exposing them through the host’s public interfaces. Containers on the same Docker network can still communicate with each other, so the connections in this diagram should not be read as a set of firewall rules.

Docker networking also allows outbound connections by default. Restricting destinations requires an explicit policy enforced outside Hermes, such as firewall rules or a controlled proxy. Browsing arbitrary websites leaves a broad path for data to leave, even if the internal services are well isolated.

I therefore treat any information Hermes can read as potentially disclosable through its allowed connections. Returning less data from trusted services reduces that exposure.

These boundaries also assume the VPS host remains trusted. Someone with root access to the host can read the containers’ volumes and secrets.

My SAT e.firma and banking credentials stay off the VPS entirely. The credentials held by the invoicing and mail services are still valuable secrets, and a host compromise would expose them.

Signal and Notifications

I wanted to interact with Hermes primarily through messaging.

For this I chose Signal, using a dedicated Signal account and a separate signal-cli container.

The Signal daemon isn’t exposed publicly. Hermes communicates with it over a private Docker network.

Command access also needs Hermes’ Signal sender allowlist or approved DM pairing, with group access disabled or explicitly restricted. A dedicated account alone does not authorise its senders.

I also integrated ntfy for simple push notifications.

The distinction is intentional:

  • Signal: conversations, commands and interactive decisions.
  • ntfy: push notifications and incoming events about jobs, reminders and failures.
  • Email: delivery of confidential documents from trusted services.

I keep ntfy’s inbound and outbound topics separate and use a long, randomly generated topic name. For an unprotected topic, anyone who learns its name can read or publish messages; ntfy supports access controls to restrict those permissions. The topic name therefore needs to be handled as a secret when it is the only protection.

Separating the topics keeps traffic apart, but any inbound event that can trigger work still needs an authenticated sender and a narrowly defined set of allowed actions. Notification messages should contain only the status details needed to be useful.

Remote Access Without a Public Dashboard

One thing I particularly wanted was access to Hermes Desktop without exposing another public service.

Hermes Desktop can connect to the remote dashboard backend. I publish that backend’s port only on the VPS loopback interface:

ports:
  - "127.0.0.1:9119:9119"

That is the host-side binding. The dashboard must listen on the container’s network interface for the published port to work, as described in the Hermes Docker setup.

I then connect from Windows PowerShell using an SSH tunnel:

ssh -N -L 127.0.0.1:19119:127.0.0.1:9119 ubuntu@YOUR_VPS_IP

Hermes Desktop connects to http://127.0.0.1:19119.

The dashboard also requires authentication, so the SSH tunnel isn’t the only protection.

This gives me access to the remote instance, including its sessions, memory and scheduled tasks, without exposing its dashboard on a public interface.

What About the LLM Provider?

My Hermes instance uses OpenRouter for model access.

I created a dedicated API key with a small spending limit and restricted the models available to the agent.

The spending limit reduces exposure to unexpected API costs. For model restrictions to survive a compromised agent, they need to be enforced by OpenRouter on the key or account; settings in a file Hermes can edit are not sufficient. OpenRouter provides model allowlists for this purpose.

Using a remote model means sending request content outside the VPS during normal operation. Hosting Hermes myself does not keep those requests local. That is another reason for sensitive services to return only the information needed for the task.

The same rule applies to the API key itself: a key available to Hermes’ process may be stolen, and its provider-enforced limits must remain useful even if it is used elsewhere.

Final Thoughts

This takes more work than connecting every account directly. It gives me a clearer way to review each new capability: what can Hermes request, what information can it receive, and what happens if it makes the request maliciously?

If a malicious webpage tricks Hermes, it could still request an unwanted invoice, send misleading notifications or leak information it can read over an allowed network path. The design reduces its access to powerful credentials and private documents; the remaining capabilities still need their own limits and monitoring.

The agent can be intelligent without needing to be trusted with everything.