---
title: GlideinWMS Workspaces for Dev & Testing
url: https://www.emergentmind.com/topics/glideinwms-workspaces
type: topic
---

# GlideinWMS Workspaces for Dev & Testing

Searching arXiv for the specified paper to ground the article in the cited source.
GlideinWMS Workspaces are development- and test-oriented “system-like containers” for GlideinWMS, a pilot- and pressure-based Workload Management System that provisions elastic HTCondor “User Pools” for scientific collaborations, including CMS and DUNE [2507.21464]. Unlike microservice-style production containers that typically run a single process, these workspaces behave like small Linux hosts: each image runs multiple services and includes tools needed to configure, operate, and inspect a full GWMS node. The approach collapses a minimal multi-node testbed into a compact, containerized setup that developers can run locally, including offline, rapidly reconfigure, and use for integration testing and teaching [2507.21464].

## 1. Concept and motivation

GlideinWMS provisions resources through a division of responsibilities between a Factory and a Frontend. A Factory submits pilot jobs, called “Glideins,” to Compute Entrypoints, while a Frontend, or other clients such as HEPCloud’s Decision Engine, requests pilots based on user demand and policies [2507.21464]. In this model, pilots join an HTCondor User Pool and execute user jobs after the resource has been provisioned.

The motivation for Workspaces is stated in operational terms. Traditional GWMS development and test required multiple virtual machines in FermiCloud to cover simultaneous releases and OS variants, plus the operational overhead of certificates, puppet, firewall rules, and shared resource contention [2507.21464]. The workspace approach replaces that VM-based arrangement with containers that emulate VM-like behavior closely enough to support simultaneous services, source-tree mounting, state inspection, and fast iteration.

The designation “system-like containers” is central. GWMS nodes are service-rich, with components such as HTCondor daemons, HTCondor-CE, the GWMS Factory, the Frontend, and the Access Point/Central Manager. Developers therefore need to run and coordinate several daemons together, inspect their state, mount source trees, and iterate quickly [2507.21464]. Emulating a small Linux host inside each container, rather than decomposing everything into single-process containers, lowers friction for development, integration testing, and teaching.

A common misconception is to treat these containers as production deployment units. The source distinguishes them from specialized production containers and places them in a development, test, and teaching role. This is reinforced by the use of self-signed host certificates generated at startup and by the emphasis on local and offline workflows rather than production hardening [2507.21464].

## 2. Base system and component interactions

The minimal deployment mirrors a real GWMS setup and uses three containers [2507.21464].

| Container | Components or role | Workspace details |
|---|---|---|
| One-node cluster | HTCondor-CE and HTCondor batch system | Configured as a SciToken-authenticated computing resource; includes scripts; can fake more cores than physically available |
| GlideinWMS Factory | Submits Glideins to CE(s), maintains pressure, monitors pilots, hosts secure mailbox | Installs and configures the Factory; downloads HTCondor tarballs needed by Glideins; can link the GlideinWMS install to a Git repository |
| Scheduler and Frontend | Runs the GlideinWMS Frontend and acts as HTCondor Access Point and Central Manager | Installs and configures the Frontend and the User Pool; includes scripts to link a Git repository, refresh credentials, and submit a smoke test job |

Docker or Podman Compose is used for orchestration. A `compose.yml` defines the three services and a bridged network with name resolution for a fictitious `glideinwms.org` domain, while shared volumes are mounted across containers to share secrets and code [2507.21464]. A startup script generates self-signed host certificates from an included trusted CA certificate.

The job lifecycle is also defined explicitly. The Frontend observes user job demand and resource status, applies experiment policies, and updates requests to the Factory to achieve prompt job starts without wasting resources. The Factory translates Frontend requests into pilot submissions to Compute Entrypoints, respecting throttles, supported protocols and authentication, and per-site policies, while maintaining requested pressure in queued and running Glideins [2507.21464]. HTCondor-CE and HTCondor receive Glidein submissions and start pilots under the site batch system.

On start, Glideins probe the resource, manage credentials, provide monitoring and audit information, auto-detect resources such as cores, memory, disk, and GPUs, and can install tools including a container runtime and distributed filesystem [2507.21464]. They then join the User Pool on the Frontend node and run user jobs, possibly multiple, in sequence or parallel. HTCondor matches user jobs to pilot `startd`s; pilots can run, finish, and be replaced according to pressure and demand.

This suggests that the three-container arrangement is intended not merely as a mock environment but as a realistic minimal deployment in which production-style control flow remains observable. The paper’s figures, including `fig/gwms_system.png` and `fig/gwms-testbed.png`, are cited as system overviews of these interactions [2507.21464].

## 3. Distinguishing characteristics of the workspace model

The principal distinction from other containers is system-like composition. Each workspace runs multiple services under one image, such as HTCondor-CE plus HTCondor, or Frontend plus AP/CM, rather than decomposing functionality into microservices [2507.21464]. This design is tied to the service-rich character of GWMS nodes and to the need for developers to inspect and reconfigure integrated services rapidly.

Service orchestration is handled by Compose, which coordinates multiple service-rich containers, provides networking, name resolution, shared volumes, and optional port exposure for hybrid tests [2507.21464]. The networking model is a bridged network with internal DNS for `glideinwms.org`, with parameterizable external connectivity for real sites and clouds.

Because `systemd` is not available inside containers, `supervisord` is used plus an emulation of `systemctl` to preserve documented service start and stop commands [2507.21464]. This is operationally significant: the preserved control flow means documented GWMS service commands still apply inside the workspace environment. A plausible implication is that the workspaces reduce the gap between operational documentation and development practice, since the same command vocabulary can be used in both settings.

Rapid reconfiguration is enabled by shared volumes that mount secrets and code and by optional scripts that link a Git repository so services run directly from a developer’s source tree [2507.21464]. Switching branches and restarting services reveals effects quickly. Persistence is also explicit: volumes hold configuration, secrets, and code so containers can be restarted without losing state relevant for iterative development.

The security posture is intentionally limited to development use. Development uses self-signed host certificates; the CE is configured with SciTokens; secrets are shared via volumes; and the Factory enforces per-site throttles, protocols, and authentication methods [2507.21464]. The paper distinguishes these practices from production requirements and states that the self-signed arrangement is appropriate for local testbeds, not for production.

## 4. Developer workflow and integration testing

The basic workflow begins by bringing up the system with Docker or Podman Compose using the provided `compose.yml`, starting the CE, Factory, and Frontend on a local bridged network; a startup script generates host certificates [2507.21464]. After the images are pulled once, the system can be run offline.

Code iteration relies on mounted source trees. Developers mount the GlideinWMS source tree into the appropriate workspace via shared volumes and may use optional linking scripts in the Factory or Frontend workspaces to run from a Git repository rather than installed packages [2507.21464]. Switching Git branches updates the code used by services. Restarting affected services through the emulated `systemctl` under `supervisord` preserves the standard operational control flow.

Service reconfiguration is similarly local and iterative. Configuration files in mounted volumes can be edited and services restarted or reloaded quickly; the Frontend workspace includes scripts to refresh credentials and submit a smoke test job to validate changes [2507.21464]. Debugging proceeds by inspecting logs and status in each workspace, including HTCondor daemons on the Frontend/AP/CM, Factory logs for pilot submissions, and CE or batch logs for pilot starts.

Integration testing extends the same model through the `testbed-workspace`, also called the Integration Test-Bed or ITB. It starts from a bare-bones image with minimal OS to exercise the full RPM-based installation procedure and dependency resolution, validating new releases in a clean environment via automated deployment scripts [2507.21464]. It deploys a minimal three-node system and runs smoke tests to verify that pilots are submitted, join the User Pool, and run test jobs.

The validation criteria are concrete: successful installation via RPM with no missing dependencies; Factory maintenance of pressure and Glidein submission to the CE; Glidein joining of the User Pool and successful completion of smoke test jobs; and successful credential refresh and SciToken flow [2507.21464]. For token-based authentication, the Fermilab Managed Token service can provision SciTokens to enable fully automated deployment and testing on a designated host. The team plans to design a similar approach for dynamic CI systems such as GitHub-hosted runners.

The `build-workspace` complements the ITB by running the GlideinWMS ReleaseManager and RPM tooling to build releases from any Git ref, serve packages via a local YUM server, or integrate with OSG’s Koji and repositories [2507.21464]. Composed into the ITB, it enables package–install–test cycles. GitHub workflows and Docker Hub provide CI/CD for workspace images.

## 5. Platforms, runtimes, IDE integration, and performance

Docker and Podman are the supported container runtimes for running the workspaces, while Apptainer is used by Glideins on worker nodes in distributed computing contexts, described as a nested container case [2507.21464]. Linux is the preferred development platform; Windows via WSL2 is supported; Apple Silicon Macs are supported natively using CoLiMa or Podman; and multi-architecture images for `linux/amd64` and `linux/arm64` are published [2507.21464].

The OS and architecture support described for `gwms-workspace` includes AlmaLinux 9 and 8, both with Python 3.9, and Scientific Linux 7 with Python 3.6, across `x86_64` and `aarch64` [2507.21464]. Multi-arch images are published via Docker Buildx, and runtime automatically selects the correct architecture. This is tied to a specific performance observation: on Apple Silicon Macs, running `x86_64` containers via emulation using QEMU or Apple’s Rosetta 2 incurs a considerable performance loss of 20% or more [2507.21464]. The publication of native `linux/arm64` images and multi-arch manifests is presented as the response.

Usability preferences are stated rather than generalized. Podman is preferred at Fermilab for running workspaces, for licensing reasons, and Docker is also supported [2507.21464]. Compose was chosen for simple, single-host orchestration without needing multi-host scaling. Apptainer is used inside Glideins in production because it provides compact single-file images, runs unprivileged, minimizes overhead, and supports nesting well.

IDE integration is described at capability level. Workspaces integrate with IDEs like Microsoft Visual Studio Code: developers can run containers from VS Code, mount local code into the workspace, and use familiar GWMS service commands via the emulated `systemctl` to restart services after code changes [2507.21464]. Debugging is simplified by keeping relevant services together in each workspace and by having logs and control scripts in one place. The source explicitly notes that it does not include specific `devcontainer.json` or remote debugging snippets.

A common misunderstanding would be to read the performance discussion as a general benchmark study. The source explicitly states that it does not present quantitative throughput or overhead formulas or tables and includes no LaTeX performance expressions [2507.21464]. The performance claims are limited to runtime selection and the observed 20% or more emulation loss on Apple Silicon.

## 6. Educational role, limitations, security, and future directions

Workspaces are described as having significantly reduced onboarding time: interns and new team members can spin up a complete GWMS system in seconds and begin exploring and modifying components, whereas a comparable VM-based setup could take weeks [2507.21464]. They were also used in the Computational HEP Traineeship Summer School 2024 to give students first-hand experience with mechanisms and components that in production run millions of jobs. The system-like design makes internal interactions observable and instructive.

The limitations are explicit. Compose-based orchestration is single-host and does not provide automatic scaling across multiple machines, unlike Kubernetes or OKD [2507.21464]. The source treats this as acceptable for development, test, and teaching goals. Another limitation is that `systemd` is not available in containers, necessitating `supervisord` plus `systemctl` emulation. Full CI automation of token acquisition is also non-trivial because SciTokens typically require interactive web portal authentication.

Security considerations are framed narrowly around local testbeds. Development uses self-signed host certificates generated at startup, trust anchored by an included CA and shared via volumes; this is appropriate for local testbeds, not for production [2507.21464]. CE and pilots use SciTokens, described as JWT-based capability tokens, rather than GSI proxies, and credential refresh scripts exist in the Frontend workspace. Secrets and code are shared via mounted volumes, and care should be taken when exposing ports or connecting to external resources during testing.

The best practices enumerated in the source emphasize the `gwms-workspace` common base and parameterization of CE, Factory, and Frontend images to keep OS and platform customizations centralized; preference for Podman or Docker Compose for single-host orchestration; use of shared volumes for secrets and source trees; and use of the bridged network and internal DNS [2507.21464]. They also recommend running services under `supervisord` and using emulated `systemctl`, using native `arm64` images on Apple Silicon, leveraging `build-workspace` and ReleaseManager for reproducible packaging, and validating changes with smoke tests from the Frontend workspace.

Future work is defined in two directions: automating the ITB in the GWMS GitHub CI workflows by solving token acquisition for dynamic CI environments, and continuing to refine multi-architecture build pipelines while extending optional components and scenarios [2507.21464]. This suggests that GlideinWMS Workspaces are intended as a stable development substrate whose scope may broaden, but still within the roles identified throughout the paper: development, integration testing, and teaching rather than production deployment.

Source: https://www.emergentmind.com/topics/glideinwms-workspaces