GlideinWMS Workspaces for Dev & Testing
- GlideinWMS Workspaces are system-like containers that emulate full Linux hosts, enabling integrated development, testing, and teaching environments.
- They use Compose orchestration to run multiple interdependent services within a single image, streamlining configuration and source code iteration.
- This containerized setup reduces onboarding time and efficiently replicates production-like pilot workflows without the complexities of VM-based systems.
Searching arXiv for the specified paper to ground the article in the cited source. GlideinWMS Workspaces are development- and test-oriented “system-like containers” for GlideinWMS, a pilot- and pressure-based Workload Management System that provisions elastic HTCondor “User Pools” for scientific collaborations, including CMS and DUNE (Mambelli et al., 29 Jul 2025). Unlike microservice-style production containers that typically run a single process, these workspaces behave like small Linux hosts: each image runs multiple services and includes tools needed to configure, operate, and inspect a full GWMS node. The approach collapses a minimal multi-node testbed into a compact, containerized setup that developers can run locally, including offline, rapidly reconfigure, and use for integration testing and teaching (Mambelli et al., 29 Jul 2025).
1. Concept and motivation
GlideinWMS provisions resources through a division of responsibilities between a Factory and a Frontend. A Factory submits pilot jobs, called “Glideins,” to Compute Entrypoints, while a Frontend, or other clients such as HEPCloud’s Decision Engine, requests pilots based on user demand and policies (Mambelli et al., 29 Jul 2025). In this model, pilots join an HTCondor User Pool and execute user jobs after the resource has been provisioned.
The motivation for Workspaces is stated in operational terms. Traditional GWMS development and test required multiple virtual machines in FermiCloud to cover simultaneous releases and OS variants, plus the operational overhead of certificates, puppet, firewall rules, and shared resource contention (Mambelli et al., 29 Jul 2025). The workspace approach replaces that VM-based arrangement with containers that emulate VM-like behavior closely enough to support simultaneous services, source-tree mounting, state inspection, and fast iteration.
The designation “system-like containers” is central. GWMS nodes are service-rich, with components such as HTCondor daemons, HTCondor-CE, the GWMS Factory, the Frontend, and the Access Point/Central Manager. Developers therefore need to run and coordinate several daemons together, inspect their state, mount source trees, and iterate quickly (Mambelli et al., 29 Jul 2025). Emulating a small Linux host inside each container, rather than decomposing everything into single-process containers, lowers friction for development, integration testing, and teaching.
A common misconception is to treat these containers as production deployment units. The source distinguishes them from specialized production containers and places them in a development, test, and teaching role. This is reinforced by the use of self-signed host certificates generated at startup and by the emphasis on local and offline workflows rather than production hardening (Mambelli et al., 29 Jul 2025).
2. Base system and component interactions
The minimal deployment mirrors a real GWMS setup and uses three containers (Mambelli et al., 29 Jul 2025).
| Container | Components or role | Workspace details |
|---|---|---|
| One-node cluster | HTCondor-CE and HTCondor batch system | Configured as a SciToken-authenticated computing resource; includes scripts; can fake more cores than physically available |
| GlideinWMS Factory | Submits Glideins to CE(s), maintains pressure, monitors pilots, hosts secure mailbox | Installs and configures the Factory; downloads HTCondor tarballs needed by Glideins; can link the GlideinWMS install to a Git repository |
| Scheduler and Frontend | Runs the GlideinWMS Frontend and acts as HTCondor Access Point and Central Manager | Installs and configures the Frontend and the User Pool; includes scripts to link a Git repository, refresh credentials, and submit a smoke test job |
Docker or Podman Compose is used for orchestration. A compose.yml defines the three services and a bridged network with name resolution for a fictitious glideinwms.org domain, while shared volumes are mounted across containers to share secrets and code (Mambelli et al., 29 Jul 2025). A startup script generates self-signed host certificates from an included trusted CA certificate.
The job lifecycle is also defined explicitly. The Frontend observes user job demand and resource status, applies experiment policies, and updates requests to the Factory to achieve prompt job starts without wasting resources. The Factory translates Frontend requests into pilot submissions to Compute Entrypoints, respecting throttles, supported protocols and authentication, and per-site policies, while maintaining requested pressure in queued and running Glideins (Mambelli et al., 29 Jul 2025). HTCondor-CE and HTCondor receive Glidein submissions and start pilots under the site batch system.
On start, Glideins probe the resource, manage credentials, provide monitoring and audit information, auto-detect resources such as cores, memory, disk, and GPUs, and can install tools including a container runtime and distributed filesystem (Mambelli et al., 29 Jul 2025). They then join the User Pool on the Frontend node and run user jobs, possibly multiple, in sequence or parallel. HTCondor matches user jobs to pilot startds; pilots can run, finish, and be replaced according to pressure and demand.
This suggests that the three-container arrangement is intended not merely as a mock environment but as a realistic minimal deployment in which production-style control flow remains observable. The paper’s figures, including fig/gwms_system.png and fig/gwms-testbed.png, are cited as system overviews of these interactions (Mambelli et al., 29 Jul 2025).
3. Distinguishing characteristics of the workspace model
The principal distinction from other containers is system-like composition. Each workspace runs multiple services under one image, such as HTCondor-CE plus HTCondor, or Frontend plus AP/CM, rather than decomposing functionality into microservices (Mambelli et al., 29 Jul 2025). This design is tied to the service-rich character of GWMS nodes and to the need for developers to inspect and reconfigure integrated services rapidly.
Service orchestration is handled by Compose, which coordinates multiple service-rich containers, provides networking, name resolution, shared volumes, and optional port exposure for hybrid tests (Mambelli et al., 29 Jul 2025). The networking model is a bridged network with internal DNS for glideinwms.org, with parameterizable external connectivity for real sites and clouds.
Because systemd is not available inside containers, supervisord is used plus an emulation of systemctl to preserve documented service start and stop commands (Mambelli et al., 29 Jul 2025). This is operationally significant: the preserved control flow means documented GWMS service commands still apply inside the workspace environment. A plausible implication is that the workspaces reduce the gap between operational documentation and development practice, since the same command vocabulary can be used in both settings.
Rapid reconfiguration is enabled by shared volumes that mount secrets and code and by optional scripts that link a Git repository so services run directly from a developer’s source tree (Mambelli et al., 29 Jul 2025). Switching branches and restarting services reveals effects quickly. Persistence is also explicit: volumes hold configuration, secrets, and code so containers can be restarted without losing state relevant for iterative development.
The security posture is intentionally limited to development use. Development uses self-signed host certificates; the CE is configured with SciTokens; secrets are shared via volumes; and the Factory enforces per-site throttles, protocols, and authentication methods (Mambelli et al., 29 Jul 2025). The paper distinguishes these practices from production requirements and states that the self-signed arrangement is appropriate for local testbeds, not for production.
4. Developer workflow and integration testing
The basic workflow begins by bringing up the system with Docker or Podman Compose using the provided compose.yml, starting the CE, Factory, and Frontend on a local bridged network; a startup script generates host certificates (Mambelli et al., 29 Jul 2025). After the images are pulled once, the system can be run offline.
Code iteration relies on mounted source trees. Developers mount the GlideinWMS source tree into the appropriate workspace via shared volumes and may use optional linking scripts in the Factory or Frontend workspaces to run from a Git repository rather than installed packages (Mambelli et al., 29 Jul 2025). Switching Git branches updates the code used by services. Restarting affected services through the emulated systemctl under supervisord preserves the standard operational control flow.
Service reconfiguration is similarly local and iterative. Configuration files in mounted volumes can be edited and services restarted or reloaded quickly; the Frontend workspace includes scripts to refresh credentials and submit a smoke test job to validate changes (Mambelli et al., 29 Jul 2025). Debugging proceeds by inspecting logs and status in each workspace, including HTCondor daemons on the Frontend/AP/CM, Factory logs for pilot submissions, and CE or batch logs for pilot starts.
Integration testing extends the same model through the testbed-workspace, also called the Integration Test-Bed or ITB. It starts from a bare-bones image with minimal OS to exercise the full RPM-based installation procedure and dependency resolution, validating new releases in a clean environment via automated deployment scripts (Mambelli et al., 29 Jul 2025). It deploys a minimal three-node system and runs smoke tests to verify that pilots are submitted, join the User Pool, and run test jobs.
The validation criteria are concrete: successful installation via RPM with no missing dependencies; Factory maintenance of pressure and Glidein submission to the CE; Glidein joining of the User Pool and successful completion of smoke test jobs; and successful credential refresh and SciToken flow (Mambelli et al., 29 Jul 2025). For token-based authentication, the Fermilab Managed Token service can provision SciTokens to enable fully automated deployment and testing on a designated host. The team plans to design a similar approach for dynamic CI systems such as GitHub-hosted runners.
The build-workspace complements the ITB by running the GlideinWMS ReleaseManager and RPM tooling to build releases from any Git ref, serve packages via a local YUM server, or integrate with OSG’s Koji and repositories (Mambelli et al., 29 Jul 2025). Composed into the ITB, it enables package–install–test cycles. GitHub workflows and Docker Hub provide CI/CD for workspace images.
5. Platforms, runtimes, IDE integration, and performance
Docker and Podman are the supported container runtimes for running the workspaces, while Apptainer is used by Glideins on worker nodes in distributed computing contexts, described as a nested container case (Mambelli et al., 29 Jul 2025). Linux is the preferred development platform; Windows via WSL2 is supported; Apple Silicon Macs are supported natively using CoLiMa or Podman; and multi-architecture images for linux/amd64 and linux/arm64 are published (Mambelli et al., 29 Jul 2025).
The OS and architecture support described for gwms-workspace includes AlmaLinux 9 and 8, both with Python 3.9, and Scientific Linux 7 with Python 3.6, across x86_64 and aarch64 (Mambelli et al., 29 Jul 2025). Multi-arch images are published via Docker Buildx, and runtime automatically selects the correct architecture. This is tied to a specific performance observation: on Apple Silicon Macs, running x86_64 containers via emulation using QEMU or Apple’s Rosetta 2 incurs a considerable performance loss of 20% or more (Mambelli et al., 29 Jul 2025). The publication of native linux/arm64 images and multi-arch manifests is presented as the response.
Usability preferences are stated rather than generalized. Podman is preferred at Fermilab for running workspaces, for licensing reasons, and Docker is also supported (Mambelli et al., 29 Jul 2025). Compose was chosen for simple, single-host orchestration without needing multi-host scaling. Apptainer is used inside Glideins in production because it provides compact single-file images, runs unprivileged, minimizes overhead, and supports nesting well.
IDE integration is described at capability level. Workspaces integrate with IDEs like Microsoft Visual Studio Code: developers can run containers from VS Code, mount local code into the workspace, and use familiar GWMS service commands via the emulated systemctl to restart services after code changes (Mambelli et al., 29 Jul 2025). Debugging is simplified by keeping relevant services together in each workspace and by having logs and control scripts in one place. The source explicitly notes that it does not include specific devcontainer.json or remote debugging snippets.
A common misunderstanding would be to read the performance discussion as a general benchmark study. The source explicitly states that it does not present quantitative throughput or overhead formulas or tables and includes no LaTeX performance expressions (Mambelli et al., 29 Jul 2025). The performance claims are limited to runtime selection and the observed 20% or more emulation loss on Apple Silicon.
6. Educational role, limitations, security, and future directions
Workspaces are described as having significantly reduced onboarding time: interns and new team members can spin up a complete GWMS system in seconds and begin exploring and modifying components, whereas a comparable VM-based setup could take weeks (Mambelli et al., 29 Jul 2025). They were also used in the Computational HEP Traineeship Summer School 2024 to give students first-hand experience with mechanisms and components that in production run millions of jobs. The system-like design makes internal interactions observable and instructive.
The limitations are explicit. Compose-based orchestration is single-host and does not provide automatic scaling across multiple machines, unlike Kubernetes or OKD (Mambelli et al., 29 Jul 2025). The source treats this as acceptable for development, test, and teaching goals. Another limitation is that systemd is not available in containers, necessitating supervisord plus systemctl emulation. Full CI automation of token acquisition is also non-trivial because SciTokens typically require interactive web portal authentication.
Security considerations are framed narrowly around local testbeds. Development uses self-signed host certificates generated at startup, trust anchored by an included CA and shared via volumes; this is appropriate for local testbeds, not for production (Mambelli et al., 29 Jul 2025). CE and pilots use SciTokens, described as JWT-based capability tokens, rather than GSI proxies, and credential refresh scripts exist in the Frontend workspace. Secrets and code are shared via mounted volumes, and care should be taken when exposing ports or connecting to external resources during testing.
The best practices enumerated in the source emphasize the gwms-workspace common base and parameterization of CE, Factory, and Frontend images to keep OS and platform customizations centralized; preference for Podman or Docker Compose for single-host orchestration; use of shared volumes for secrets and source trees; and use of the bridged network and internal DNS (Mambelli et al., 29 Jul 2025). They also recommend running services under supervisord and using emulated systemctl, using native arm64 images on Apple Silicon, leveraging build-workspace and ReleaseManager for reproducible packaging, and validating changes with smoke tests from the Frontend workspace.
Future work is defined in two directions: automating the ITB in the GWMS GitHub CI workflows by solving token acquisition for dynamic CI environments, and continuing to refine multi-architecture build pipelines while extending optional components and scenarios (Mambelli et al., 29 Jul 2025). This suggests that GlideinWMS Workspaces are intended as a stable development substrate whose scope may broaden, but still within the roles identified throughout the paper: development, integration testing, and teaching rather than production deployment.
