Open Connectome REST API

• Open Connectome REST API is a stateless, efficient web interface designed to access and analyze multi-terabyte 3D neuroimaging datasets.
• It uses spatial partitioning with cuboids, HDF5 payloads, and multi-resolution pyramids to optimize data transport and processing.
• The API achieves high throughput via parallel GET requests and tailored I/O on heterogeneous hardware for both read and write operations.

The Open Connectome REST API provides a stateless, uniform, and high-performance web interface specialized for spatial analysis and annotation of high-throughput brain imaging data, as deployed within the Open Connectome Project Data Cluster. Its primary function is to facilitate scalable access to multi-terabyte 3D (and higher-dimensional) neuroimaging datasets and their derived annotations, supporting connectome-building workloads such as parallel computer vision algorithms. The API design leverages distributed NoSQL/internet-scale principles and embodies data-intensive computing patterns, particularly well-suited for large-scale neuroscience applications (Burns et al., 2013).

1. API Structure and Underlying Architecture

The Open Connectome REST API adheres to a strict RESTful and stateless approach. All operations are performed using plain HTTP GET, PUT, or DELETE against human-readable URL patterns. Each HTTP request is self-contained—there is no session state on the server, as every request carries all necessary context and parameters.

Payloads are transferred using HDF5 format (application/octet-stream), ensuring efficient transport and representation of large, dense multi-dimensional arrays in contrast to conventional web formats such as JSON or XML.

The data representation is spatially partitioned: each dataset consists of a dense D-dimensional array (typically $D=3\dots5$) which is divided into fixed-size "cuboids" containing $2^{18}$ voxels each. Cuboids are indexed and laid out in a Morton-order (Z-order) space-filling curve, with the index $M(x,y,z)$ defined for a 3D voxel at $(x, y, z)$ as:

$$M(x, y, z) = \sum_{i=0}^{\lceil \log_2 N \rceil-1} \Big(\operatorname{bit}_i(z) \ll 3i \mid \operatorname{bit}_i(y) \ll (3i+1) \mid \operatorname{bit}_i(x) \ll (3i+2)\Big)$$

This layout ensures that power-of-two-aligned subvolumes are contiguous in storage. The storage hierarchy implements a multi-resolution pyramid: each level halves $X$, $Y$ dimensions (not $Z$), maintaining near-cubic cuboid shape in world units.

Cluster hardware is heterogeneous. "Database nodes" with large RAID-6 spinning-disk arrays store image and annotation data for sequential read workloads. "SSD-I/O nodes" provision active annotation projects (RAID-0 Vertex SSDs, ≈20K IOPS) for small random writes. Separate "file-server" nodes provide 2D tiles for web visualization (e.g., with CATMAID). Read traffic is routed to database nodes, and write traffic is directed to SSD-I/O nodes to prevent I/O contention. When an SSD-based project becomes inactive, it is migrated back to disk nodes via MySQL dump/restore (Burns et al., 2013).

2. Core Endpoints and Resource Patterns

The API exposes all operations via well-defined URI templates, typically returning or expecting HDF5. Primary resource types are structured as follows:

Operation	URI Template Example	Purpose
3D image cutout	GET /{token}/{format}/{res}/{x₀,x₁}/{y₀,y₁}/{z₀,z₁}/<br> /bock11/hdf5/4/512,1024/...	Retrieving a spatial subvolume at any resolution
Annotation by ID	GET /{annoproj}/{annoID}/{dataOption}/	Object metadata, bounding box, sparse/dense cutout
Batch read/write	GET /{annoproj}/{id1,id2,…}/, PUT /{token}/{dataOption}/	Efficient multi-object access or update
Metadata query	GET /objects/{field}/{value}/	List/filter objects by type, confidence, etc.

3D cutouts specify the desired spatial region using resolved pyramid level (res) and voxel ranges (x₀,x₁, etc.). Annotations are accessed as dense or sparse representations, with options for querying voxels, metadata, or region cutouts masked to object IDs. Batch endpoints support efficient parallel workflows. Metadata-only queries return lists of object IDs or filtered datasets based on arbitrary field predicates (equality, range).

3. Request Semantics, Query Options, and Filtering

Each endpoint requires a set of parameters:

• Spatial arguments: res (integer pyramid level), voxel coordinate ranges (x₀,x₁, y₀,y₁, z₀,z₁), and channel selection for multi-modal data (via separate URI namespaces).
• Annotation write semantics (PUT):
  • overwrite: Replace extant voxel labels.
  • preserve: Retain previous label, ignore new label in conflicts.
  • exception: Keep the old label and register new label in an "exceptions" list.

Metadata queries support both equality and range predicates over integer, float, or string fields, e.g. /objects/type/[synapse](https://www.emergentmind.com/topics/synapse-synergistic-associative-processing-semantic-encoding)/confidence/geq/0.99/. Batch APIs provide natural paging via object ID lists, facilitating scalable, high-throughput manipulation or extraction.

4. Access Control and Authentication

The Open Connectome REST API adopts an open-access paradigm for public datasets: no authentication token is necessary for access. For private datasets, a required short URL "token" is embedded as the leading segment of the request path (e.g., /token/hdf5/...). No OAuth or explicit API-key infrastructure is described; possession of the full URL is equivalent to access rights. This scheme simplifies usage but should be evaluated in the context of broader data governance requirements (Burns et al., 2013).

5. Performance Characteristics and Scalability

Performance benchmarks for the production system are as follows (single database node, 11 × 3TB RAID-6, 64GB RAM):

• Cutout throughput:
  • In-cache, cuboid-aligned: 173 MB/s (single request)
  • Random-aligned: 121 MB/s
  • Unaligned (arbitrary byte ranges): 61 MB/s
• Batching and parallelism:

Up to 16 concurrent GETs achieves near-linear scaling, saturating memory bandwidth for aggregate cutouts ≥256 MB.

• Annotation write throughput:
  • RAID node: ≈19 MB/s for 2 MB annotation regions, with significant degradation beyond this due to MySQL index contention.
  • SSD node: Small random write throughput ≈150% higher than RAID.
• I/O orchestration:
  • Sequential read/cutout on spinning disks.
  • Random write/annotation on SSDs.
  • Visualization workloads (tiles) isolated on file-server nodes.

A plausible implication is that workloads heavily reliant on many small, random annotation updates will benefit significantly from the SSD-I/O tier. Aligning cutout bounding boxes to cuboid boundaries and utilizing batched, parallel data access are necessary to fully leverage system throughput.

6. Usage Patterns and Client Integration

The API is amenable to scripting and automated workflows via standard HTTP clients and supports a variety of programming languages (e.g., Python via requests and h5py). Typical interactions include:

• Downloading a 3D image cutout:

  curl -X GET "http://openconnecto.me/bock11/hdf5/2/1024,1536/2048,2560/0,256/" --output subvol.h5

• Querying annotations with metadata predicates:

  curl -X GET "http://openconnecto.me/objects/type/synapse/confidence/geq/0.99/"

• Downloading a dense cutout for a single annotation object:

  curl -X GET "http://openconnecto.me/annoproj/75/cutout/0/3000,3200/4000,4200/100,120/" --output syn75.h5

• Loading a subvolume in Python:

  import requests, h5py, io url = "http://openconnecto.me/bock11/hdf5/1/0,512/0,512/0,64/" resp = requests.get(url) f = h5py.File(io.BytesIO(resp.content), 'r') volume = f['data'][:] # 512×512×64 uint8 array

Efficient usage is best achieved by aligning cutout coordinates to cuboid boundaries (multiples of $2^{18}$ voxels), batching annotation operations (20–40 objects per PUT), leveraging multicore concurrency (8–16 GETs), and caching frequently-accessed cutouts. For large-scale vision tasks, lower resolution pyramid levels should be preferred to reduce data transfer and storage overhead.

7. Recommendations and Best Practices

For optimal performance and reliability when using the Open Connectome REST API:

• Align requests to cuboid boundaries to prevent computationally expensive in-server data repacking.
• Deploy multiple concurrent GET requests (8–16 threads) to saturate memory bandwidth and maximize I/O efficiency.
• Amortize network and protocol overheads by batching annotation writes (ideally 20–40 objects per PUT).
• Use the coarsest resolution level supporting analysis needs, as demonstrated by large-scale synapse detection for efficiency.
• For intense random-write workloads (e.g., annotation), utilize SSD-I/O nodes and limit outstanding parallel client requests (~50) to avoid MySQL contention.
• Cache commonly used HDF5 cutouts client-side to accelerate access, and consider prefetching neighboring regions to support exploratory analysis.
• When a project’s write activity diminishes, anticipate migration back to disk storage.

The practical upshot is that adherence to these patterns is critical for achieving advertised system throughput and avoiding architectural bottlenecks. For most tasks, the API’s stateless, uniform design and its reliance on spatially-indexed HDF5 payloads provide sufficient abstraction and performance for large-scale connectomics and related computational neuroscience workloads (Burns et al., 2013).