Skip to content

Operate Space ​

Processes and probes ​

The Space image runs serve, worker and migrate up. Scale API replicas independently. Run one worker. If another worker already owns execution, the new process waits for it to release the lock without dispatching schedules. In Kubernetes use one worker replica with the Recreate strategy.

API and worker have separate system HTTP listeners, defaulting to port 9100:

ProbePurpose
/healthThe process responds
/readyProcess and database readiness with migrations applied. A worker also needs the execution lock

While waiting for the lock, the worker answers 200 on /health and 503 on /ready. It logs reason=lock_busy once and reason=lock_acquired when it can start. If the wait persists, check whether the previous worker is still running. Keep one replica and let the supervisor restart a process that fails because of a database error.

Keep probes on the internal network. Do not expose the system port with the public API. Readiness alone does not confirm Orpheus or model-provider availability.

Run commands from the Space example directory. For a local connection, keep COMPOSE_FILE in .env:

sh
docker compose ps
docker compose logs --tail=100 api worker migrate
curl -fsS http://localhost:9110/ready

Data and updates ​

Back up the Space database separately from Orpheus. Space stores schedules, occurrence history and browser sessions. Orpheus stores messages and full results. Preserve configuration, the SAML pair and API keys in your secret store as well.

To update the example, stop the worker, change image versions in .env, run migrations and start the processes:

sh
docker compose stop worker
docker compose --profile execution pull
docker compose run --rm migrate
docker compose up -d api web worker

Take a backup before updating. Afterwards, check readiness, login and an upcoming occurrence. Following downtime, only the latest due occurrence is considered. See schedule behavior.

Diagnostics ​

For startup or process failures, inspect phase, reason and error_type in the logs. Database errors can also include sqlstate. These fields identify the failing stage without exposing connection strings or credentials.

Distinguish submission state from Orpheus execution run_status.

Submission stateMeaning
pendingWaiting for submission
dispatchingSubmitting or reconciling the request outcome
acceptedOrpheus accepted the run. See run_status for its outcome
skippedOccurrence skipped
failedSubmission failed
cancelledPending work cancelled

In the occurrence card, observed_at is the last successful observation of Orpheus state. sync_error_code records a subsequent synchronization error, preserving the earlier status. run_error_code relates to agent execution, while error_code relates to submission.

SymptomAction
dispatching with core_unavailableCheck the Orpheus address and connectivity from Space. Its worker can still be healthy
No new occurrencesCheck schedule status, time zone, worker readiness and Orpheus connectivity
run_not_found / session_not_foundCheck the Orpheus address and restore access to the original run. A missing record does not prove completion, so further execution remains blocked
Result returns 503Check the Orpheus URL and key on Space API. Stored history remains available
Saving returns 403Match the browser origin to the public URL and ensure the proxy passes the CSRF header
Saving services returns 422A code is absent from the core service catalog. Select an available service or update the catalog
Submission is failed with unknown_serviceThe selected service was removed before submission. Correct the selection or restore the catalog entry
Run accepted, then failed due to ENVSupply the variable value to the Orpheus worker and inspect the run error
error_code=session_unavailable or run_error_code=sandbox_lostThe environment cannot be continued. Use Reset context on the schedule card when no occurrence is active. Check retention for long intervals
Run now or context reset returns 409 schedule_busyInspect the unfinished occurrence, including an unknown submission outcome. Wait for completion or synchronization to recover
Manual command stays pendingCheck Space worker readiness. Accepting the command does not mean that Orpheus has received it

Pausing cancels pending cron occurrences but keeps pending manual commands. Deletion cancels both kinds. Already prepared requests and accepted work retain their reconciliation rules. See pausing and pending commands.

If the run record is lost ​

Check that Space points to the original Orpheus instance. Restore the record with the Orpheus database backup. Once Space observes a terminal status, planning resumes.

If recovery is impossible, check the submission state:

  • accepted: Orpheus already accepted the run, and Space will not resubmit it. Confirm that the original work is neither running nor able to resume. Then delete the schedule and create a new one. The old history remains. Space may keep trying to read the lost run, but these attempts do not block the new schedule.
  • dispatching: the submission outcome is unknown. Deleting the schedule does not stop submission attempts. Restore connectivity to the original Orpheus instance and establish the request outcome first. Do not create a replacement before then, to avoid duplicate execution.

Result retrieval does not overwrite stored history. See fields and error codes.