System Operations
Admins have access to operational tooling that supervisors and customers don't.
Background jobs (Sidekiq)
The portal runs background jobs for Five9 syncing, S3 audio movement, billing aggregation, and CSV exports. Admins can view the background job dashboard at https://portal.cicops.ai/sidekiq (opens in a new tab).
From there you can:
- See job queues and how backed up they are
- Retry failed jobs
- View the recurring schedule
- See execution history
Use this when investigating "why isn't X happening" — most of the time the answer is "the job is queued and will run on its schedule" or "the job failed and needs to be retried or fixed."
Recurring schedule
| Job | Frequency |
|---|---|
| Five9 recordings sync | Every 15 minutes (0,15,30,45 * * * *) |
| Five9 daily backfill | Every day at 00:03 |
| S3 audio move from SFTP | Every 15 minutes (offset, 5,20,35,50 * * * *) |
| Billing — companies aggregation | Mondays at 01:10 |
| Billing — organizations aggregation | Mondays at 01:10 |
| Billing — past-due invoice check | Daily at 00:10 |
Error tracking (Sentry)
The portal sends application errors to Sentry. If something is broken, the engineering team typically learns about it from Sentry first. Admins can be granted access to the Sentry project — ask engineering.
API documentation (Rswag)
Admins can view the portal's API documentation at https://portal.cicops.ai/api-docs (opens in a new tab). The portal exposes a small REST API (companies, inbound numbers, profile, playbook) that external systems can consume via OAuth2.
Use this if you're working with an integrator who needs to read CIC data programmatically. See API Reference for the auth flow.
Five9 sync status
Five9 recordings sync every 15 minutes during the day, with a daily backfill at 00:03 in the early morning. If recordings are missing:
- Check the background job dashboard (opens in a new tab) — is the Five9 sync job failing?
- Check whether Five9 itself is having issues (their status page).
- The daily backfill catches things missed during the day, so missing recent recordings often resolve themselves overnight.
S3 audio storage
Call audio files are pulled from Five9's SFTP server and moved to S3 every 15 minutes (offset from the Five9 sync). If a recording's audio won't play, it may be that the S3 move job hasn't run yet. Wait, then check the job dashboard.
When jobs back up
If the Sidekiq dashboard shows hundreds of pending jobs:
- Check whether one job class is stuck (look at the Busy tab)
- Check whether the workers are hitting an external service that's down (Five9, Stripe, S3)
- Check the Retries queue for systematic failures
A backlog of a few minutes is normal during peak hours. A backlog of hours means escalate to engineering.
When the API is misbehaving
The portal exposes an OAuth2-secured REST API. If integrators report 5xx errors:
- Check Sentry for the matching error
- Check Sidekiq — API requests don't usually queue, but downstream effects might
- Check whether the integrator's OAuth token is still valid (Doorkeeper logs token issuance and expiration)
Next
- Security — IP restrictions and access control
- On-call Playbook — incident response
- API Reference — auth and endpoints