Our engineering team is continuing to work on this problem. At this moment, this is a top priority for our infrastructure team.
We have an open communication channel with our upstream provider since March 2nd and have managed to identify two separate issues, one of which has been resolved by March 5th (details below).
Issue 1: Connectivity issues with GCP storage, AWS S3, and Docker pull, overall network connectivity issues.
Summary: An urgent unscheduled maintenance on our upstream provider's network started on March 2nd and continued until March 5th. On the Semaphore end, this manifested as general instability of the network and increased download and upload times towards resources outside of the Semaphore environment.
These issues were resolved on March 5th.
If you're still experiencing this problem please reach out to our support team at email@example.com
Issue 2: Slow Docker push towards AWS
Summary: On a small number of projects docker push will run drastically slower than expected. This issue is sporadic and so far it was limited to pushes to AWS ECR and DockerHub (also hosted on AWS). We are still working with the upstream providers to pinpoint the exact cause of this behavior.
Any insights from our users that are affected by this will help us build a more statistically relevant sample of the issue and bring us closer to resolving it.
If you are affected by this problem we urge you to reach out to our support team at firstname.lastname@example.org
and provide a link to the affected jobs.
Our engineers are preparing a docker-debug script that collects additional metrics and we will be sending it out next week to all affected parties that are interested in helping us get to the bottom of this.