11/14/2018 2:43 pm ET - issue causing intermittent maintenance pages on some sites - brief planned maintenance period (1-2 minutes) at midnight ET for sites that experienced today's issue
10/30/2018 beginning at 10:34 am ET Reduced network load from upstream sources. Major backbone provider (Cogent) experiencing widespread routing issues. They provide service to a large portion of the US market - we attempted to route CV3 traffic around these issues to reliable channels. Systems stabilized 11:17 pm ET.
8/29/2018 10:27 pm ET - End of DDoS attack
8/29/2018 5:03 pm ET - Our new blocks preventing DDoS attacks were aggressively limiting access to our API as well. This has been adjusted and legitimate traffic should be able to connect.
8/29/2018 4:35 pm investigating issue impacting web service connectivity
8/29/2018 3:48pm ET - All services continue to operate normally. We are still seeing (and blocking) a high level of fraudulent requests to our network.
8/29/2018 3:26pm ET - We completed our new blocks and all stores and services appear to be settling back into normal operation.
8/29/2018 3:18 pm ET - Our new blocks started working which restored partial service to CV3. Some traffic is getting through while other ISPs and regions are unable to reach CV3. We continue to work to resolve the issues.
8/29/2018 3:15 pm ET The DDoS attack vector changed again and stores are currently unavailable. All day we have been working to block a sustained attack of 6M requests / second and have managed to limit its impact since the outage this morning. We are currently moving our blocks to cover the new vector, which should restore network connectivity.
8/29/2018 12:32 pm ET - We are continuing to block much higher than normal levels of invalid traffic, but otherwise services are operating normally.
8/29/2018 12 pm ET - Traffic is currently restored to stores and admin. We are still blocking around 200GB/sec of attacking vectors, and some ancillary services have yet to come online. This is a positive step but this incident is not yet fully resolved.
8/29/2018 11:10 am ET - We successfully mitigated the initial denial of service attack and briefly restored most network traffic. However, the initial attack has evolved into a secondary attack that we are also working to mitigate. We are moving region by region to re-enable traffic from most ISPs. If you still cannot reach CV3 and/or your storefront, you are likely in one of the areas we have not re-enabled yet. All of these issues are being blocked well upstream of all CV3 services and no data breach has been detected.
8/29/2018 10:55 am ET - We have successfully mitigated the denial of service attack and network traffic has been restored for the vast majority of networks and ISPs. If you are still unable to reach your storefront, you may be on an ISP or in a region where we have not yet fully restored service.
8/29/2018 10 am ET - Large DNS attack currently hitting CV3. We are working to mitigate the attack and to determine whether this is part of a more widespread Internet attack. All indications are that once we eliminate the attack, service will resume quickly. This attack has taken down our network but stops at our firewall, so CV3 services appear to be normal and we are waiting for the network to be restored.
8/29/2018 9:50 am ET - investigating admin connectivity issue
8/8/18 14:44 ET 3rd party providers experiencing connectivity issues
6/27/2018 00:00 am ET TLS 1.0 & 1.1 permanently disabled on CV3 platform
4/30/2018 8:58 - 9:00 am ET Issue caused maintenance pages on subset of stores
4/18/2018 2:04 - 2:37 pm ET - Issues with cart add/remove on subset of stores
4/9/2018 9:56 - 10:03 am ET - Issue caused maintenance pages on subset of stores
4/4/2018 9:35 - 9:43 am ET and 12:50 - 12:53 pm ET - Issue caused maintenance pages on subset of stores
3/26/2018 - 00:00 am - 1:30 pm ET - disabled TLS 1.0 to allow client verification of any systems using our web service
3/5/2018 12:16 - 12:29 pm ET Issue caused maintenance pages on subset of stores
2/20/18 11:16 - 11:24 AM Issue caused maintenance pages on subset of stores
2/19/18 Intermittent payment data / order export issue was resolved at 10:20 am ET
2/14/18 06:58 - 07:29 am ET issue causing maintenance page in admin & staging - live sites were not affected
2/13/18 06:00 am ET admin updates completed
2/12/18 11:00 am - 12:20 pm ET Issues causing intermittent maintenance pages on a subset of stores
1/3/18 08:05 - 09:12 ET - Admin/staging intermittently inaccessible
11/13/17 issues causing maintenance pages on a subset of stores resolved 9:10 am ET and 12:53 pm ET; lingering latency issues resolved
PayPal database maintenance - 11/9/17 - details
beginning 11/6/17 - Comcast/Xfinity experiencing connectivity issues
11/5/17 Template, Image & File Libraries inaccessible due to planned maintenance from 22:00 ET - 00:00 ET. Live sites not impacted.
11/2/17 intermittent maintenance pages primarily affecting store admin
10/28/17 intermittent maintenance pages affecting store admin
10/14/17 RESOLVED - Site Maintenance in admin. We apologize for the inconvenience and will return admin access soon.
10/13/2017 RESOLVED - Experiencing delays with import/export processes and slow product/category edits
10/13/2017 Maintenance pages on subset of stores 5:31 am - 6:12 am ET
10/10/2017 Network issue affected subset of stores 8:20 pm - 8:56 pm ET / webservice issue 9:20 pm - 10:11 pm ET
9/22/2017 PayPal database maintenance - details
9/17/2017 USPS rate tool planned outage - details
9/15/2017 PayPal database maintenance - details
9/14/2017 2:40-3:56 pm ET Amazon Simple Storage Service experienced Increased Error Rates for S3 requests in the US-EAST-1 Region.
7/13/2017 - 8/14/2017 Rackspace email issues - see Rackspace status page for details
7/26/2017 Store admin down 6:58 - 7:22 pm ET
7/24/2017 Issue with items disappearing from cart affecting subset of stores 3:40 - 3:53 pm ET
7/24/2017 from 00:00 - 00:15 - Mandatory Maintenance Window
USPS Rate Tool is experiencing issues.
Admin issue affecting order download, product edit, image upload was resolved 10:43 am ET on July 3, 2017.
Rackspace email delivery delays - see Rackspace status page for details
Investigating issues with publishing. Resolved
6/7/2017 6:34 p.m. - 7:49 pm ET
Intermittent issues causing maintenance pages or white pages in admin for a subset of stores. Live sites were not affected. Resolved
6/7/2017 5:03 - 6:19 p.m. ET
Experienced white pages in admin. Live sites and web service were not impacted. Resolved
(6/7/2017 4:02 - 4:30 p.m. ET)
Intermittent white pages in admin. Resolved
(4/3/2017 11:32 a.m. - 1:32 p.m. ET)
Major Amazon Web Services (AWS) outage took out complete regions, initially impacting images and S3-dependent applications before expanding in severity to include issues with other services, including storage gateway issues. Cascading effects caused maintenance pages beginning around 2:11 ET. Things started to resolve around 5:33 pm ET.
(2/28/2017 12:53 p.m. - 5:33 pm ET)
Experienced issues intermittently impacting store admin and staging. Live sites and web service were NOT impacted. Resolved
(2/22/2017 6:24 a.m. - 8:50 a.m. ET)
Rackspace experienced issues that may have resulted in login failures. Resolved
(12/5/2016 1:41 p.m. - 5:14 pm ET)
Rackspace experienced issues that may have impacted hosted email. Resolved
(12/5/2016 10:06 a.m. - 12:24 pm ET)
Bronto experienced issues that may have impacted some stores using the Bronto integration. Resolved
(2016-11-25 11:35 am ET)
General disturbance due to a DNS provider outage. Many providers experienced issues (PayPal, Twitter, etc). CV3's DNS is not hosted by the affected provider. Resolved
(2016-10-21 8:00 am ET)
Issue Resolved - PayPal experiencing a system issue affecting PayPal APIs.
Resolved, per PayPal status page: This issue has been resolved as of Sep 21, 2016 09:27 AM CDT.
(2016-09-21 10:11am ET)
Issue Resolved - Intermittent Site Maintenance on admin and staging only - live sites not affected.
(2016-08-16 2:40pm - 3:46pm ET)
Investigated a network issue affecting a subset of stores
Scheduled maintenance window April 7, 2016 beginning at 00:00 ET that lasted < 5 minutes. During that time sites displayed a Site Maintenance message
(2016-04-07 00:00 am ET)
Investigated Site Maintenance Pages on some sites
(2016-03-03 12:13 pm - 12:17 pm ET)
Investigating Site Maintenance Pages on some sites at this time
(2016-03-03 12:13 pm ET)
(2016-02-18 11:14 - 11:59 am ET)
Intermittent issues affecting admin and staging
(2016-02-12 4:20 - 4:34 pm ET)
Rackspace delivery issues
(2016-02-10 11:43 am - 4:13 pm ET)
AWS issues in US East region affecting admin and staging. No impact to live sites.
(2016-02-08 3:20 - 4:38 pm ET)
Major Authorize.net outage.
(01/29/16 ~12:14 ET - 16:17 ET)
Site Maintenance Pages on some sites observed and investigated
(01/29/16 ~08:36 - 09:06 ET)
(01/21/16 ~ 08:50 ET - 12:00 ET)
Major Authorize.net outage.
(01/06/16 ~09:00 - 10:02 ET)
Amazon DNS services - DNS resolution issues.
(01/12/16 ~11:00 - 11:52 ET)
Major Authorize.net outage.
(01/06/16 ~09:00 - 10:02 ET)
Observed a brief spike that prompted some alerts. This was immediately resolved and we continued to monitor.
(12/16/15 10:49 ET)
We have isolated the cause and sites should be returning to full capacity
(12/15/15 17:04 ET)
We are currently investigating an issue with slow page loads and Site Maintenance pages.
(12/15/15 16:40 ET)
Webservice once again available.
(11/23/2015 - 11:30 ET)
Webservice is temporarily unavailable.
(11/23/2015 - 10:10 ET)
Brief Site Maintenance Window from 01:15 ET - 02:00 ET. Site Maintenance pages may appear for less than 5 min.