KEMBAR78
tag:us.githubstatus.com,2005:/history GitHub Enterprise Cloud US Status - Incident History 2025-10-23T09:06:11Z GitHub Enterprise Cloud US tag:us.githubstatus.com,2005:Incident/26848116 2025-10-22T17:35:48Z 2025-10-22T17:35:48Z US - Disruption with GHEC With Data Residency Signup <p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>17:35</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.</p><p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>17:35</var> UTC</small><br><strong>Update</strong> - We have now recovered and will resolve this incident.</p><p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>16:45</var> UTC</small><br><strong>Update</strong> - We have applied a mitigation and are monitoring for recovery.</p><p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>14:33</var> UTC</small><br><strong>Update</strong> - We have identified the problem and are working on a mitigation. We will provide further updates as we have them.</p><p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>14:03</var> UTC</small><br><strong>Update</strong> - We are currently experiencing disruptions with the GHEC with Data Residency signup process. Some requests to provision new GHEC with Data Residency enterprises will not complete at this time. We are investigating and will provide further updates as we have more information</p><p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>13:56</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p> tag:us.githubstatus.com,2005:Incident/26788410 2025-10-17T14:12:45Z 2025-10-20T12:56:41Z US - Disruption with push notifications <p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>14:12</var> UTC</small><br><strong>Resolved</strong> - On October 17th, 2025, between 12:51 UTC and 14:01 UTC, mobile push notifications failed to be delivered for a total duration of 70 minutes. This affected github.com and GitHub Enterprise Cloud in all regions. The disruption was related to an erroneous configuration change to cloud resources used for mobile push notification delivery.<br /><br />We are reviewing our procedures and management of these cloud resources to prevent such an incident in the future.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>14:01</var> UTC</small><br><strong>Update</strong> - We're investigating an issue with mobile push notifications. All notification types are affected, but notifications remain accessible in the app's inbox. For 2FA authentication, please open the GitHub mobile app directly to complete login.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>13:12</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p> tag:us.githubstatus.com,2005:Incident/26633372 2025-10-03T03:47:29Z 2025-10-10T15:45:09Z US - Incident with Copilot <p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>03:47</var> UTC</small><br><strong>Resolved</strong> - <p>On October 3rd, between approximately 10:00 PM and 11:30 Eastern, the Copilot service experienced degradation due to an issue with our upstream provider. Users encountered elevated error rates when using the following Claude models: Claude Sonnet 3.7, Claude Opus 4, Claude Opus 4.1, Claude Sonnet 4, and Claude Sonnet 4.5. No other models were impacted.</p><p>The issue was mitigated by temporarily disabling affected endpoints while our provider resolved the upstream issue. GitHub is working with our provider to further improve the resiliency of the service to prevent similar incidents in the future.</p></p><p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>03:47</var> UTC</small><br><strong>Update</strong> - This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.</p><p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>03:04</var> UTC</small><br><strong>Update</strong> - The upstream provider is implementing a fix. Services are recovering. We are monitoring the situation.</p><p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>02:42</var> UTC</small><br><strong>Update</strong> - We’re seeing degraded experience across Anthropic models. We’re working with our partners to restore service.</p><p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>02:41</var> UTC</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Copilot</p> tag:us.githubstatus.com,2005:Incident/26615319 2025-10-02T22:33:22Z 2025-10-06T22:39:22Z US - Degraded Gemini 2.5 Pro experience in Copilot <p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>22:33</var> UTC</small><br><strong>Resolved</strong> - Between October 1st, 2025 at 1 AM UTC and October 2nd, 2025 at 10:33 PM UTC, the Copilot service experienced a degradation of the Gemini 2.5 Pro model due to an issue with our upstream provider. Before 15:53 UTC on October 1st, users experienced higher error rates with large context requests while using Gemini 2.5 Pro. After 15:53 UTC and until 10:33 PM UTC on October 2nd, requests were restricted to smaller context windows when using Gemini 2.5. Pro. No other models were impacted.<br /><br />The issue was resolved by a mitigation put in place by our provider. GitHub is collaborating with our provider to enhance communication and improve the ability to reproduce issues with the aim to reduce resolution time.</p><p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>22:26</var> UTC</small><br><strong>Update</strong> - We have confirmed that the fix for the lower token input limit for Gemini 2.5 Pro is in place and are currently testing our previous higher limit to verify that customers will experience no further impact.</p><p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>17:13</var> UTC</small><br><strong>Update</strong> - The underlying issue for the lower token limits for Gemini 2.5 Pro has been identified and a fix is in progress. We will update again once we have tested and confirmed that the fix is correct and globally deployed.</p><p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>02:52</var> UTC</small><br><strong>Update</strong> - We are continuing to work with our provider to resolve the issue where some Copilot requests using Gemini 2.5 Pro return an error indicating a bad request due to exceeding the input limit size.</p><p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>18:16</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate and test solutions internally while working with our model provider on a deeper investigation into the cause. We will update again when we have identified a mitigation.</p><p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>17:37</var> UTC</small><br><strong>Update</strong> - We are testing other internal mitigations so that we can return to the higher maximum input length. We are still working with our upstream model provider to understand the contributing factors for this sudden decrease in input limits.</p><p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>16:49</var> UTC</small><br><strong>Update</strong> - We are experiencing a service regression for the Gemini 2.5 Pro model in Copilot Chat, VS Code and other Copilot products. The maximum input length of Gemini 2.5 prompts been decreased. Long prompts or large context windows may result in errors. This is due to an issue with an upstream model provider. We are working with them to resolve the issue.<br /><br />Other models are available and working as expected.</p><p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>16:43</var> UTC</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Copilot</p> tag:us.githubstatus.com,2005:Incident/26592342 2025-09-29T19:12:42Z 2025-10-06T23:22:00Z US - Disruption with Gemini 2.5 Pro and Gemini 2.0 Flash in Copilot <p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>19:12</var> UTC</small><br><strong>Resolved</strong> - On September 29, 2025, between 17:53 and 18:42 UTC, the Copilot service experienced a degradation of the Gemini 2.5 model due to an issue with our upstream provider. Approximately 24% of requests failed, affecting 56% of users during this period. No other models were impacted.<br /><br />GitHub notified the upstream provider of the problem as soon as it was detected. The issue was resolved after the upstream provider rolled back a recent change that caused the disruption. GitHub will continue to enhance our monitoring and alerting systems to reduce the time it takes to detect and mitigate similar issues in the future.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>19:12</var> UTC</small><br><strong>Update</strong> - The upstream model provided has resolved the issue and we are seeing full availability for Gemini 2.5 Pro and Gemini 2.0 Flash.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>18:40</var> UTC</small><br><strong>Update</strong> - We are experiencing degraded availability for the Gemini 2.5 Pro & Gemini 2.0 Flash models in Copilot Chat, VS Code and other Copilot products. This is due to an issue with an upstream model provider. We are working with them to resolve the issue.<br /><br />Other models are available and working as expected.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>18:39</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p> tag:us.githubstatus.com,2005:Incident/26591282 2025-09-29T17:33:52Z 2025-10-06T19:58:46Z US - Disruption with some GitHub services <p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>17:33</var> UTC</small><br><strong>Resolved</strong> - On September 29, 2025 between 16:26 UTC and 17:33 UTC the Copilot API experienced a partial degradation causing intermittent erroneous 404 responses for an average of 0.2% of GitHub MCP server requests, peaking at times around 2% of requests. The issue stemmed from an upgrade of an internal dependency which exposed a misconfiguration in the service.<br /><br />We resolved the incident by rolling back the upgrade to address the misconfiguration. We fixed the configuration issue and will improve documentation and rollout process to prevent similar issues.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>17:28</var> UTC</small><br><strong>Update</strong> - Customers are getting 404 responses when connecting to the GitHub MCP server. We have reverted a change we believe is contributing to the impact, and are seeing resolution in deployed environments.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>16:45</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p> tag:us.githubstatus.com,2005:Incident/26538993 2025-09-24T09:18:32Z 2025-09-29T15:40:36Z US - Claude Opus 4 is experiencing degraded performance <p><small>Sep <var data-var='date'>24</var>, <var data-var='time'>09:18</var> UTC</small><br><strong>Resolved</strong> - Anthropic reported degraded Claude Opus 4 and 4.1 through their status page. We were seeing this affect our model(s) performance, where we activated the relevant warning messages directing users to try other models. This was resolved after an hour on Anthropic's end, and shortly after we saw recovery, returning to our baseline success rate.</p><p><small>Sep <var data-var='date'>24</var>, <var data-var='time'>09:16</var> UTC</small><br><strong>Update</strong> - Between around 8:16 UTC and 8:51 UTC we saw elevated errors on Claude Opus 4 and Opus 4.1, up to 49% of requests were failing. This has recovered to around 4% of requests failing, we are monitoring recovery.</p><p><small>Sep <var data-var='date'>24</var>, <var data-var='time'>09:08</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p> tag:us.githubstatus.com,2005:Incident/26534606 2025-09-24T00:26:29Z 2025-10-01T21:21:17Z US - Incident with Copilot <p><small>Sep <var data-var='date'>24</var>, <var data-var='time'>00:26</var> UTC</small><br><strong>Resolved</strong> - Between 20:06 UTC September 23 and 04:58 UTC September 24, 2025, the Copilot service experienced degraded availability for Claude Sonnet 4 and 3.7 model requests.<br /><br />During this period, 0.46% of Claude 4 requests and 7.83% of Claude 3.7 requests failed.<br /><br />The reduced availability resulted from Copilot disabling routing to an upstream provider that was experiencing issues and reallocating capacity to other providers to manage requests for Claude Sonnet 3.7 and 4.<br />We are continuing to investigate the source of the issues with this provider and will provide an update as more information becomes available.</p><p><small>Sep <var data-var='date'>24</var>, <var data-var='time'>00:26</var> UTC</small><br><strong>Update</strong> - The issues with our upstream model provider have been resolved, and Claude Sonnet 3.7 and Claude Sonnet 4 are once again available in Copilot Chat, VS Code and other Copilot products.<br /><br />We will continue monitoring to ensure stability, but mitigation is complete.<br /></p><p><small>Sep <var data-var='date'>23</var>, <var data-var='time'>22:22</var> UTC</small><br><strong>Update</strong> - We are experiencing degraded availability for the Claude Sonnet 3.7 and Claude Sonnet 4 model in Copilot Chat, VS Code and other Copilot products. This is due to an issue with an upstream model provider. We are working with them to resolve the issue.<br /><br />Other models are available and working as expected.</p><p><small>Sep <var data-var='date'>23</var>, <var data-var='time'>22:22</var> UTC</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Copilot</p> tag:us.githubstatus.com,2005:Incident/26450309 2025-09-15T18:28:35Z 2025-09-18T23:23:31Z US - Disruption with some GitHub services <p><small>Sep <var data-var='date'>15</var>, <var data-var='time'>18:28</var> UTC</small><br><strong>Resolved</strong> - On September 15th between 17:55 and 18:20 UTC, Copilot experienced degraded availability for all features. This was due a partial deployment of a feature flag to a global rate limiter. The flag triggered behavior that unintentionally rate limited all requests, resulting in 100% of them returning 403 errors. The issue was resolved by reverting the feature flag which resulted in immediate recovery.<br /><br />The root cause of the incident was from an undetected edge case in our rate limiting logic. The flag was meant to scale down rate limiting for a subset of users, but unintentionally put our rate limiting configuration into an invalid state.<br /><br />To prevent this from happening again, we have addressed the bug with our rate limiting. We are also adding additional monitors to detect anomalies in our traffic patterns, which will allow us to identify similar issues during future deployments. Furthermore, we are exploring ways to test our rate limit scaling in our internal environment to enhance our pre-production validation process.<br /></p><p><small>Sep <var data-var='date'>15</var>, <var data-var='time'>18:21</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p> tag:us.githubstatus.com,2005:Incident/26259510 2025-08-27T21:27:59Z 2025-08-29T19:03:28Z US - Disruption with some GitHub services <p><small>Aug <var data-var='date'>27</var>, <var data-var='time'>21:27</var> UTC</small><br><strong>Resolved</strong> - On August 27, 2025 between 20:35 and 21:17 UTC, Copilot, Web and REST API traffic experienced degraded performance. Copilot saw an average of 36% of requests fail with a peak failure rate of 77%. Approximately 2% of all non-Copilot Web and REST API traffic requests failed.<br /><br />This incident occurred after a stale schema cache was used following a database migration. This led to a large number of failed queries. At 21:15 UTC, we applied a fix and by 21:17 UTC, all services had fully recovered.<br /><br />We have implemented a block for this failure mode as an immediate solution and are actively working to add safeguards to prevent similar issues from occurring in the future.</p><p><small>Aug <var data-var='date'>27</var>, <var data-var='time'>21:27</var> UTC</small><br><strong>Update</strong> - API Requests and Issues are operating normally.</p><p><small>Aug <var data-var='date'>27</var>, <var data-var='time'>21:25</var> UTC</small><br><strong>Update</strong> - We've discovered the cause of the service disruption and applied a mitigation.</p><p><small>Aug <var data-var='date'>27</var>, <var data-var='time'>20:58</var> UTC</small><br><strong>Update</strong> - API Requests and Issues are experiencing degraded performance. We are continuing to investigate.</p><p><small>Aug <var data-var='date'>27</var>, <var data-var='time'>20:55</var> UTC</small><br><strong>Update</strong> - The team is aware of the root cause of this issue and is working to mitigate the issue quickly.</p><p><small>Aug <var data-var='date'>27</var>, <var data-var='time'>20:49</var> UTC</small><br><strong>Update</strong> - API Requests is experiencing degraded availability. We are continuing to investigate.</p><p><small>Aug <var data-var='date'>27</var>, <var data-var='time'>20:48</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p> tag:us.githubstatus.com,2005:Incident/25873106 2025-07-21T07:50:52Z 2025-07-25T16:57:35Z US - Some Copilot Models Experiencing Degraded Performance <p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>07:50</var> UTC</small><br><strong>Resolved</strong> - On July 21, 2025, between 07:20 UTC and 08:00 UTC, the Copilot service experienced degraded availability for Claude 4 requests. 2% of Claude 4 requests failed during this time. The issue was caused by an upstream problem affecting our ability to serve requests.<br />We mitigated by rerouting capacity and monitoring recovery. We are improving detection and mitigation to reduce future impact.</p><p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>07:36</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p> tag:us.githubstatus.com,2005:Incident/25820185 2025-07-16T08:58:41Z 2025-07-22T17:27:18Z US - Disruption with some GitHub services <p><small>Jul <var data-var='date'>16</var>, <var data-var='time'>08:58</var> UTC</small><br><strong>Resolved</strong> - On July 16, 2025, between 05:20 UTC and 08:30 UTC, the Copilot service experienced degraded availability for Claude 3.7 requests. Around 10% of Claude 3.7 requests failed during this time. The issue was caused by an upstream problem affecting our ability to serve requests.<br />We mitigated by rerouting capacity and monitoring recovery. We are improving detection and mitigation to reduce future impact.</p><p><small>Jul <var data-var='date'>16</var>, <var data-var='time'>08:45</var> UTC</small><br><strong>Update</strong> - We have seen recovery on our provider's side but have not yet confirmed if the issue is fully resolved. We will update our status in the next 20 minutes as we know more.</p><p><small>Jul <var data-var='date'>16</var>, <var data-var='time'>08:21</var> UTC</small><br><strong>Update</strong> - We are experiencing degraded availability for the Claude 3.7 Sonnet model in Copilot Chat, VS Code and other Copilot products. This is due to an issue with an upstream model provider. We are working with them to resolve the issue.</p><p><small>Jul <var data-var='date'>16</var>, <var data-var='time'>08:16</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p> tag:us.githubstatus.com,2005:Incident/25673660 2025-07-03T07:12:02Z 2025-07-03T20:11:25Z US - Disruption with some GitHub services <p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>07:12</var> UTC</small><br><strong>Resolved</strong> - On 7/3/2025, between 5:21:17 AM and 7:11:49 AM UTC, customers were prevented from SSO authorizing Personal Access Tokens and SSH keys via the GitHub UI. Approximately 1300 users were impacted.<br /><br />A code change modified the content type of the response returned by the server, causing a lazily-loaded dropdown to fail to render, prohibiting the user from proceeding to authorize. No authorization systems were impacted during the incident, only the UI component. We mitigated the incident by reverting the code change that introduced the problem.<br /><br />We are making improvements to our release process and test coverage to catch this class of error earlier in our deployment pipeline. Further, we are improving monitoring to reduce our time to detection and mitigation of issues like this one in the future.<br /></p><p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>07:11</var> UTC</small><br><strong>Update</strong> - The rollback has been deployed successfully on all environments. Customers should now be able to SSO authorize their Classic Personal Access Tokens and SSH keys on their GitHub organizations.</p><p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>06:46</var> UTC</small><br><strong>Update</strong> - The root cause for the rendering bug that prevented customers from SSO authorizing Personal Access Tokens and SSH keys has started rolling out. We are continuously monitoring this rollback.</p><p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>06:07</var> UTC</small><br><strong>Update</strong> - We have identified the root cause for the rendering bug that prevented customers from SSO authorizing Personal Access Tokens and SSH keys.The changes that caused the issue are being rolled back.</p><p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>05:45</var> UTC</small><br><strong>Update</strong> - We are investigating an issue with SSO authorizing Classic Personal Access Tokens and SSH keys.</p><p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>05:39</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p> tag:us.githubstatus.com,2005:Incident/25662977 2025-07-02T10:16:27Z 2025-07-08T19:55:59Z US - Disruption with some GitHub services <p><small>Jul <var data-var='date'> 2</var>, <var data-var='time'>10:16</var> UTC</small><br><strong>Resolved</strong> - On July 2nd, 2025, between approximately 08:40 and 10:16 UTC, the Copilot service experienced degradation due to an infrastructure issue which impacted the Claude Sonnet 4 model, leading to a spike in errors. No other models were impacted.<br /><br />The issue was mitigated by rebalancing load within our infrastructure. GitHub is working to further improve the resiliency of the service to prevent similar incidents in the future.</p><p><small>Jul <var data-var='date'> 2</var>, <var data-var='time'>10:16</var> UTC</small><br><strong>Update</strong> - We are no longer experiencing degradation—Claude Sonnet 4 is once again available in Copilot Chat and across IDE integrations.<br /><br />We will continue monitoring to ensure stability, but mitigation is complete.</p><p><small>Jul <var data-var='date'> 2</var>, <var data-var='time'>09:57</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p> tag:us.githubstatus.com,2005:Incident/25641813 2025-06-30T19:55:38Z 2025-07-07T18:49:30Z US - Disruption with Claude 3.7 Sonnet in Copilot Chat <p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>19:55</var> UTC</small><br><strong>Resolved</strong> - On June 30th, 2025, between approximately 18:20 and 19:55 UTC, the Copilot service experienced a degradation of the Claude Sonnet 3.7 model due to an issue with our upstream provider. Users encountered elevated error rates when using Claude Sonnet 3.7. No other models were impacted.<br /><br />The issue was resolved by a mitigation put in place by our provider. GitHub is working with our provider to further improve the resiliency of the service to prevent similar incidents in the future.<br /></p><p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>19:55</var> UTC</small><br><strong>Update</strong> - The issues with our upstream model provider have been resolved, and Claude Sonnet 3.7 is once again available in Copilot Chat and across IDE integrations [VSCode, Visual Studio, JetBrains].<br />We will continue monitoring to ensure stability, but mitigation is complete.</p><p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>19:14</var> UTC</small><br><strong>Update</strong> - We are experiencing degraded availability for the Claude 3.7 Sonnet model in Copilot Chat, VS Code and other Copilot products. This is due to an issue with an upstream model provider. We are working with them to resolve the issue.<br /><br />Other models are available and working as expected.</p><p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>19:13</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p> tag:us.githubstatus.com,2005:Incident/25612499 2025-06-28T01:26:06Z 2025-06-28T01:26:06Z US - Disruption with the GitHub mobile android application <p><small>Jun <var data-var='date'>28</var>, <var data-var='time'>01:26</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>12:56</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p> tag:us.githubstatus.com,2005:Incident/25581905 2025-06-24T12:31:20Z 2025-06-25T08:02:10Z US - Repository Navigation Bar Missing <p><small>Jun <var data-var='date'>24</var>, <var data-var='time'>12:31</var> UTC</small><br><strong>Resolved</strong> - On June 24, 2025 from 09:42 UTC to 12:11 UTC, the Repository UI was degraded for customers using GitHub Enterprise with Data Residency. During this timeframe, the navigational elements on repository pages would not have been visible, impacting roughly 600 users.<br /><br />The issue was caused by a change that referenced a new resource for this information, which had not yet been configured for the GitHub Enterprise with Data Residency. Additionally, the change had incomplete coverage of the guardrails we employ to prevent these types of exceptions. Due to safe resiliency practices for these elements, the resulting impact was the loss of those elements on the page. A rollback of the impacting change was performed to restore full functionality to these pages.<br /><br />To prevent future incidents like this, we are investing in additional monitoring for specific errors resulting from these workflows, expanded coverage of end-to-end testing, and alerting when a sharp increase in resilient fallback behavior happens.</p><p><small>Jun <var data-var='date'>24</var>, <var data-var='time'>12:26</var> UTC</small><br><strong>Update</strong> - We have deployed a mitigation to the affected instances, and telemetry indicates the error is no longer occurring.</p><p><small>Jun <var data-var='date'>24</var>, <var data-var='time'>11:44</var> UTC</small><br><strong>Update</strong> - A mitigation is in progress of rolling out to the affected instances. We expect customers to see mitigation within the next hour.</p><p><small>Jun <var data-var='date'>24</var>, <var data-var='time'>11:34</var> UTC</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for the Repositories UI. We have identified that the navigation bar is missing in GitHub Enterprise Cloud with data residency instances for the repositories related pages and are currently attempting a mitigation.</p> tag:us.githubstatus.com,2005:Incident/25528326 2025-06-18T23:13:52Z 2025-06-26T16:49:40Z US - Disruption with some GitHub services <p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>23:13</var> UTC</small><br><strong>Resolved</strong> - On June 18, 2025 between 22:20 UTC and 23:00 UTC the Claude Sonnet 3.7 and Claude Sonnet 4 models for GitHub Copilot Chat experienced degraded performance. During the impact, some users would receive an immediate error when making a request to a Claude model. This was due to upstream errors with one of our model providers, which have since been resolved. We mitigated the impact by disabling the affected provider endpoints to reduce user impact, redirecting Claude Sonnet requests to additional partners.<br /><br />We are working to update our incident response playbooks for infrastructure provider outages and improve our monitoring and alerting systems to reduce our time to detection and mitigation of issues like this one in the future.</p><p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>22:42</var> UTC</small><br><strong>Update</strong> - We are experiencing degraded availability for the Claude 4 model in Copilot Chat, VS Code and other Copilot products. This is due to an issue with an upstream model provider. We are working with them to resolve the issue.<br /><br />Other models are available and working as expected. We recommend using Claude 3.7 as an alternative.</p><p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>22:40</var> UTC</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Copilot</p> tag:us.githubstatus.com,2005:Incident/25275515 2025-05-27T13:31:58Z 2025-05-29T05:12:57Z US - Incident with Actions <p><small>May <var data-var='date'>27</var>, <var data-var='time'>13:31</var> UTC</small><br><strong>Resolved</strong> - On May 27, 2025, between 09:31 UTC and 13:31 UTC, some Actions jobs experienced failures uploading to and downloading from the Actions Cache service. During the incident, 6% of all workflow runs couldn’t upload or download cache entries from the service, resulting in a non-blocking warning message in the logs and performance degradation. The disruption was caused by an infrastructure update related to the retirement of a legacy service, which unintentionally impacted Cache service availability. We resolved the incident by reverting the change and have since implemented a permanent fix to prevent recurrence.<br /><br />We are improving our configuration change processes by introducing additional end-to-end tests to cover the identified gaps, and implementing deployment pipeline improvements to reduce mitigation time for similar issues in the future.</p><p><small>May <var data-var='date'>27</var>, <var data-var='time'>13:03</var> UTC</small><br><strong>Update</strong> - Mitigation is applied and we’re seeing signs of recovery. We’re monitoring the situation until the mitigation is applied to all affected repositories.</p><p><small>May <var data-var='date'>27</var>, <var data-var='time'>12:29</var> UTC</small><br><strong>Update</strong> - We are experiencing degradation with the GitHub Actions cache service and are working on applying the appropriate mitigations.</p><p><small>May <var data-var='date'>27</var>, <var data-var='time'>12:26</var> UTC</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Actions</p> tag:us.githubstatus.com,2005:Incident/25196536 2025-05-20T20:02:03Z 2025-05-23T23:18:34Z US - Incident with Copilot <p><small>May <var data-var='date'>20</var>, <var data-var='time'>20:02</var> UTC</small><br><strong>Resolved</strong> - On May 20, 2025, between 18:18 UTC and 19:53 UTC, Copilot Code Completions were degraded in the Americas. On average the error rate was 50% of requests to the service in the affected region. This was due to a misconfiguration in load distribution parameters after a scale down operation.<br /><br />We mitigated the incident by addressing the misconfiguration.<br /><br />We are working to improve our automated failover and load balancing mechanisms to reduce our time to detection and mitigation of issues like this one in the future.</p><p><small>May <var data-var='date'>20</var>, <var data-var='time'>20:02</var> UTC</small><br><strong>Update</strong> - The issues with Copilot Code Completions have been resolved.<br />We will continue monitoring to ensure stability, but mitigation is complete.</p><p><small>May <var data-var='date'>20</var>, <var data-var='time'>20:01</var> UTC</small><br><strong>Update</strong> - Copilot is operating normally.</p><p><small>May <var data-var='date'>20</var>, <var data-var='time'>19:43</var> UTC</small><br><strong>Update</strong> - We are experiencing degraded availability for Copilot Code Completions in the America’s.<br />We are working on resolving the issue.</p><p><small>May <var data-var='date'>20</var>, <var data-var='time'>19:37</var> UTC</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Copilot</p> tag:us.githubstatus.com,2005:Incident/25189635 2025-05-20T16:08:37Z 2025-05-27T22:25:14Z US - Elevated error rates for Claude Sonnet 3.7 <p><small>May <var data-var='date'>20</var>, <var data-var='time'>16:08</var> UTC</small><br><strong>Resolved</strong> - On May 20, 2025, between 12:09 PM UTC and 4:07 PM UTC, the GitHub Copilot service experienced degraded availability, specifically for the Claude Sonnet 3.7 model. During this period, the success rate for Claude Sonnet 3.7 requests was highly variable, down to approximately 94% during the most severe spikes. Other models remained available and working as expected throughout the incident.<br />The issue was caused by capacity constraints in our model processing infrastructure that affected our ability to handle the large volume of Claude Sonnet 3.7 requests.<br />We mitigated the incident by rebalancing traffic across our infrastructure, adjusting rate limits, and working with our infrastructure teams to resolve the underlying capacity issues. We are working to improve our infrastructure redundancy and implementing more robust monitoring to reduce detection and mitigation time for similar incidents in the future.</p><p><small>May <var data-var='date'>20</var>, <var data-var='time'>16:08</var> UTC</small><br><strong>Update</strong> - Copilot is operating normally.</p><p><small>May <var data-var='date'>20</var>, <var data-var='time'>16:07</var> UTC</small><br><strong>Update</strong> - The issues with our upstream model provider have been resolved, and Claude Sonnet 3.7 is once again available in Copilot Chat, VS Code and other Copilot products.<br /><br />We will continue monitoring to ensure stability, but mitigation is complete.</p><p><small>May <var data-var='date'>20</var>, <var data-var='time'>14:59</var> UTC</small><br><strong>Update</strong> - We are continuing to work with our model providers on mitigations to increase the success rate of Sonnet 3.7 requests made via Copilot.</p><p><small>May <var data-var='date'>20</var>, <var data-var='time'>14:15</var> UTC</small><br><strong>Update</strong> - We’re still working with our model providers on mitigations to increase the success rate of Sonnet 3.7 requests made via Copilot.</p><p><small>May <var data-var='date'>20</var>, <var data-var='time'>13:33</var> UTC</small><br><strong>Update</strong> - We are experiencing degraded availability for the Claude Sonnet 3.7 model in Copilot Chat, VS Code and other Copilot products. This is due to an issue with an upstream model provider. We are working with them to resolve the issue.<br /><br />Other models are available and working as expected.</p><p><small>May <var data-var='date'>20</var>, <var data-var='time'>13:10</var> UTC</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Copilot</p>