Python Infrastructure Status - Incident History

PyPI Search Index Degraded

Tue, 29 Jul 2025 19:58:32 +0000

Jul 29, 19:58 UTC
Resolved - This incident has been resolved.

Jul 29, 14:41 UTC
Monitoring - The PyPI search index has degraded and is not returning results.
A reindexing job has begun, and results will be available once it completes.

PyPI Database Upgrades

Wed, 11 Jun 2025 19:23:02 +0000

Jun 11, 19:23 UTC
Completed - This maintenance is complete! All metrics have returned to baseline.

Jun 11, 19:16 UTC
Verifying - All database tasks are complete and we are now monitoring for stability post-upgrade. Services should be back online in full.

Jun 11, 19:10 UTC
Update - The core upgrade is complete, we are awaiting the completion of post-upgrade statistics updates which are taking longer than anticipated.

Jun 11, 19:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jun 11, 18:50 UTC
Scheduled - PyPI will be undergoing maintenance for a major version database upgrade. We anticipate up to 5-10 minutes of disruption when the cut-over occurs.

PyPI Partial Outage

Tue, 27 May 2025 12:53:04 +0000

May 27, 12:53 UTC
Resolved - We have not seen a return of the traffic that caused this incident after removing the temporary block. We will open a new incident if we need to take action again.

May 25, 12:16 UTC
Update - We have removed the temporary block and are continuing to monitor.

May 24, 23:54 UTC
Monitoring - Our block is in place and systems are stable, but we are not satisfied with having to block such a generic user-agent. If you are aware of a system that began a mass crawl of PyPI's JSON release endpoints starting at around 22:05 UTC 2025-05-25, please contact admin@pypi.org.

May 24, 23:37 UTC
Update - We have identified a flood of requests from a cluster using the `python-requests/2.32.3` User-Agent, and attempting to hit all JSON project/releasese en masse.

This causes excessive cache misses, which are overwhelming our backends. A temporary block has been put in place.

May 24, 23:11 UTC
Identified - We have identified a large increase in CDN cache misses causing excess load on our backends. We are working to determine the reason for this increase.

May 24, 22:57 UTC
Investigating - We are working to investigate a partial outage in the PyPI backends.

PyPI Elevated Error Rates

Thu, 24 Apr 2025 12:18:39 +0000

Apr 24, 12:18 UTC
Resolved - This incident is resolved. We're going to do some further investigation to understand what led to the lock contention.

Apr 24, 12:09 UTC
Monitoring - The locking process has been terminated and we're monitoring as things recover.

Apr 24, 12:09 UTC
Identified - We've identified an issue with database contention arising from locking.

Apr 24, 12:02 UTC
Investigating - We're investigating elevated error rates for pypi.org

PyPI Simple and JSON API Outage

Tue, 08 Apr 2025 11:00:37 +0000

Apr 8, 11:00 UTC
Resolved - This incident has been resolved.

Apr 8, 10:31 UTC
Update - Errors have subsided, and we are monitoring as things continue to recover.

Apr 8, 10:26 UTC
Monitoring - We've increased capacity in the worker pools that service /simple and /json and are monitoring as things recover.

Apr 8, 10:23 UTC
Identified - We are working to resolve an outage impacting our /simple and /json apis.

Download file corruption

Tue, 11 Mar 2025 13:39:47 +0000

Mar 11, 13:39 UTC
Resolved - This incident has been resolved.

Mar 11, 13:05 UTC
Update - We are continuing to monitor for any further issues.

Mar 11, 12:50 UTC
Monitoring - Our CDN provider has reported that the node in question has been removed from service. We are monitoring for continued reports.

Mar 11, 12:47 UTC
Identified - This issue appears to be isolated to a single edge node in our CDN provider's network. They have validated the issue and are working to resolve.

Mar 11, 12:13 UTC
Investigating - We are investigating issues with files from files.pythonhosted.org being served with corrupted data. A support ticket has been opened with our CDN provider and we are awaiting their response.

Infrastructure Outage

Fri, 13 Dec 2024 15:09:31 +0000

Dec 13, 15:09 UTC
Resolved - This incident has been resolved.

Dec 13, 14:58 UTC
Monitoring - We have begun seeing recovery across all services as of 14:44 UTC, we are monitoring for stability and ensuring all services are healthy.

Dec 13, 14:45 UTC
Update - We are continuing to work on a fix for this issue.

Dec 13, 14:23 UTC
Update - We are continuing to work on a fix for this issue.

Dec 13, 14:00 UTC
Identified - Private networking in our datacenter for these services is currently intermittent, we have opened a ticket with the provider requesting assistance.

Dec 13, 13:59 UTC
Update - We are continuing to investigate this issue.

Dec 13, 13:48 UTC
Update - We are continuing to investigate this issue.

Dec 13, 13:44 UTC
Investigating - We are currently investigating an infrastructure outage impacting python.org/downloads backends, docs.python.org backends, bugs.python.org backends, wiki.python.org backends, as well as other services.

PyPI Database Maintenance

Wed, 13 Nov 2024 19:32:39 +0000

Nov 13, 19:32 UTC
Completed - The scheduled maintenance has been completed.

Nov 13, 19:30 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Nov 13, 19:15 UTC
Scheduled - PyPI will be undergoing a brief period of database maintenance to ensure that a set of migrations can complete without contention.

During this time all PyPI backends will be temporarily unavailable and uploads will be disabled. Some requests that are not cached in our CDN may fail, particularly for recently updated projects and pages.

PyPI Simple Index Timing Out

Wed, 21 Aug 2024 20:23:44 +0000

Aug 21, 20:23 UTC
Resolved - This incident has been resolved.

Aug 21, 19:53 UTC
Identified - An error serving /simple requests has caused timeouts for some requests. We have identified the issue and are working on a revert.

Issue with uploads

Fri, 09 Aug 2024 19:26:53 +0000

Aug 9, 19:26 UTC
Resolved - Uploads are now being accepted properly.

Aug 9, 19:23 UTC
Monitoring - A fix has been implemented and we are monitoring the results.

Aug 9, 19:08 UTC
Identified - The issue has been identified and a fix is being implemented.

Aug 9, 18:22 UTC
Investigating - We are currently investigating an issue with PyPI uploads causing uploads to fail.

PyPI Backend Outage

Wed, 31 Jul 2024 03:34:54 +0000

Jul 31, 03:34 UTC
Resolved - This incident has been resolved.

Jul 31, 03:21 UTC
Investigating - We are currently investigating an issue with the PyPI backend due to a recent deploy.

PyPI Database Upgrades

Fri, 12 Jul 2024 16:18:57 +0000

Jul 12, 16:18 UTC
Resolved - This maintenance is complete.

Jul 12, 14:28 UTC
Identified - We are performing upgrades to PyPI's database storage which may impact performance and availability of the service as they rollout.

PyPI Database Upgrades

Sat, 08 Jun 2024 14:52:25 +0000

Jun 8, 14:52 UTC
Resolved - All maintenance is complete and telemetry around database and application performance have stabilized. This incident is complete.

Jun 8, 14:42 UTC
Update - All upgrade steps are complete and performance is stabilizing. A final maintenance task is taking place in the background which will minimally impact performance until it is complete.

Jun 8, 14:26 UTC
Update - We are in the final steps of the upgrade and should see performance and availability stabilize in the next 30 minutes.

Jun 8, 13:58 UTC
Monitoring - Underlying upgrades are complete, we are running database statistics updates to optimize performance.

Jun 8, 13:43 UTC
Identified - We will be performing database maintenance including version upgrades and storage re-configuration. This will result in several brief outages for the service.

Outage in python.org and us.pycon.org

Wed, 05 Jun 2024 16:05:50 +0000

Jun 5, 16:05 UTC
Resolved - This incident has been resolved.

Jun 5, 15:57 UTC
Monitoring - A cascading failure due to an expired certificate led to this outage. The certificate has been expired and systems are coming back online.

Jun 5, 15:38 UTC
Identified - An outage in our infrastructure is impacting availability of us.pycon.org and python.org

PyPI New User Registration and New Project Creation Temporarily Suspended.

Thu, 28 Mar 2024 12:56:27 +0000

Mar 28, 12:56 UTC
Resolved - This incident has been resolved.

Mar 28, 02:16 UTC
Identified - We have temporarily suspended new project creation and new user registration to mitigate an ongoing malware upload campaign.

PyPI workers currently paused

Tue, 12 Mar 2024 20:41:19 +0000

Mar 12, 20:41 UTC
Resolved - This incident has been resolved.

Mar 12, 20:19 UTC
Monitoring - Workers have been restarted and are clearing backlog of enqueued tasks.

Mar 12, 20:00 UTC
Identified - A maintenance event requires us to pause our workers, which will delay cache purges and email delivery. We anticipate this will persist for up to 30 minutes.

Site wide issue in PSF Infrastructure

Mon, 12 Feb 2024 21:06:16 +0000

Feb 12, 21:06 UTC
Resolved - This incident has been resolved.

Feb 12, 18:07 UTC
Monitoring - Our new internal CA has been rolled out across the infrastructure and we are monitoring for remaining service impact.

Feb 12, 18:00 UTC
Identified - Our internal Certificate Authority certificate has expired causing cascading outages across multiple services. An updated CA has been created and we are working to roll it out to restore service.

Feb 12, 17:50 UTC
Investigating - A site wide issue is impacting many services hosted in the PSF infrastructure.

bugs.python.org under maintenance

Fri, 19 Jan 2024 12:18:42 +0000

Jan 19, 12:18 UTC
Resolved - Maintenance is complete.

Jan 19, 12:04 UTC
Monitoring - Maintenance and migration is complete. DNS may take some time to propagate. We will monitor as that occurs.

Jan 19, 11:31 UTC
Identified - bugs.python.org (and bugs.jython.org, and issues.roundup-tracker.org) is currently under maintenance.

PyPI new user registration temporarily suspended

Tue, 02 Jan 2024 15:59:07 +0000

Jan 2, 15:59 UTC
Resolved - New user registrations are now permitted.

Dec 27, 20:11 UTC
Monitoring - New user registration on PyPI is temporarily suspended. The volume of malicious users and malicious projects being created on the index in the past week has outpaced our ability to respond to it in a timely fashion, especially with multiple PyPI administrators on leave.

PyPI new user registration temporarily suspended

Mon, 04 Dec 2023 13:47:36 +0000

Dec 4, 13:47 UTC
Resolved - This incident has been resolved.

Nov 24, 23:14 UTC
Monitoring - New user registration on PyPI is temporarily suspended. The volume of malicious users and malicious projects being created on the index in the past week has outpaced our ability to respond to it in a timely fashion, especially with multiple PyPI administrators on leave.

Database Storage Latencies

Tue, 17 Oct 2023 22:44:39 +0000

Oct 17, 22:44 UTC
Resolved - This incident has been resolved.

Oct 17, 19:44 UTC
Identified - A burst balance for our cloud storage disks that host our databases was exhausted. We are going to migrate to storage that will allow us to respond by adding more IOPS.

Oct 17, 19:32 UTC
Investigating - We are investigating and responding to widespread issues with databases in our infrastructure showing poor performance due to cloud block storage.

Reports of failed connections from us-east-2

Wed, 20 Sep 2023 22:59:14 +0000

Sep 20, 22:59 UTC
Resolved - This incident has been resolved.

Sep 20, 22:17 UTC
Monitoring - Fastly has routed traffic away from an impacted point of presence.
We will monitor for further impact.

Sep 20, 21:25 UTC
Investigating - We're receiving reports from users that attempts to download packages from PyPIs files archive may be failing when clients use IPv6 as their transport.

Our network provider Fastly is also investigating the issue, see their status page: https://www.fastlystatus.com/incident/376079

Intermittent outages in PyPI backends.

Thu, 17 Aug 2023 15:08:38 +0000

Aug 17, 15:08 UTC
Resolved - This incident has been resolved.

Aug 17, 15:05 UTC
Monitoring - The migration has been completed and we are monitoring to ensure stability.

Aug 17, 15:00 UTC
Identified - A slow migration PyPI admins are attempting to apply is causing database contention leading to intermittent outages in the PyPI backends. We are working on mitigating the impact to end users.

Some PyPI web requests failing.

Tue, 01 Aug 2023 20:09:01 +0000

Aug 1, 20:09 UTC
Resolved - This incident has been resolved.

Aug 1, 20:03 UTC
Monitoring - The revert has been deployed and we are monitoring the result.

Aug 1, 19:56 UTC
Update - We have initiated a rollback of the change that introduced this outage.

Aug 1, 19:42 UTC
Identified - An update to our database connection library has caused some web requests including uploads to fail. We have identified the issue and are working to resolve.

upload.pypi.org migration.

Mon, 17 Jul 2023 13:01:59 +0000

Jul 17, 13:01 UTC
Completed - The scheduled maintenance has been completed.

Jul 17, 13:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jul 17, 12:51 UTC
Scheduled - Our endpoint for upload.pypi.org will undergo a migration. During this time there may be short periods of unavailability while TLS is provisioned.