ItInsightsWorld CloudComputing Review: Central US Reliability Work, GitHub Actions Capacity, and September Incident Retrospectives

What this review adds for cloud operators

Cloud teams reading migration updates usually care about risk per customer, not task lists. This ItInsightsWorld CloudComputing note uses the recent public record to separate progress from exposure: two months of work across August and September produced database relief, more headroom for GitHub Actions, and better containment, while the August 17 github.com outage, later incidents, and an unfinished capacity plan show why measurement must follow availability, capacity, then features.

For readers tracing publication continuity, the earlier note only reached content through August, and a September edition did not appear; separate public messages did cover the August 17 github.com outage. Across August and September, the platform reduced pressure on shared databases, introduced overload controls, expanded pull request containment, gave GitHub Actions more operating room, and moved additional production workloads onto Azure. Serious incidents, a capacity pause, and a missed migration target still show that the work is unfinished.

Regional migration and traffic share measured on October 7

Progress snapshot

On October 7, the deployment tally showed 120 of 160 production services in Central US and 13 of 51 primary MySQL databases moved.

How to read the traffic mix

The Azure platform carried 65.4% of read traffic for the core GitHub application, 48.3% of Git read traffic, and 80.2% of stream-processing traffic. Because these workloads differ, a cloud architect should not add them into one migration score.

Capacity pauses and gaps

Central US deployment steps and traffic ramps stopped on September 11 after uneven capacity across availability zones.

Temporary relief allowed permanent ramps to restart on September 18. The team is still working toward opening all available capacity by the end of October.

The more aggressive September 30 target, which called for every production service to be deployed in Central US, was not met. During a planned database failover, an automated emergency process disconnected Central US database copies and disrupted customers in that region; that process was disabled during planned handoffs while recovery improvements are pursued.

Database cleanup effects outside the region move

Apart from the regional migration, 24 authentication-core tables were relocated away from the oldest shared database used by GitHub, cutting approximately one million queries per second from its replicas.

In the cluster supporting issues and pull requests, approximately 980 million unnecessary edit-history rows were removed, creating an estimated 1.8 terabytes of recoverable storage.

Weekly deadlock counts dropped approximately 77% from their June peak, and ongoing cleanup now stops the same unbounded growth from recurring.

Pull request containment and overload protection

Half of sampled pull request traffic can now keep running with limited features when a shared dependency fails.

Production evidence showed that several read operations stayed available during a shared-database incident.

Database traffic routing can add capacity automatically during spikes; one group grew from 14 to 19 instances during a September 23 rise in pull request activity.

A production overload control lets responders shield higher-priority API and web workflows by rejecting lower-priority requests before wider saturation occurs. Documentation and training for operators are complete.

GitHub Actions headroom in a comparison table

The table separates production changes from a separate infrastructure cache, then lists the measured signal and the reported reduction.

Source of change	Signal	Effect
Production changes	Measured Redis operations	reduced by approximately 28%
Production changes	Relative CPU	reduced by approximately 30%
Separate infrastructure cache	Redis bandwidth	reduced by 84%
Separate infrastructure cache	Average response time	reduced by 66%
Separate infrastructure cache	Median response time for the primary inventory request	reduced by 96%
Separate infrastructure cache	Queued requests	reduced by 95%

Incident retrospectives as cloud risk cases

September 13 09:16 UTC (lasting 1 hour and 28 minutes)

What happened?

A background cleanup task added load to a shared database. The task included a guard meant to slow writes when database replicas lagged, but the guard ignored incoming load on the primary server, so the job kept running until the primary reached its connection limit. Dependent requests then could not complete reliably.

What worsened it?

Two additional problems amplified the impact and delayed recovery. The database calls lacked a short timeout, so request handlers waited on the stalled database instead of failing quickly. A retry loop around token creation kept resending writes that were already failing, holding the database saturated rather than allowing it to recover.

Response path

• The background data-cleanup task began writing into a shared database.
• Its guard measured replica lag but failed to account for incoming load on the primary server.
• Connection exhaustion on the primary caused dependent-service requests to fail or slow down.
• Automated monitoring surfaced the incident, and shared-database load was identified as the cause.
• Internal load was reduced and the cleanup task was paused.
• The database recovered and affected-service error rates returned to normal.

What safeguards reduce recurrence?

More safeguards are being added to improve database resilience and service availability. The planned controls will tie background processing to available database capacity, cut unnecessary retries during failures, and reduce the impact of slow database requests. Service startup behavior is also being improved so that brief database connectivity problems do not unnecessarily interfere with recovery.

September 28 21:16 UTC (lasting 52 minutes)

What happened?

Some Copilot code review results were not delivered to customers, leaving completed reviews missing from pull requests.

Why it failed?

Copilot code review uses a separate service to run reviews, and that service returns 2 versions of the results: a limited copy without the review comments and a full copy with the completed review. Copilot code review publishes reviews only from the full copy. A change to the service started sending the full copy only for some customers; for other customers, only the limited copy was sent.

Because of that mismatch, Copilot code review could not build a review but still reported successful completion. Missing review results went undetected due to insufficient monitoring. The problem was detected when some reviews remained incomplete for more than 30 minutes, triggering an alert to first responders.

The incident was mitigated by reverting the change.

Response path

• The rollout of the service change that runs Copilot code reviews began, and review output stopped appearing on pull requests in every environment.
• Review requests began timing out, alerting the Copilot code review team; an incident was declared.
• The team traced the failures to the rollout and started the revert.
• The incident was posted on the GitHub Status page while required checks ran.
• Once the revert reached all affected regions, the incident was resolved, and customers were told to request the affected reviews again.

What safeguards reduce recurrence?

Reliability work for Copilot code review is being strengthened by adding safeguards across the review and delivery process. These changes should help completed reviews be delivered successfully, let failed reviews recover automatically, and provide earlier notification when problems occur.

Monitoring and deployment processes are also being improved so that problems can be identified and handled more quickly.

What the incident set teaches cloud governance

The August 6 actions outage, the August 17 github.com outage, the September failover disruption, and the controlled capacity pause reinforce a simple rule: measured improvements matter only when they lower customer risk under real conditions. The guiding order remains availability, then capacity, then features.

CloudComputing FAQ

How should a cloud architect read the traffic shares?

The October 7 Azure figures describe different traffic classes: 65.4% of read traffic for the core GitHub application, 48.3% of Git read traffic, and 80.2% of stream-processing traffic. They are diagnostic signals, not a single blended migration percentage.

What remains unfinished after cleanup and headroom gains?

The September 11 pause, the September 18 restart, the September 30 miss, and the end of October capacity target show that regional capacity work is still incomplete, even though database cleanup and GitHub Actions headroom produced measurable relief.

What delivery risk did Copilot code review expose?

The September 28 incident showed a gap between reported success and actual customer delivery. Because the publishing path accepted only the full copy of 2 versions, missing delivery could go unnoticed until reviews remained incomplete for more than 30 minutes.

Reliability review checklist

• Separate service progress from data-layer progress: 120 of 160 production services versus 13 of 51 primary MySQL databases.
• Read Azure traffic by workload: 65.4% of read traffic, 48.3% of Git read traffic, and 80.2% of stream-processing traffic.
• Track capacity dates: September 11 pause, September 18 restart, September 30 miss, and end of October target.
• Check cleanup proof: 24 authentication-core tables, approximately one million queries per second, approximately 980 million unnecessary edit-history rows, estimated 1.8 terabytes of recoverable storage, and the June deadlock drop.
• Review resilience controls: Half of sampled pull request traffic, 14 to 19 instances on September 23, and the GitHub Actions relief table.
• Inspect incident dates and durations: September 13 09:16 UTC, 1 hour and 28 minutes; September 28 21:16 UTC, 52 minutes.

Common misconceptions to avoid

• Do not add the traffic shares; they cover different workloads.
• Do not equate cleanup with capacity solved; incidents, a pause, and the September 30 miss still show unfinished work.
• Do not treat a successful report from Copilot code review as proof the customer saw the review; the September 28 failure could hide missing results until more than 30 minutes passed.

Action advice for cloud selection, operations, and risk owners

1. Score migration by availability risk, capacity state, traffic type, database move status, and customer exposure rather than task completion.
2. Make background jobs capacity-aware: bind them to available database headroom, add short timeouts, suppress futile retries, and protect service startup from brief connectivity blips.
3. Keep overload playbooks ready so lower-priority traffic can be shed before API and web workflows saturate, and train operators before incidents.
4. Verify both processing success and delivery success when a system returns 2 versions of a result.
5. Consult the engineering section on the GitHub Blog for the cited source detail.

Final takeaway for ItInsightsWorld CloudComputing readers

The value of this update is not the list of completed tasks, but the way it separates capacity relief from residual risk: August and September produced measurable progress in Central US, shared-database cleanup, pull request containment, and GitHub Actions headroom, while September incidents and an unfinished capacity target keep availability, capacity, then features as the correct order of work.