Artifacts lost when lost communication with the server

[david-thrower]

Why are you starting this discussion?

Bug

What GitHub Actions topic or product is this about?

Metrics & Insights

Discussion Details

Artifacts are lost when the error "The hosted runner lost communication with the server" occurs with a job.

• The run: https://github.com/david-thrower/cerebros-core-algorithm-alpha/actions/runs/17053636900/job/48346863287 failed out probably becasue of RAM being exhausted.
• The UI tab to see the logs for the job that failed out now only shows the head of the logs, where it was showing a substantially longer section of the logs before the job failed out.
• The log archive downloaded is missing the logs from the last task altogether as well.
• It may be a great architectural improvement if the logs were streamed to a BLOB storage outside of the runner, so that they are preserved, even if the runner container fails. We may permanently have lost some important data here (the best trial hyperparameters and metrics, there were logs from 10 trials showing on the screen before the failure occured. Now only 5 trails are appearing).
• Clicking ["Gear button"] > ["view raw logs"] returns the error: "Failed to generate URL to download logs."

Additionally: Inconsistancies in the run status as presented by the UI

• The panel to the left shows "The task failed" (red "x" )
• The tab to view the logs from the run shows that the run is still running.
  • It is almost 24 hours later on a run set with timeout-minutes: 420.
  • refreshing the browser tab doesn't change that.

[Gihchathur]

Hi @david-thrower,

That’s a frustrating issue, losing artifacts and logs because the runner lost communication is a serious problem. Here are a few ideas + things to try:

What’s happening (based on your description)

1. The job failed (likely due to resource exhaustion), and when it dropped connection, the logs and artifacts from the last tasks were lost.
   GitHub
2. The UI now only shows partial logs, and downloading the full logs fails (“Failed to generate URL to download logs”).
   GitHub
3. The run status in the UI is inconsistent (shows “running” though it already failed) and does not refresh.
   GitHub

[AhmedSAAhmed]

You hit the nail on the head with your diagnosis this is almost certainly caused by the runner experiencing an Out of Memory (OOM) error.

When you are running heavy workloads (like the hyperparameter optimization in your screenshot) and RAM is exhausted, the host OS's OOM killer steps in and forcefully terminates processes to keep the machine alive. Unfortunately, it often kills the GitHub Actions runner agent process itself.

Here is a breakdown of why you are seeing those specific symptoms:

1. Why the logs are truncated and fail to download:
   GitHub Actions does actually stream logs to the backend in real-time (which is how you see them live in the UI). However, it streams them in chunks. When the runner agent is abruptly killed by the OS, it doesn't get a chance to gracefully flush its final buffers or send the "job complete/failed" signal to the backend. This leaves the backend log file corrupted or unfinalized, resulting in the "Failed to generate URL" error when you try to download the raw logs.

2. Why the artifacts are lost:
   Artifacts aren't streamed; they are explicitly uploaded during a specific step (e.g., actions/upload-artifact). If the runner dies during the compute step, it never reaches the upload step, and the data dies with the ephemeral VM.

3. The UI Inconsistency (Red X vs. Spinning Wheel):
   This is a known desync issue. The GitHub backend realizes the runner is dead because it stops receiving the periodic "heartbeat" ping, so it marks the overall job as Failed (the red "X"). However, the log viewer in the UI is waiting for a specific WebSocket event saying "Step X finished," which the dead runner can never send. So, the UI spins forever.

Recommendations for your ML workload:
Relying on standard GitHub Actions logs and end-of-job artifacts for distributed random search or ML training is risky because of this exact scenario.

Short term: You likely need to switch to GitHub Larger Hosted Runners with more RAM to prevent the OOM crash.

Architectural improvement: Instead of relying on runner stdout logs, you might want to integrate an external experiment tracker (like Weights & Biases, MLflow, or Comet). These libraries stream your trial metrics, hyperparameters, and checkpoints directly to an external server during the run. If the GitHub runner gets OOM killed on trial 8, the data for trials 1-7 is already safe on your tracking server.