{"meta":{"title":"Troubleshooting Actions Runner Controller errors","intro":"Learn how to troubleshoot Actions Runner Controller errors.","product":"GitHub Actions","breadcrumbs":[{"href":"/en/actions","title":"GitHub Actions"},{"href":"/en/actions/tutorials","title":"Tutorials"},{"href":"/en/actions/tutorials/use-actions-runner-controller","title":"Actions Runner Controller"},{"href":"/en/actions/tutorials/use-actions-runner-controller/troubleshoot","title":"Troubleshoot"}],"documentType":"article"},"body":"# Troubleshooting Actions Runner Controller errors\n\nLearn how to troubleshoot Actions Runner Controller errors.\n\n## Logging\n\nThe Actions Runner Controller (ARC) resources, which include the controller, listener, and runners, write logs to standard output (`stdout`). We recommend you implement a logging solution to collect and store these logs. Having logs available can help you or GitHub support with troubleshooting and debugging. For more information, see [Logging Architecture](https://kubernetes.io/docs/concepts/cluster-administration/logging/) in the Kubernetes documentation.\n\n## Resources labels\n\nLabels are added to the resources created by Actions Runner Controller, which include the controller, listener, and runner pods. You can use these labels to filter resources and to help with troubleshooting.\n\n### Controller pod\n\nThe following labels are applied to the controller pod.\n\n```yaml\napp.kubernetes.io/component=controller-manager\napp.kubernetes.io/instance=<controller installation name>\napp.kubernetes.io/name=gha-runner-scale-set-controller\napp.kubernetes.io/part-of=gha-runner-scale-set-controller\napp.kubernetes.io/version=<chart version>\n```\n\n### Listener pod\n\nThe following labels are applied to listener pods.\n\n```yaml\nactions.github.com/enterprise= # Will be populated if githubConfigUrl is an enterprise URL\nactions.github.com/organization= # Will be populated if githubConfigUrl is an organization URL\nactions.github.com/repository= # Will be populated if githubConfigUrl is a repository URL\nactions.github.com/scale-set-name= # Runners scale set name\nactions.github.com/scale-set-namespace= # Runners namespace\napp.kubernetes.io/component=runner-scale-set-listener\napp.kubernetes.io/part-of=gha-runner-scale-set\napp.kubernetes.io/version= # Chart version\n```\n\n### Runner pod\n\nThe following labels are applied to runner pods.\n\n```yaml\nactions-ephemeral-runner= # True | False\nactions.github.com/organization= # Will be populated if githubConfigUrl is an organization URL\nactions.github.com/scale-set-name= # Runners scale set name\nactions.github.com/scale-set-namespace= # Runners namespace\napp.kubernetes.io/component=runner\napp.kubernetes.io/part-of=gha-runner-scale-set\napp.kubernetes.io/version= # Chart version\n```\n\n## Checking the logs of the controller and runner set listener\n\nTo check the logs of the controller pod, you can use the following command.\n\n```bash copy\nkubectl logs -n <CONTROLLER_NAMESPACE> -l app.kubernetes.io/name=gha-runner-scale-set-controller\n```\n\nTo check the logs of the runner set listener, you can use the following command.\n\n```bash copy\nkubectl logs -n <CONTROLLER_NAMESPACE> -l auto-scaling-runner-set-namespace=arc-systems -l auto-scaling-runner-set-name=arc-runner-set\n```\n\n## Using the charts from the `master` branch\n\nWe recommend you use the charts from the latest release instead of the `master` branch. The `master` branch is highly unstable, and we cannot guarantee that the charts in the `master` branch will work at any given time.\n\n## Troubleshooting the listener pod\n\nIf the controller pod is running, but the listener pod is not, inspect the logs of the controller first and see if there are any errors. If there are no errors and the runner set listener pod is still not running, ensure the controller pod has access to the Kubernetes API server in your cluster.\n\nIf you have a proxy configured or you're using a sidecar proxy that's automatically injected, such as [Istio](https://istio.io/), ensure it's configured to allow traffic from the controller container (manager) to the Kubernetes API server.\n\nIf you have installed the autoscaling runner set, but the listener pod is not created, verify that the `githubConfigSecret` you provided is correct and that the `githubConfigUrl` you provided is accurate. See [Authenticating ARC to the GitHub API](/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/authenticating-to-the-github-api) and [Deploying runner scale sets with Actions Runner Controller](/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/deploying-runner-scale-sets-with-actions-runner-controller) for more information.\n\n## Runner pods are recreated after a canceled workflow run\n\nOnce a workflow run is canceled, the following events happen.\n\n* The cancellation signal is sent to the runners directly.\n* The runner application terminates, which also terminates the runner pods.\n* On the next poll, the cancellation signal is received by the listener.\n\nThere might be a slight delay between when the runners receive the signal and when the listener receives the signal. When runner pods start terminating, the listener tries to bring up new runners to match the desired number of runners according to the state it's in. However, when the listener receives the cancellation signal, it will act to reduce the number of runners. Eventually the listener will scale back down to the desired number of runners. In the meantime, you may see extra runners.\n\n## Error: `Name must have up to n characters`\n\nARC uses the generated names of certain resources as labels for other resources. Because of this requirement, ARC limits resource names to 63 characters.\n\nBecause part of the resource name is defined by you, ARC imposes a limit on the number of characters you can use for the installation name and namespace.\n\n```bash\nError: INSTALLATION FAILED: execution error at (gha-runner-scale-set/templates/autoscalingrunnerset.yaml:5:5): Name must have up to 45 characters\n\nError: INSTALLATION FAILED: execution error at (gha-runner-scale-set/templates/autoscalingrunnerset.yaml:8:5): Namespace must have up to 63 characters\n```\n\n## Error: `Access to the path /home/runner/_work/_tool is denied`\n\nYou may see this error if you're using Kubernetes mode with persistent volumes. This error occurs if the runner container is running with a non-root user and is causing a permissions mismatch with the mounted volume.\n\nTo fix this, you can do one of the following things.\n\n* Use a volume type that supports `securityContext.fsGroup`. `hostPath` volumes do not support this property, whereas `local` volumes and other types of volumes do support it. Update the `fsGroup` of your runner pod to match the GID of the runner. You can do this by updating the `gha-runner-scale-set` helm chart values to include the following. Replace `VERSION` with the version of the `actions-runner` container image you want to use.\n\n  ```yaml copy\n  template:\n    spec:\n      securityContext:\n        fsGroup: 123\n      containers:\n        - name: runner\n          image: ghcr.io/actions/actions-runner:latest\n          command: [\"/home/runner/run.sh\"]\n  ```\n\n* If updating the `securityContext` of your runner pod is not a viable solution, you can work around the issue by using `initContainers` to change the mounted volume's ownership, as follows.\n\n  ```yaml copy\n  template:\n    spec:\n      initContainers:\n        - name: kube-init\n          image: ghcr.io/actions/actions-runner:latest\n          command: [\"sudo\", \"chown\", \"-R\", \"1001:123\", \"/home/runner/_work\"]\n      volumeMounts:\n        - name: work\n          mountPath: /home/runner/_work\n      containers:\n        - name: runner\n          image: ghcr.io/actions/actions-runner:latest\n          command: [\"/home/runner/run.sh\"]\n  ```\n\n## Error: `failed to get access token for GitHub App auth: 401 Unauthorized`\n\nA `401 Unauthorized` error when attempting to obtain an access token for a GitHub App could be a result of a Network Time Protocol (NTP) drift. Ensure that your Kubernetes system is accurately syncing with an NTP server and that there isn't a significant time drift. There is more leeway if your system time is behind GitHub's time, but if the environment is more than a few seconds ahead, 401 errors will occur when using GitHub App.\n\n## Runner group limits\n\nYou can have a maximum of 10,000 self-hosted runners in one runner group. If this limit is reached, adding a new runner will not be possible.\n\n## Runner updates\n\n> \\[!WARNING] Any updates released for the software, including major, minor, or patch releases, are considered as an available update. If you do not perform a software update within 30 days, the GitHub Actions service will not queue jobs to your runner. In addition, if a critical security update is required, the GitHub Actions service will not queue jobs to your runner until it has been updated.\n\nValidate that your runner software version and/or custom runner image(s) in use are running the latest version.\n\nFor more information, see [Self-hosted runners reference](/en/actions/reference/runners/self-hosted-runners).\n\n## Legal notice\n\nPortions have been adapted from <https://github.com/actions/actions-runner-controller/> under the Apache-2.0 license:\n\n```text\nCopyright 2019 Moto Ishizawa\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n```"}