Keeping a Self-Hosted Git Server in Sync With GitHub

If you're running a self-hosted Git server (OneDev, in my case) alongside GitHub as the real source of truth — for CI you can't run on GitHub Actions, for example — its copy of a repo is not a live mirror by default. You have to build that yourself. Here's the working pattern, plus every gotcha that cost real time along the way.

Why polling instead of a webhook

The obvious answer is "just use a GitHub webhook to push updates in." That needs your self-hosted box reachable from the public internet. Worth actually testing this rather than assuming — I checked from a few independent external network locations and got connection-refused on every port, even though the box's own firewall was correctly configured and outbound traffic worked fine. That's a routing/security-group problem one layer above the box itself, not something fixable by SSHing in and poking at iptables. If you hit the same wall, polling is the fallback, and it works fine.

The working job

Add to your buildspec (commit this through normal git flow, not through the UI's buildspec editor — more on why below):

- name: sync-from-github
  steps:
  - type: PullRepository
    name: Pull from remote
    remoteUrl: https://github.com/<org>/<repo>.git
    passwordSecret: github-pull-token
    refs: refs/heads/* refs/tags/*
    withLfs: false
    force: true
    condition: SUCCESSFUL
    optional: false
  triggers:
  - type: ScheduleTrigger
    cronExpression: 0 */5 * * * ?
  retryCondition: never
  maxRetries: 3
  retryDelay: 30
  timeout: 14400

No checkout step — this job only syncs the repo itself, it doesn't build anything.

Field-syntax gotchas that each burned a real cycle

1. `refs` takes bare patterns, space-separated — not full refspecs. The UI auto-templates whatever you type as {input}:{input}. Type the full refs/heads/*:refs/heads/* yourself and you get a quadrupled, invalid refspec — fatal: invalid refspec with a visibly doubled string is the tell. Just write refs/heads/* refs/tags/*.
2. Force is its own toggle, not something to embed as a `+` in the refs field. Given gotcha #1's auto-templating, typing +refs/heads/* produces +refs/heads/*:+refs/heads/* — invalid, + on both sides. There's a dedicated Force checkbox; the underlying YAML field is force: true. You need this if your self-hosted copy's default branch can ever diverge from GitHub's (e.g. someone edited a buildspec directly through the UI) — a non-force fetch then rejects with non-fast-forward. If GitHub is your real source of truth, force-overwriting the local copy is intentional, not risky.
3. Cron triggers use Quartz syntax — 6 fields, seconds first — not standard 5-field Unix cron. */5 * * * * fails with Invalid cron expression: Unexpected end of expression. "Every 5 minutes" is 0 */5 * * * ? (seconds, minutes, hours, day-of-month, month, day-of-week; Quartz requires day-of-week be ? when day-of-month is unconstrained).
4. Scheduled runs need an explicit access token; manual test-runs don't. I hit "This build is not authorized to sync to project: X" only on the cron-triggered run, never on a manual test-run of the identical job. Best theory: a manually-triggered build inherits the clicking user's real permissions, while a cron-triggered build has no human present, so the platform demands an explicit token credential instead. The fix: add a job secret containing an access token with management permission, at the parent/organization project level rather than the individual repo — it gets picked up automatically through project-hierarchy inheritance, no explicit reference needed anywhere in the buildspec.
5. Any commands block needs an even total count of `@` characters, full stop — this bites you even from things that have nothing to do with the platform's own @secret:x@ templating syntax, like curl's own -d @file argument. One unpaired @ fails the entire buildspec's parse, blocking every job in that project, including the sync job itself.

The bootstrapping chicken-and-egg problem

Adding this job through the UI, then running a force sync from that same job, immediately overwrites the commit that defined the job — the job deletes its own definition on first run if GitHub doesn't have that buildspec yet. The sequence that actually works:

1. Add the job through the UI first (fast iteration, validates field names as you go).
2. Once it's confirmed working, recover the exact YAML from the server's own git history for that project before it gets overwritten (git log --oneline on the underlying bare repo → find the last relevant commit; if it's already been overwritten, git fsck --unreachable --no-reflogs finds dangling commits that survive until garbage collection even without reflogs).
3. Commit that YAML properly to GitHub through a normal PR.
4. One more manual bootstrap to get the merged GitHub state back into your self-hosted copy (the job that would normally do this doesn't exist there yet — genuine chicken-and-egg). The cleanest method I found: a git bundle, not a token-bearing push —

git bundle create sync.bundle origin/master   # from an already-authenticated local clone
scp sync.bundle yourserver:/tmp/               # existing SSH key, no new credential needed
ssh yourserver "<fetch the bundle into the bare repo on the server>"

A bundle is a self-contained file, fetched as if it were a remote — zero GitHub or platform credentials need to touch the server at all for this one bootstrap step.

Applying this to another repo

1. Add the pull-token job secret (repo-scoped, or reuse a shared one at the parent/org level).
2. Add the sync job above, pointed at that repo.
3. Bootstrap once, as above.
4. Your existing branch/PR-update triggers on that repo's real build jobs then fire automatically as the sync job keeps the mirror current.

Once inbound network access opens up

If you later get a webhook path unblocked, most self-hosted CI platforms expose a trigger-by-URL endpoint:

POST https://<reachable-host>/~api/trigger-job?project=<org>/<repo>&job=sync-from-github&branch=master&access-token=<token>

Worth actually testing the HTTP method rather than trusting the docs — I found it silently needed to be POST despite reading like a simple webhook payload URL; GET just failed. This is also useful today, without any webhook at all: calling it manually forces an immediate sync instead of waiting for the next cron tick — handy right after merging something you want reflected on the self-hosted side immediately.

Bonus: reporting build results back to the PR

Once the sync loop is solid, a real build job can get a second step that reports its own pass/fail back to the GitHub PR as a comment. A few things worth knowing before building this:

• Don't rely on a "current PR number" variable that's only populated for PRs created within the self-hosted platform itself — since the sync job only pulls branches, not GitHub's PR refs, the platform never sees these as PRs, just branch builds. Query GitHub's API directly (GET /repos/{repo}/pulls?head={org}:{branch}&state=open) to find the PR number for the branch that just built. No open PR for that branch means skip silently, don't error.
• The report step needs to run unconditionally (even on failure), and the main build step needs restructuring so it doesn't hard-exit before capturing its own outcome — wrap the real commands in a subshell, redirect to a log file, capture the exit code to a status file, then re-exit with that code so the platform's own UI still shows correct pass/fail. The report step reads those two files afterward regardless of what happened.
• Build the JSON payload with a real JSON tool (jq -n --arg ...), never hand-escape a log tail into a string literal — build logs contain quotes, backslashes, and newlines that break naive string concatenation immediately.

Source

• OneDev documentation