Synchronizing Code Across Boundaries: Architecting Bidirectional Repository Pipelines with Google Copybara

The Monorepo vs. Polyrepo Dilemma in Modern Engineering

In modern software engineering, repository architecture often dictates organizational velocity. Large enterprises frequently gravitate toward monorepos for their unified dependency management, simplified cross-team collaboration, and atomic commits. However, this approach runs into a wall when interfacing with the open-source community or external partners.

You cannot easily expose a proprietary monorepo to the public, nor is it practical to manually copy-paste changes between a private repository and a public GitHub repository. This is where repository synchronization becomes critical.

Simply running a naive git merge or git cherry-pick across repository boundaries fails because it doesn't account for file path translations, sensitive internal comment stripping, proprietary dependency scrubbing, or licensing compliance. To solve this, Google developed and open-sourced Copybara, a specialized tool designed to transform and move code between different repository systems.

In this guide, we will deep-dive into the mechanics of Google Copybara, explore its Starlark-based configuration engine, and architect a robust, automated pipeline to handle bidirectional code synchronization between a private monorepos and a public open-source target.

---

Understanding the Core Architecture of Copybara

Copybara does not operate like a standard Git client. Instead of syncing branches directly, it treats code migration as a pipeline composed of four primary components:

1. Origin: The source of truth where changes originate (e.g., a specific folder inside a private monorepo).
2. Destination: The target repository where changes are pushed (e.g., a public GitHub repository).
3. Transformations: A sequence of deterministic operations applied to the codebase during transit. This includes moving directories, rewriting import paths, replacing strings, and scrubbing sensitive metadata.
4. Metadata: Mechanisms to rewrite commit messages, change authorship, and maintain a history of migrated revisions to prevent duplicate migrations.

Copybara configurations are written in Starlark, a dialect of Python 3 designed for deterministic execution. Because Starlark is hermetic, Copybara operations are highly reproducible across local developer machines and CI/CD pipelines.

---

Step-by-Step Tutorial: Setting Up a Push Pipeline

Let's architect a scenario where we have a private monorepo containing an internal library located at projects/core-library. We want to publish this library to a public GitHub repository named github.com/oss-community/core-library while stripping out internal package namespaces and proprietary configurations.

Step 1: Defining the Copybara Configuration File

Create a file named copy.bara.sky in the root of your migration workspace or within your monorepo. Here is the foundational structure for a push workflow:

source_repo = "[email protected]:internal-org/monorepo.git"
dest_repo = "[email protected]:oss-community/core-library.git"

core.workflow(
    name = "push_to_public",
    origin = git.origin(
        url = source_repo,
        ref = "main",
    ),
    destination = git.destination(
        url = dest_repo,
        push_to_refs_prefix = "refs/heads/main",
    ),
    origin_files = glob(["projects/core-library/**"]),
    destination_files = glob(["**"]),
    authoring = authoring.overwrite("OSS Sync Bot <[email protected]>"),
    transformations = [
        # Move the nested folder to the root of the destination repo
        core.move("projects/core-library", ""),
        
        # Scrub proprietary internal configurations
        core.remove(glob(["**/internal_config.json", "**/*.local.properties"])),
        
        # Replace internal namespace imports with public ones
        core.replace(
            before = "com.internal.monorepo.core",
            after = "com.github.oss.core",
            paths = glob(["**/*.java", "**/*.gradle"]),
        ),
        
        # Rewrite commit messages to keep them clean
        metadata.save_author(),
        metadata.squash_notes(
            prefix = "Imported changes from monorepo\n\n",
        ),
    ],
)

Step 2: Analyzing the Transformations

• origin_files = glob(["projects/core-library/**"]): This acts as a filter. Copybara will only look at changes inside this directory in the origin repository.
• core.move("projects/core-library", ""): Since the destination repository expects the code to be at its root rather than nested inside projects/core-library, this transformation flattens the directory structure.
• core.remove(...): This ensures that proprietary configuration files never leak into the public domain.
• core.replace(...): This is a powerful regex-based search-and-replace engine. It rewrites internal import namespaces to match the public repository's package structure.
• metadata.squash_notes(...): Instead of exposing every minor internal commit message, this squashes the incoming commits and formats them with a clean, standardized message.

---

Bridging the Gap: Implementing Bidirectional Sync (Pull Pipeline)

One of the biggest challenges in repository synchronization is handling external contributions. If an open-source contributor submits a Pull Request (PR) to your public repository, how do you pull those changes back into your private monorepo without breaking the history?

To achieve this, we define a second workflow in our copy.bara.sky file: a pull_from_public workflow.

core.workflow(
    name = "pull_from_public",
    origin = git.origin(
        url = dest_repo,
        ref = "main",
    ),
    destination = git.destination(
        url = source_repo,
        push_to_refs_prefix = "refs/heads/sync-from-oss",
    ),
    origin_files = glob(["**"]),
    destination_files = glob(["projects/core-library/**"]),
    authoring = authoring.pass_thru("OSS Sync Bot <[email protected]>"),
    transformations = [
        # Reverse the directory flattening
        core.move("", "projects/core-library"),
        
        # Reverse the namespace changes
        core.replace(
            before = "com.github.oss.core",
            after = "com.internal.monorepo.core",
            paths = glob(["projects/core-library/**/*.java", "projects/core-library/**/*.gradle"]),
        ),
        
        # Prefix the commit messages to identify external contributions
        metadata.expose_label("COPYBARA_INTEGRATE_REVIEW"),
    ],
)

Preventing Infinite Sync Loops

When syncing bidirectionally, a major risk is the "infinite loop" phenomenon: syncing a change from Private to Public, which Copybara then sees as a new change on Public, attempting to sync it back to Private, and so on.

Copybara solves this natively using metadata tracking. Every time Copybara pushes a change to a destination, it appends a hidden label to the commit metadata (or git tags) containing the last migrated SHA-1 hash of the origin. When the pull workflow runs, Copybara inspects these tags/labels, realizes the change originated from the monorepo, and safely ignores it.

---

Automating the Pipeline with CI/CD

Running Copybara manually on a developer machine is prone to human error and latency. To ensure real-time synchronization, you should integrate Copybara into your CI/CD pipelines (such as GitHub Actions or GitLab CI).

Below is a production-grade GitHub Actions workflow that executes the push_to_public workflow every time a commit is merged into the monorepo's main branch.

name: Mirror Codebase to Public OSS

on:
  push:
    branches:
      - main
    paths:
      - 'projects/core-library/**'

jobs:
  sync:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Monorepo
        uses: actions/checkout@v3
        with:
          fetch-depth: 0 # Copybara needs full history to track SHAs

      - name: Set up SSH Keys
        run: |
          mkdir -p ~/.ssh
          echo "${{ secrets.COPYBARA_SSH_PRIVATE_KEY }}" > ~/.ssh/id_rsa
          chmod 600 ~/.ssh/id_rsa
          ssh-keyscan github.com >> ~/.ssh/known_hosts

      - name: Run Copybara
        uses: docker://gcr.io/copybara/copybara:v0.1.0 # Or use a specific release tag
        env:
          COPYBARA_SUBCOMMAND: run
          COPYBARA_WORKFLOW: push_to_public
        with:
          args: --git-destination-path=/tmp/dest --folder-directory=. copy.bara.sky push_to_public

Security Best Practices for CI/CD Execution

1. Principle of Least Privilege: Create a dedicated machine user (or GitHub App) for the destination repository. Do not use personal access tokens (PATs) that have global write access to your entire organization.
2. Ephemeral Environments: Always run Copybara inside isolated containers. This prevents local file caching issues and ensures that environment variables containing sensitive decryption keys or API tokens do not leak.
3. Audit Trail: Configure your monorepo branch protection rules to require reviews on the branches targeted by the incoming pull_from_public workflow. Never auto-merge external pulls without automated integration testing and manual approval.

---

Conclusion

Architecting repository pipelines with Google Copybara allows engineering teams to maintain the security and speed of a private monorepo while actively participating in the open-source community. By treating code synchronization as a series of deterministic, programmatic transformations rather than manual file operations, you eliminate human error, protect intellectual property, and streamline your developer workflows. As your codebase grows, investing in a robust Starlark-based pipeline will ensure your code safely and seamlessly crosses organizational boundaries.