Split repo by platform

[wipfli]

I am wondering if we should split MapLibre GL Native up into multiple repos for the different platforms such as iOS, Android, Qt, Node, and maybe more.

We discussed this topic a bit in the last two technical steering committee meetings and I would be curious to hear what people think...

[wipfli]

From the meeting notes:

@nyurik advocated that a monorepo is simpler to maintain than multiple split repos because CI will tell directly side-effects of your changes to the other platforms. @atierian found that the same can be achieved if CI in one repo triggers CI in another repo. @HarelM said that having to take care of side-effects in a different platform which is maybe broken for unrelated reasons discourages people from contributing.

[nyurik]

One other option is to let whoever will spend full time developing this project to decide this. I feel we would do more damage than good if we start refactoring it without being too deeply involved in the project.

[brendan-ward]

As a (very minor) contributor, I'll share the intimidation that comes with trying to engage in a single project that supports so many different platforms, nearly all of which require expertise I don't have to even engage with them let alone troubleshoot if failures showing up on CI are random, specific to a particular platform, or something that I broke in the core used by multiple platforms.

The singular repo also means that there are a bunch of platform-specific submodules that are orthogonal to working on the core, which overall makes the project heavier to check out, update, and work on if all one wants to do is work on the core. For downstream projects that use the core but none of the platforms, that then requires a bit of specific workarounds in CMake and submodules to avoid pulling in things that are not used.

I also agree that it is a good idea to catch issues that break downstream platform-specific builds as early as possible - but that maybe that doesn't all need to be in the same repo to achieve. I'm not at all familiar with the category of changes to the core that would break specific platforms, but would some of the concern be mitigated if there was a stronger test suite around the core? Something that would enable catching unexpected breaking changes via unit tests at the core level, rather than via integration testing into the platform-specific code (or their respective unit tests, at least).

Some of it comes down to who needs to fix something; for example, if I break something in an Android build due to a change in the core, that I can't trace back to my change in the core, now I'm stuck until someone familar with the Android build can help troubleshoot that (i.e., burden is on the producer). Instead, if the platform-specific stuff was split out into separate repos, that troubleshooting instead happens when trying to update to use a newer commit from the core (i.e., burden is on the consumer). But also, the upgrade to use a newer core commit is deliberate (requiring maintenance labor) rather than automatic, but also preserves a previous point in time build that works.

Splitting the platforms from the core would require likely more governance and documentation, so that any major changes to the core that would break downstream platforms (e.g., API changes) is done carefully in a controlled fashion and communicated clearly, so that it is easier to update the platforms-specific repos.

Some of this comes down to deciding what things:

• are so tightly coupled they need to be updated simultaneously
• are tightly coupled but can be locked to specific versions with deliberate upgrade process
• are likely to share a similar community of contributors and requisite expertise vs those that are very different groups of devs

[HarelM]

I generally agree with what was said here. Having said that, splitting the repo isn't the only solution to being able to update the core separately from the binding update, but it's probably the most straight forward one.
I also agree with @nyurik that this is mainly a question for the maintainers.
It's also worth saying that we can always change the decision we make according to what's the most effective and productive from the maintainers point of view.

[wipfli]

Here is what @petr-pokorny-1 wrote about the repo structure: https://github.com/maplibre/maplibre-gl-native/blob/main/ARCHITECTURE.md#repository-structure

[wipfli]

Mapbox announced the repo split in 2019 and justified it with the following statement: mapbox/mapbox-gl-native#15971

[nyurik]

@wipfli so 3 repos - also makes sense, and since MB did this with all the repo structure knowledge, it probably makes sense for us as well, but unless you want to actively work on this repo, I'll deffer to the "official native maintainer" to make this call :) Otherwise if someone who already worked with this repo, like @petr-pokorny-1 , comes back, they will have to re-adjust it again

[wipfli]

The link is just for reference. I think keeping MapLibre GL Native as one repo works as well...

[wipfli]

@nyurik what do you mean actually by official maintainer? Do you mean the people with admin rights? https://docs.github.com/en/organizations/managing-access-to-your-organizations-repositories/repository-roles-for-an-organization

[nyurik]

no, i mean whoever steps forward to lead this project (and who will, by extension, get admin rights soon after)

[ntadej]

I also support moving specific SDKs to their own repositories and keep the "core" separate. This may also allow stable SDK using unstable core combinations and vice versa.

[birkskyum]

For anyone interested in a split - make a design proposal as described in CONTRIBUTING.md.

[nyurik]

Hi @birkskyum , I thought there was some pushback on this from @roblabs (I might be mistaken). I am ok either way

[wipfli]

I suggest we draft a roadmap for how to move the native repo forward for until the next Technical Steering Committee meeting and discuss stuff there...

[ntadej]

I see no reason we rush and currently Android CI will not run if only Qt or iOS change. I also agree that we should first have tests under control and then start splitting.

[roblabs]

I agree with having a roadmap so that we may "plan our work, and work our plan".

@wipfli "I suggest we draft a roadmap for how to move the native repo forward for until the next Technical Steering Committee meeting and discuss stuff there..."

[louwers]

Swift's package managers prefer a source-only distribution. Right now we have to provide a binary distribution (or do we?) because of the monorepo. Some thoughts from @1ec5 w.r.t. to this: https://osmus.slack.com/archives/C04K0NBUEEM/p1674662156251219

[roblabs]

If we go away from the current MapLibre for Swift binary distribution, then we are expecting consumers of the package to have to build the library every time.

Developers already have the choice of building from source, why replicate that in a Swift Package when the purpose is to distribute releases?

I would recommend a binary distribution for iOS & macOS even if we spilt repos. That allows a developer the promise of a Swift Package: pull it in and focus on your own project, not building from open source.

---

@louwers — "Swift's package managers prefer a source-only distribution. Right now we have to provide a binary distribution (or do we?) because of the monorepo"

[ntadej]

Do you have a reference on Swift and source-only?

[roblabs]

@louwers can answer for the part about "source only", as I wasn't aware of the preference until he mentioned it.

Regarding your question, here are references from developer.apple.com that discuss Swift Packages

• Swift packages
• Distributing binary frameworks as Swift packages

---

🕐 Edit: I updated the README for the binary distribution repo to update the latest docs around Swift Packages.

https://github.com/maplibre/maplibre-gl-native-distribution#swift-packages-development

[louwers]

Do you have a reference on Swift and source-only?

I was just paraphrasing the thoughs shared by @1ec5 on Slack. 🙂

A lot has been written about the monorepo approach in the context of gl-native, but one key consideration is that two of the main package managers on Apple platforms are geared towards source-only distribution. MapLibre currently relies on a separate repo to distribute artifacts to these package managers, but it’s suboptimal, especially if the ultimate goal is to move towards Swift.

@roblabs Thanks for the links!

[louwers]

@ntadej Someone from Apple arguing why source distribution is better (no requirement to maintain binary compatibility): https://developer.apple.com/videos/play/wwdc2019/416 (long video).