OSUOSL datacenter migration and hosting costs for buildbot workers

[Robert Marko]

On Sat, 15 Nov 2025 at 18:44, Baptiste Jonglez
<baptiste at bitsofnetworks.org> wrote:
>
> Hi Pono, all,
>
> As part of the OSUOSL datacenter migration, we have received an offer from
> them to upgrade our build workers with much newer hardware (servers with
> 2x Xeon Platinium 8280 CPU, perfect for running a large number of builds
> in parallel).  We could benefit from having 2 such servers.
>
> But before accepting, I feel we should clarify the funding aspect.
>
> @Pono: did you have a chance to discuss hosting costs with SFC and OSUOSL
> already?  What kind of budget could we allocate to infrastructure costs?
>
> @all: any opinion on paying hosting costs for build workers?  In my
> opinion, the limiting resource is sysadmin time (mostly ynezz and me in
> our spare time).  OSUOSL is really great in that regards, because their
> hosting is very stable and long-term.

Hi Baptiste,
I am very much in favor of upgrading the HW and paying for the buildbots from
our donations.

Regards,
Robert
>
>
> As a reminder, we have an overview of infrastructure costs here:
>
>   https://openwrt.org/infrastructure#hosting_infrastructure_and_costs
>
> Please update it in case it's not up-to-date.
>
> Of course we need to keep running build workers at multiple providers
> for redundancy (many thanks to the people/org sponsoring build workers!)
>
> Baptiste
>
> On 07-07-25, Daniel Pono Takamori wrote:
> > Heya, Pono from Software Freedom Conservancy (OpenWrt's fiscal home)
> > here.
> >
> > Just wanted to quickly chime in a mention that we at SFC are working
> > with the OSL, together with our member projects, to figure out the best
> > way forward for hosted infrastructure. Sounds like it's been working out
> > with hosting buildbot workers over there, so if that seems to be the
> > preferred option, we can work on getting a contract with OSL (and find
> > out hosting costs). On that note I wanted to mention that there are
> > funds available at SFC for things like this, so we can avoid having
> > members pay for CI!
> >
> > I'll update again after we get a chance to talk internally and with
> > Lance at OSUOSL about the best way forward for SFC and its member
> > projects to continue to host over there.
> >
> > Thanks!
> > -Pono
> > Community Manager
> > Software Freedom Conservancy
> >
> > On Mon, Jul 07, 2025 at 11:21:53PM +0200, Baptiste Jonglez wrote:
> > > Hi,
> > >
> > > OSUOSL is migrating to a new datacenter that will increase their hosting
> > > costs, and they want to discuss how we could possibly contribute to costs.
> > > Details quoted below.
> > >
> > > They currently provide us 6 build workers (2 physical hosts and 4 VMs):
> > >
> > >   https://openwrt.org/infrastructure#buildbot_workers
> > >
> > > This infra has been working quite well since 2018, with different
> > > iterations.  We haven't paid anything so far.
> > >
> > > I see several possible directions:
> > >
> > > 1) establish a contract and start paying OSUOSL for the service using OpenWrt funds
> > >
> > > 2) stop using these buildbot workers
> > >
> > > 3) keep using these buildbot workers for free (same as today)
> > >
> > > Personally I would prefer to continue working with OSUOSL, and I think we
> > > have the funds to contribute financially.  They have been very
> > > professional and responsive, the level of service is the same or even
> > > better than companies like Hetzner.  We don't need to spend too much time
> > > since they take care of everything (e.g. initial OS installation, hardware
> > > repairs)
> > >
> > > We could even maybe increase the number of workers: it seems more sustainable
> > > than having project members paying buildbot workers from their own pockets.
> > >
> > > What do you think?
> > >
> > >
> > > OSUOSL request below:
> > >
> > > > Due to significant infrastructure issues at the Kerr Datacenter, including a
> > > > critical failure in one of the two main UPS systems and the high cost of
> > > > necessary repairs, Oregon State University has made the decision to vacate this
> > > > facility rather than invest further in it. As a result, OSL is required to
> > > > relocate all of its equipment within the next few months (potentially sooner).
> > > >
> > > > Relocating to a new datacenter will increase OSL's operational costs for power,
> > > > space, and connectivity compared to Kerr. Historically, OSL covered most costs,
> > > > but higher expenses in new facilities mean sustainable hosting for dedicated
> > > > physical hardware requires projects to contribute more directly to covering
> > > > these operational expenses.
> > > >
> > > > For projects with existing contracts:
> > > >
> > > > We don't anticipate an immediate increase in your contractual costs. However,
> > > > higher OSL costs overall may necessitate future discussions about pricing upon
> > > > contract review/renewal.
> > > >
> > > > For projects without contracts:
> > > >
> > > > Facing significantly higher operational expenses in a new facility compared to
> > > > Kerr, sustainable hosting for dedicated physical hardware will require projects
> > > > to contribute more directly to covering these operational expenses. We
> > > > understand that fully covering the total costs might be challenging for some
> > > > projects, and we are flexible in working with you to find a solution. Our aim is
> > > > to ensure the costs directly associated with hosting your physical equipment are
> > > > addressed to make continued co-location sustainable.
> > > >
> > > > We want to partner with you to find a workable solution for contributing to
> > > > these costs. We offer two primary paths for contribution: establishing a formal
> > > > hosting contract (our preferred method) or providing an annual donation based on
> > > > an estimate of your hosting costs. We are committed to working with you to find
> > > > a mutually agreeable solution and will not remove any co-located hardware until
> > > > a solution is in place. Please let us know in your reply if you would like to
> > > > discuss these contribution options in more detail.
> > >
> >
> >
> >
> > > _______________________________________________
> > > openwrt-adm mailing list
> > > openwrt-adm at lists.openwrt.org
> > > https://lists.openwrt.org/mailman/listinfo/openwrt-adm
> _______________________________________________
> openwrt-adm mailing list
> openwrt-adm at lists.openwrt.org
> https://lists.openwrt.org/mailman/listinfo/openwrt-adm