New CDN via fastly.com

[Paul Spooren]

On Fr, Jan 22, 2021 at 16:47, Baptiste Jonglez 
<baptiste at bitsofnetworks.org> wrote:
> Hi,
> 
> On 21-01-21, Paul Spooren wrote:
>>  it took a little time but now we're sponsored by fastly to speed up 
>> our
>>  downloads.
> 
> Cool, what is the deal?  Is there a contract duration or an upper 
> bound on
> the generated traffic?
> 
>>  In cooperation with Ted I'm setting up sources.cdn.openwrt.org and 
>> suggest
>>  to enable CDN as preferred mirror. Specifically  that means to 
>> revert the
>>  follow commit:
>> 
>>  bf96eb55c8 Revert "scripts/download: add sources CDN as first 
>> mirror."
> 
> Like jow, I don't see the point of putting an OpenWrt-managed CDN in 
> front
> of all project-specific mirrors.

Where did he say that, I only have a incomplete record of the adm mails 
sorry?

> 
> Think about it by assuming you pay for the CDN usage.  Why would you 
> pay
> to serve data to all buildroot users when there are already many 
> different
> mirrors to handle this traffic?  Also, it centralizes source downloads
> while there is no need to do it.
> 
> It's fine to add it as a fallback like we do currently with
> sources.openwrt.org.  In both cases, the sources.openwrt.org origin 
> server
> is still a single point of failure.

I described my problem a while ago when we used KeyCDN:

----%<----

I'm currently downloading `curl` from a mirror[0] with 3kB/s. This is
likely not the fault of uxnr.de but of a bad fiber connection from
Hawaii to Germany. This throughput reminded me of this email and I'd
like to ask again if there are any reasons against offering
sourcescdn.openwrt.org as a CDN for sources.openwrt.org.

For testing I set it up at https://sourcescdn-1212d.kxcdn.com and it
works as expected. With a modified download.pl I now receive sources
with 200Mbit/s.

---->%----

This doesn't work as a fallback as extremely low downloads speeds are 
not considered as "failure".

> 
>>  Due to the recent buildbot issues with updating feeds I'm wondering 
>> if we
>>  should add all standard feeds as folders to sources.o.o and change 
>> the
>>  default feeds file to download from the CDN rather than git.o.o. 
>> This should
>>  give a massive release to the git servers and future failures.
>> 
>>  Additionally we could use a non-public origin server which feeds 
>> the CDN and
>>  mirrors and distribute downloads.o.o directly via CDN.
> 
> For downloads.openwrt.org, the CDN service should be equivalent to the
> current service.  Downloads from device should not be automatically
> redirected to HTTPS (for opkg packages, and also for sysupgrade 
> downloads
> from the device itself).  I think this is currently handled through
> user-agent matching.

The sources.c.o.o is set to serve whatever is asked for, no magic 
headers needed. I'd implement the same mechanism for downloads.c.o.o, 
no redirects.

> 
> And it should definitely have IPv6.

Should be solved.

Paul