New CDN via fastly.com

[Baptiste Jonglez]

On 22-01-21, Paul Spooren wrote:
> > Like jow, I don't see the point of putting an OpenWrt-managed CDN in
> > front
> > of all project-specific mirrors.
> 
> Where did he say that, I only have a incomplete record of the adm mails
> sorry?

That's in the commit message you linked above: https://git.openwrt.org/bf96eb55c8

> > 
> > Think about it by assuming you pay for the CDN usage.  Why would you pay
> > to serve data to all buildroot users when there are already many
> > different
> > mirrors to handle this traffic?  Also, it centralizes source downloads
> > while there is no need to do it.
> > 
> > It's fine to add it as a fallback like we do currently with
> > sources.openwrt.org.  In both cases, the sources.openwrt.org origin
> > server
> > is still a single point of failure.
> 
> I described my problem a while ago when we used KeyCDN:
> 
> ----%<----
> 
> I'm currently downloading `curl` from a mirror[0] with 3kB/s. This is
> likely not the fault of uxnr.de but of a bad fiber connection from
> Hawaii to Germany. This throughput reminded me of this email and I'd
> like to ask again if there are any reasons against offering
> sourcescdn.openwrt.org as a CDN for sources.openwrt.org.
> 
> For testing I set it up at https://sourcescdn-1212d.kxcdn.com and it
> works as expected. With a modified download.pl I now receive sources
> with 200Mbit/s.
> 
> ---->%----
> 
> This doesn't work as a fallback as extremely low downloads speeds are not
> considered as "failure".

Then you can just change the mirror order in your own tree.  Or the
download function needs to be more intelligent.  Or talk with the mirror
operators.

As I said, I don't think it's acceptable that the OpenWrt project handles
the main download load for non-openwrt sources.  Sponsorship is always
delicate: at some point, CDN sponsorship might stop, and we shouldn't pay
for other people's traffic.  The previous CDN experience highlights just
that: if I understand correctly, they terminated the sponsorship because
the source CDN was generating too much traffic for them.

> > >  Due to the recent buildbot issues with updating feeds I'm wondering
> > > if we
> > >  should add all standard feeds as folders to sources.o.o and change
> > > the
> > >  default feeds file to download from the CDN rather than git.o.o.
> > > This should
> > >  give a massive release to the git servers and future failures.
> > > 
> > >  Additionally we could use a non-public origin server which feeds
> > > the CDN and
> > >  mirrors and distribute downloads.o.o directly via CDN.
> > 
> > For downloads.openwrt.org, the CDN service should be equivalent to the
> > current service.  Downloads from device should not be automatically
> > redirected to HTTPS (for opkg packages, and also for sysupgrade
> > downloads
> > from the device itself).  I think this is currently handled through
> > user-agent matching.
> 
> The sources.c.o.o is set to serve whatever is asked for, no magic headers
> needed. I'd implement the same mechanism for downloads.c.o.o, no redirects.

I don't see the point about the sources mirror, I'm talking about the
download server.  Adding a CDN should provide equivalent service to what
we have today, otherwise this is a regression. 

> > 
> > And it should definitely have IPv6.
> 
> Should be solved.

Thanks!

Baptiste
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.openwrt.org/pipermail/openwrt-adm/attachments/20210123/a386b2f2/attachment.sig>