Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

If you host large, publicly available data in a cloud blob service, but you don't have a budget for it, one option is to use the "Requester Pays" feature that Amazon and Google provide. This makes the data available to anyone to download, but they need to pay the download cost themselves.

This is at the tradeoff of making your data significantly more irritating to access, as it's no longer just plugging in a URL into a program, plus everyone who wants your dataset needs to set up a billing account with Amazon or Google.



Or just post a magnet link.


Sure, that's a great option for helping to reduce the cost for well-meaning general use, but the other way makes your costs 100% predictable, which is great if you're on an academic budget (but, again, way more annoying for the downloaders unless they're also using AWS).


So if there's no other seeders, you end up eating the full cost as the only seeder....


torrents were supported by S3 in the past https://docs.aws.amazon.com/AmazonS3/latest/dev/S3Torrent.ht...


That probably wouldn't work here. That s3 bucket is hosting models downloaded at runtime/startup [1] and looks like under normal runs it would be cached. If this is being used in Apple's CI pipeline though the whole thing is being torn down between builds so every build and test has to fetch it again.

[1] https://github.com/huggingface/pytorch-transformers/search?q...


Or use Cloudflare in front of it, especially with mostly static data.


Cloudflare generally doesn’t want to cache data files on non enterprise plans (the terms generally disallow it). Only websites HTML and related content.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: