sourcehut Updates

Hello everyone! It’s time for another quarterly update, keeping you up to date on what we’re cooking up here at SourceHut. Drew’s update This past quarter I found myself mostly focused on “invisible” labor for SourceHut, which will make for a boring update from me this time. Most of my time was spent preparing a grant proposal, jointly with some other open source forges and related partners, to apply for funding from the EU. We’ll learn how that went sometime next quarter! Otherwise I’ve been focused on greasing the wheels and keeping the lights on – doing code reviews, fixing little bugs here and there, handling user support, mitigating rolling DDoS attacks (Conrad will elaborate on these in a moment), dealing with the finances (it’s tax season), and enjoying some rest after dealing with all of the above. In the coming quarter, I plan to write our annual financial report, and to invest more time in user-visible improvements. There’s a lot of work going into our GraphQL APIs now (led by Simon Martin!) which I want to build on. With this momentum I also plan to look into anonymous API access and more standardized and uniform GraphQL API designs, such as support for the connections specification for resource enumeration. We’ll leverage these API improvements to facilitate some long-awaited features, such as linking resource pages (e.g. git repos) back to the projects they belong to on the project hub. I also plan on doing some more work on the billing system, to finalize the migration to the EU, so if this works out all customers will be moved into the EU billing system soon enough. Conrad’s update While I did get good deal done this quarter, some of that work was certainly of the kind I wish I wouldn’t have to do in the first place. Let’s start with the elephant in the room: the DDoS. We still remain cautious about sharing too many details, but we wanted to at least offer a little glimpse into what we were facing. The below graph was provided by our network provider. For scale, note that the baseline traffic you can make out is not just ours - it’s us plus other customers. The visible spikes, however, were unfortunately directed at us alone… The graph is from some time ago. A few more waves came in after that. We are still on alert and of course discussing what if any mitigations we can put in place for such events in the future. There is a small silver lining to this. The DDoS came in several waves of different traffic patterns, but it was mostly aimed at network resource exhaustion. This “helped” us identify several places where internal network traffic (such as inter-service requests) was still routed over public (that is, saturated) interfaces. Those were all fixed and we were happy to see that afterwards those few requests that made it to our servers could successfully be handled. Hot on the heels of the DDoS we were targeted by another huge wave of spam sign-ups. These are accounts that get created solely for link farming. They basically get an advertisement with one or more links in their bio and never get used again. This time around, there seems to have been a serious campaign going on, creating over 300 accounts in a single month. We’ve seen such campaigns before, but we were mostly able to stall them by blocking the email domains they were using, which often seemed to be from obscure, hijacked relays or such. Unfortunately, by now, the main offender for fake accounts has become: Gmail… sad trombone So we had to resort to other means, and I added a keyword capability to our abuse detection system. All profile updates are now checked against certain keywords, and if there are a certain number of matching keywords the account is suspended right away. We will be very careful with the keywords we add to this to avoid false positives. The kind of crap we are dealing with is fortunately pretty easy to detect with 100% accuracy. Let’s talk about more interesting stuff. My favorite this quarter is of course that I managed just right on time to finish git.sr.ht deploy keys! In the “Access” tab of your repository settings, you can now add SSH keys which will be able to access only this very repository, either read-write or read-only. This is intended for keys used for example in CI or similar automation. This work was preceded by a clean-up of the meta.sr.ht SSH key handling, with the user-visible side effect that finally SHA256 fingerprints are used everywhere as opposed to the legacy MD5 fingerprints. Besides the few fixes here and there I also floated a first patch to replace the builds.sr.ht shell (currently Python) with a Go implementation. It might need a few fix-ups, but it already went through an RFC phase, so I think it’s fair to mention this now and call it a day. Everyone else SourceHut is 100% free and open source software, and the community is invited to participate in its development. Let’s take a moment to acknowledge the work of the volunteers who use and depend on SourceHut and sent along patches to improve it over the past few months. Simon Martin has been back at it again this quarter, writing many patches to improve the project hub. Thanks to Simon, the project hub now has a writable GraphQL API, allowing you to manage projects and project resources via the API. He has a few more patches queued up to improve the API further and reduce our Python footprint there. Simon also added some improvements for lists.sr.ht’s patch review view, associating new patchset revisions with their previous versions and adding a UI for navigating between different versions of a patch. Thanks for these and many other patches, Simon! Other community-led improvements include CismonX ’s improvemnts to PGP keys, allowing one to update an existing PGP key, for example, to bump its expiry date or update subkeys, and some other smaller improvements as well. Our volunteer build image maintainers have also been quietly keeping your build images up to date this month – CismonX was back to update FreeBSD to 14.4 and drop the EOL 13.x branch, and some small improvements for Debian were put forth by Michael Forney and Andrew Oberstar. Haowen Liu began the work to ship Ubuntu 25.10 and 26.04, which still has some growing pains for us to sort out. Big thanks to everyone who contributed to SourceHut this quarter!

Hello again! It’s a new year and time for a new quaterly update from SourceHut. You may have read our last update in Q4 2025 – let us get you up to speed on how those plans have played out, and what’s coming in the near future. Drew’s update First up, I have an update on the pricing changes we proposed in December : the new prices are now in effect for new customers. Existing users are grandfathered into their current price-point, but there is a button on your billing dashboard which will allow you to opt-in to the new price point if you wish to support us by doing so (and lots of love to everyone who asked for the ability to do that). If you cannot afford the new prices, there is a special price that matches the earlier minimum price, and of course as always if that’s still too much then we are happy to provide financial aid for anyone who needs it. Moving on, after last quarter’s focus on refactoring, I wanted to switch gears to adding some new user-facing features for a while. The most obvious of these changes is probably the new profile pages – here’s mine for reference . Each user’s profile now includes tabs linking to your resources (git repos, mailing lists, etc) across all services. We also added avatars and pronouns to your profile details ( add them here ) to show off here. We intend to be very sparing with avatars, by the way, to reduce distractions on the user interface – right now there are no plans to show them anywhere except for your profile page. Another UI change I made was, finally, answering the long-awaited requests for format=flowed support on the lists.sr.ht UI, with assistance on the style and ironing out some bugs from Ember Sawady and aoife cassidy. Emails written with format=flowed will now wrap properly and be much easier to read on mobile devices in particular. I’ve also seen through, with Simon Martin’s help, the rollout of “resource IDs”, or rids, to our GraphQL APIs. The implementation is not totally complete, and we could stand to add these to more resources, but they work and are providing some useful functionality for the most commonly used API resources. RIDs are inspired by ULIDs – they are unique identifiers for resources on SourceHut which have some useful properties: They are not predictable, preventing users from enumerating all resources by enumerating the namespace of predictable IDs They are lexically sortable by the moment of their creation They are stable even when a resource is renamed or transfers ownership This allows us to expand the GraphQL API with support for directly fetching arbitrary resources by ID, perhaps reaching relatively deep into the graph, such as fetching a specific comment on todo.sr.ht without going through the ticket and the tracker first. Throughout the UI we have also started adding a subtle copy of the resource ID for whatever you’re looking at, for example at the bottom of git repo summary pages . I intend to follow up on some earlier work in the coming quarter, for instance adding more resource IDs to resources that make sense throughout our GraphQL APIs. I’m also looking forward to working together with Simon on his coming work to establish a project hub API, which unblocks some important work like backreferences in the UI from resources to the projects that they belong to. I’d also like to start accepting payments in GBP in this quarter, and there are some tech debt goals that I’d be pleased to move the needle on. But, the big thing I’m working on this quarter is organizations. It will be a lot of work so I cannot promise that it will be done soon, but I’m pleased to share that – finally – we are making progress on support for them. Conrad’s update It’s one of those quarters where this write-up feels a bit therapeutic, because at first glance, I didn’t get any of the things done that I had planned. But sometimes you just have to take a step back and review, so let’s do that! As a follow-up to last quarter, sourcehut-migrate gained support for initializing databases, replacing the various -initdb Python scripts. That concludes our Alembic replacement with all functionality restored! Then there have been the usual, (mostly) invisible infrastructure updates. We finally have a structured - and more importantly, ready-to-go in case of emergencies - PXE-boot setup for our servers. So in case a server refuses to boot from disk, we can quickly bring it back online for diagnosis and recovery without having to hastily assemble DHCP configuration files. Kudos to Alpine Linux for being so stable that I have kept this such a low priority for far too long. I also at long last finalized and codified the monitoring of all our baseboard management controllers via IPMI. In software space, I have spent a significant amount of time refactoring various Go packages, with the final (and yet to realize) goal of rewriting the builds.sr.ht shell in Go, further reducing our Python usage. And for the last item, I need to do something which I usually try to avoid: making promises about future work. But in this case, it sort of provides the backdrop, so here goes: I have been on the somewhat tedious quest to clean up our SSH key database. It contains some surprising entries, mostly for historic reasons. This is of course not very exciting for users, but the reason I am doing this is because I am trying to refactor the SSH keys table. And this is still not very exciting, but when this table has reached a state with which we are content, I will use it as a blueprint for adding SSH deploy keys to git.sr.ht. There, I said it. Now the game is on for the next quarter… Everyone else SourceHut is 100% free and open source software, and the community is invited to participate in its development. Let’s take a moment to acknowledge the work of the volunteers who use and depend on SourceHut and sent along patches to improve it over the past few months. Once again, we have to thank Simon Martin for his many patches throughout the platform. This quarter, Simon added many useful features. Thanks to him, one now has finer-grained control over how builds are submitted, allowing one to whitelist or blacklist submissions from git.sr.ht to specific branches or refs, or disable it altogether; a similar feature is now available to tune build submissions for patches sent to mailing lists. He also added a feature to builds.sr.ht that allows users to edit job tags after submission, to organize your jobs post-facto. He’s also implemented multi-line selections for paste.sr.ht, added more detailed software version information to our APIs, and added links in the lists.sr.ht UI from superseded patchsets to the patchset that supersedes them. Lots of great stuff, thanks Simon! As usual we have a lot of updates from the community maintainers who keep your builds.sr.ht images in good working order. Gary Kim shipped Fedora 43 and oversaw the deprecation of Fedora 41, now end-of-life. CismonX released FreeBSD 15.x, and updated the image refresh schedule so that FreeBSD current is rebuilt daily. Marek Marecki kindly shipped OpenBSD 7.8 support, and Achill Gilgenast helped get Alpine 3.23 ready for use as well. Thank you, everyone! Last, but not least, we’d like to thank Varun Narravula for a small patch which addressed a bug that prevented todo.sr.ht tickets from being resolved via email. There are a handful of other small bug fixes and minor improvements from many other contributors as well – thank you to everyone who helped to make SourceHut even better in the past few months.

As we wrap up our end-of-year budget here at SourceHut, we find ourselves with a good opportunity to reconsider our pricing. We have always promised to evaluate any potential pricing changes in consultation with our community, which is why we’re bringing our proposal forward to you now for discussion. We’re looking forward to hearing your thoughts! tl;dr : We want to increase the base rates to €4/€8/€12, with current users grandfathered into their current rates. Furthermore, users in the EU who pay in US dollars will soon have to pay VAT when we rehome them with our EU business. Free service is still available to users who can’t pay. Read on for more details! New here? SourceHut is a software development platform which offers repository hosting, bug tracking, continuous integration, mailing lists, and more to projects hosted with us. SourceHut is 100% free and open source software. Welcome! SourceHut’s pricing has been the same since we first started accepting payments in 2018: $2, $5, and $10 per month, with a “2 months free” discount for users who pay annually. There is no difference between the price tiers; we use an honor-system model of “pay what you think is fair” based on your financial means and how much you use SourceHut. In the years since, inflation has effectively reduced the prices by 30%. All of you have been subject to the same inflation and stagnant wages in the meantime, and we have preferred to take the hit to our margins rather than pass the costs onto you in an increasingly stressed economy. But, at some point, we will have to consider raising our prices. Before we get into the proposed changes, I’d first like to reiterate that SourceHut is committed to never pricing anyone out of our services. If you cannot afford to pay for your account for any reason, then you are eligible for free service by applying for financial aid. Our proposal is to set the standard prices for new customers from January 2026 onwards in Euro at €4, €8, and €12, and in US dollars at $5, $10, and $15. 1 In addition to these standard rates, the billing UI will be updated to offer a reduced rate of €2/$2 (the same as the present-day minimum payment) in addition to financial aid as an option for users who cannot pay the standard rates. A mock-up of changes to payment process shows the offer of a reduced rate hidden behind a button click. Click to view the images at full size. Users with an existing paid subscription will be grandfathered into their current price indefinitely. If you cancel your paid subscription and sign up again later, you’ll pay the new price. Users who pay at a grandfathered price point will also get a notice in their billing profile asking them to volunteer to pay the new price: Moving US accounts to Europe One more change to prices coming up soon: as many of you know, we are in the process of moving our business from the US to Europe. Our US business was below the thresholds required to charge VAT, so users in Europe have not been charged VAT up until now. However, our European business is of course required to charge VAT to European customers. As we shutter the US business entity and move these customers to our European business, we will have to start charging VAT for customers in the EU. If your account is affected, you can expect to receive an email with the details at least 3 months prior to your account being migrated. You will have an opportunity to choose a lower price tier (or a higher one, for that matter), or cancel your subscription, before VAT is applied to your renewal payments. Your payment will still be denoted in US dollars, though you will have the option to switch to paying in Euro. Customers who are already paying in Euro are already being charged VAT, so if you’re one of them then there will be no changes for your account. There are still a number of technical hurdles to overcome planning this rollout, so the exact timeline is uncertain, but we’re hoping to complete this by Q2 – and finally close our US business shortly thereafter! 🇪🇺🎉 Why do we need to raise prices? As briefly explained at the top, we have not adjusted our prices for inflation since our inception, and that means that our margins have been steadily shrinking as the costs of doing business have gone up. SourceHut is still profitable and sustainable today, but for our long-term financial health it is necessary to increase our revenue. The price increases proposed exceed the price changes you would expect if we simply adjusted for inflation. The rationale is: These prices are future-proof, so we don’t have to change prices frequently. This allows us to offer established users the option to remain at their current price point, making up for it with new subscriptions and users who volunteer to pay the higher rates. The extra revenue gives us more resources to invest in SourceHut. With more revenue we are better able to invest in infrastructure upgrades, new services and features, and in our staff. Our staff makes below market salaries (standardized at €2800/month), and while we probably won’t be paying ourselves market rates any time soon, it would be nice to increase our compensation modestly, or hire additional staff, with the extra revenue. As always, all of our profits are always invested 100% back into free and open source software. Also: we’re planning on putting together a new financial transparency report soon, so you’ll have more insights into our finances and how the new prices are working out when the time comes. Share your thoughts? This proposal, all of its finer details and broad strokes, are up for discussion with the SourceHut community. That includes you! Please let us know your thoughts. We’re discussing it on IRC, on the #sr.ht channel on Libera Chat, and on the sr.ht-discuss mailing list in this thread . We’re looking forward to hearing your thoughts. Updated 2025-12-02: Clarify the situation around who will owe VAT. As we move our primary business into the EU, our base pricing is reckoned in Euro. Prices for other currencies like US dollars are selected by taking the current exchange rate from Euro and rounding the figures towards an attractive price gradient. Note that you can also pay in the currency of your choice and let your bank do the conversion from your native currency if your bank offers a good exchange rate.  ↩︎

Hi everyone! We hope you’re reading today’s update under as many blankets as possible, ideally in front of a warm fireplace, as the cold of winter starts to set in. You may have read our last update in the third quarter – it’s time again to take a look at what’s new and what’s next. Drew’s update I’ve been knee deep in refactoring over the past few months, and it has been very rewarding. The pile of tech debt has been steadily shrinking! Two of the goals I mentioned in my last status update were met: first, I rolled out ariadne-codegen to all of our frontends to handle their GraphQL requests to each other (and to their own backends) and I have found that it improves the frontend code substantially. And, the one we’ve been waiting years for: the legacy REST API has finally been removed (almost) entirely 🎉 There is still one internal user of the legacy API, which is a legacy webhook subscription used by all services to receive account updates from meta.sr.ht. Completely paying back all of the tech debt incurred by supporting the legacy API is blocked on the removal of this last piece of the puzzle. Conrad and I are debating a few ways to replace it – do we add a user details cache, do we have services fetch user data from meta every time they need it, or do we upgrade it to a GraphQL webhook, or do some other thing? Ideally we like to reduce the number of sources of truth so that we don’t have to worry about them disagreeing with one another – the solution we come up with in the end is something that will have to wait until the next quarter’s update to become clear. Speaking of the next quarter, I am planning on working on more user-facing improvements. In particular, I have plans to introduce first-class support for commit trailers into the git.sr.ht API as well as into the lists.sr.ht’s API for patchsets. Based on this work I will be updating the project hub to more tightly integrate the various components of SourceHut projects together, for instance by adding ticket references when a patch mentioning them is being discussed on the mailing list. The main motivator for working on this, however, is to add a commit trailer that lets you identify the dependencies of a patch which are not yet merged upstream, and have the CI pull those patches down when testing your work. The other user-facing changes I mentioned in the last update, such as an improved profile page on the project hub, or FIDO/U2F support for two-factor auth, are still planned for the near term and I hope to find time for them in the coming months. Multiple email addresses may end up being on hold until we figure out what to do with the internal legacy user profile update thing. Another large change I’ve made is to our process for handling tickets related to SourceHut itself. Each of the old bug trackers, one per service, has been closed to submissions and we’ve declared a sort of bankruptcy on all of the tickets that were there – we were not keeping up with it and many of them were duplciates, obsolete, or irrelevant. Instead we now have a single “sourcehut” bug tracker . I retrieved most of the tickets worth saving from the old bug trackers, and organized them onto this new one, which we are now attempting to better maintain so that you can get a view of our priorities. If we missed any tickets you cared about, please let us know on the sr.ht-discuss mailing list . One last treat for you before I let Conrad take the mic: just this week I put out an overhaul for our install docs , for those of you who want to run your own SourceHut instance. I won’t pretend that SourceHut is any easier to install than it has ever been, but these docs should give experienced sysadmins a better starting point for their deployments. Enjoy! Conrad’s update Welcome back to the department of under-the-hood improvements! I certainly have some treats for you this time around. As hinted at last time, we’ve made great progress with our Ceph cluster. The Alpine upgrade in the previous quarter unlocked an upgrade of Ceph itself, which I performed this quarter. We are now on Ceph 18 (“Reef”). There was another round of Alpine upgrades, too, so the path to Ceph 19 (“Squid”) is clear. Though I am not sure it will still happen this year. We are prudent with the Ceph version we are running and not in a rush… The absolute highlight for me this quarter, however, was finally dropping Alembic from our stack! It has been completely replaced by sourcehut-migrate , and we are quite happy with the result. The whole operation went smoothly, but there is one little regression: the xxx.sr.ht-initdb scripts got dropped without proper replacements. But fear not: I am almost done implementing an alternative solution in sourcehut-migrate ! As is customary, I will sprinkle in some user-facing changes: we enabled GraphQL introspection on all services. This facilitates the usage of various GraphQL tools to work with our API, without having to feed them with our schemas first. Another one that only few users will care about, but those probably all the stronger: lists.sr.ht finally supports emails in non-UTF8 encodings. I don’t think this was ever a conscious decision, it was just a few bits and pieces missing, but now it should be fixed. While we’d certainly still appreciate everyone switching to UTF8, we did not mean to lock anyone out. Incidentally, I think the most prominent cause of encoding issues was in fact folks importing old(ish) mailing list archives from elsewhere. So while this will hopefully improve the process, there is still plenty of other reasons why an import might fail. But I have good news about that, too: lists.sr.ht will now send you an email if an import fails, providing you with actionable details. Going back under the hood, I played a bit with golangci-lint . While we had tons of rather uninteresting warnings, it also did uncover a few very valid issues. So I bit the bullet and cleaned up the entire Go codebase to the point that we now run golangci-lint in CI, so that hopefully no new issues will creep in. We also moved one more of our ancillary services to Kubernetes: there is now mirror.srht.network , which will soon replace mirror.sr.ht . Right now we only use it for infrastructure-related packages, but with the next Alpine upgrade we will likely switch completely, allowing us to shut down yet another VM. Last but not least, if you’re still reading, your attention span is certainly long enough, so why not take a look at our new build worker RFC . I cannot take credit for all the ideas put forth in there, but as I sort of started driving this I’ll take the opportunity to solicit feedback from you, the community. Because the TL;DR is that we want to allow you, the community, to run your own build workers! Wouldn’t that be a nice user-facing feature for me to write about? Everyone else SourceHut is 100% free and open source software, and the community is invited to participate in its development. Let’s take a moment to acknowledge the work of the volunteers who use and depend on SourceHut and sent along patches to improve it over the past few months. Simon Martin has been hard at work on dozens of bug fixes and little improvements, and also found the time to develop some nice features, too: thanks to his efforts you’ll now find a little checkbox on your lists.sr.ht dashboard which will cause the mailing list software to copy you on your own emails, if you prefer for it to. We’re not quite done reviewing his work on this next one, but soon we’ll be rolling out another feature he developed which allows you to narrow the conditions under which build jobs are submitted, such as omitting feature branches from CI in your git repos. Simon also fixed a long-standing problem with the project hub recently, de-duplicating events so that an email sent to one mailing list shared among many projects does not fill their event logs up with dozens of events. Thanks, Simon! It’s been a relatively slow quarter for distro releases, so the workload of our volunteer build image maintainers has been pretty light. But I’d like to thank Maxell G for handling the roll-out of Fedora 44 as the new Rawhide release, and the EOL of Fedora 40, and to welcome Alex Kramer as the new maintainer of our Arch Linux images. There were dozens of other small patches in the past couple of months, fixing little bugs or making small improvements all over SourceHut, from many different contributors. Thank you to everyone who played a part in making SourceHut better! Thanks for reading our update – we welcome you to discuss it on sr.ht-discuss if you have comments or feedback. See you again in Q4!

Hello everyone! It’s time for another quarterly update on what we’re up to at SourceHut. There’s a lot of great stuff going on since you last heard from us ! Let’s get started. Drew’s update We finally rolled out the billing overhaul! God, that was so much work. Since the announcement in July we have successfully accepted a couple hundred payments in Euro – thank you to everyone for your support! The rollout was nearly flawless, with just a few things to sort out here and there and just few embarrassing problems to deal with (a fun one: a handful of users got invoices with a “SAMPLE” watermark). There are still some knock-on things to deal with for billing, especially when our Q3 tax bill is due, but most of it is not user-impacting. We’ll probably roll out a couple of new payment currencies and move all users to our European entity by the end of the year (which will, unfortunately, come with an annoying price increase for some users who will start being charged VAT, you’ll get an email before that happens if you’re affected). Other than that, the billing plans are now pleasantly unremarkable. Now that I’m free of the huge pile of billing work, I’ve been spending most of my time aggressively paying back tech debt, as promised in previous updates. The biggest change in this respect is that we’re getting serious about removing the legacy REST API in favor of GraphQL. All of the API documentation has been removed, the meta.sr.ht functionality for registering new legacy OAuth clients and personal access tokens has been removed, and I have done an analysis of contemporary legacy API usage and removed all of the API features which are not still in active use. There are about 30 users who are still relying on the legacy API – I have emailed them to inform them of the looming removal of the legacy API and offered to assist in migrating them to GraphQL. Please check your inbox if you’re one of them! Unfortunately one of those contemporary users of the legacy API is ourselves. I have been working to reduce internal consumers of the legacy API over the past few months, upgrading bits of the project hub, the build submitter for hg.sr.ht, rewriting how man.sr.ht communicates with git.sr.ht, and so on. It’s a work in progress but I’m making good strides towards refactoring all of the remaining code – at this point we’re down to a few legacy webhooks, and those are a bit more complicated to replace. I’ve also been looking into some more experimental changes. A big one that I’ve been researching is using ariadne-codegen to generate GraphQL clients for our Python frontends to use to talk to their own backends and the APIs of other services with which they integrate. A long term goal is to remove our frontend’s access to the database and get rid of SQLAlchemy, and having our frontends communicate with GraphQL for their work – providing a single source of truth for business logic and database access. Making this work well requires us to have a robust approach to GraphQL clients in the frontend code – and Ariadne seems to be a promising option for that. I’m getting ready to roll it out for meta.sr.ht’s frontend for a start later this week. I’ve been up to a few other things – I’ve replaced our dependency on minio’s Go SDK to talk to radosgw with the AWS SDK instead, for all of our services that speak to S3. That includes git.sr.ht and builds.sr.ht artifacts, as well as pages.sr.ht’s basic functionality. Since we moved from minio to radosgw, it had seemed a bit odd for us to continue to depend on the minio client SDK as well. I also replaced our dependency on the now-deprecated go-yaml/yaml with goccy/go-yaml, which had some knock-on effects that were resolved with ~meroje’s help. There’s a bunch of little things all over the place that I’ve improved in my quest to pay back our tech debt – more than I can elaborate on any further without boring you. I plan to continue my work in the coming months, but I will also be working on some new features for you. Some small, nice-to-have features that I have planned include support for multiple registered email addresses on your account, as well as webauthn (FIDO/U2F) support for 2FA. I also intend on overhauling the project hub’s profile page, to act as a sole unified page for your profile across all services – which should reduce some confusion and give you a nice page to show off your work. Conrad’s update For me as well, finally, some of the long-term work is coming to fruition! The first one is the site-wide rollout of sourcehut-ssh , our new unified SSH ingress. This is of course one of those under-the-hood changes, but it will make life a lot easier going forward for both admins and developers. It will also be the foundation for our planned SSH ingress handling in Kubernetes. Another one, hinted at in the last update, is sourcehut-migrate , our anticipated Go replacement for Alembic. It has already been rolled out for pages.sr.ht (which was not actually using Alembic before), but with paste.sr.ht the first switch is in the pipeline as well. A more user-facing highlight are some of the improvements to the GraphQL playground. My part in that was actually quite a while ago, adding support for variables and making some minor UI improvements. Conveniently, that prompted Drew to add a field providing the currently composed query as a curl request, providing a simple entrypoint for scripting. He also replaced the unannotated schema definition on that page with an integration of docs.sourcehut.org . We hope that all this will help people adopt and experiment with our GraphQL API. Our monitoring infrastructure also got some upgrades: metrics.srht.network is now on Prometheus 3 and we added a ntfy.sh sink to alertmanager. The latter is not public (for now) and the alerts will require some tuning to the new use cases that this opens up, but it definitely already improves incident response. However, my personal highlight for this quarter - and this may sound odd at first - was performing a full Alpine Linux upgrade and subsequent reboot on one of our two Ceph cluster nodes, without any interruption whatsoever. You see, Ceph is not really designed to be operated at this small scale. But we cannot afford a whole server zoo just for storage. Ever since its inception, SourceHut has been operating with a single storage server (initially not Ceph). The anticipated redundancy has always been disk-level, not server-level (we have off-site backups for that). Not too long ago, we added a second storage server, and with this we still don’t have full server-level redundancy. But what we are able to do now is shuffle data around so that we can temporarily take down one server in a very controlled fashion. I feel like it took me a Ph.D. in Cephology to achieve this, so I am now busy writing a lot of documentation. But since this is something that every Ceph expert will (somewhat rightfully) tell you you shouldn’t be doing, I am also very happy right now that I learned the Ceph internals well enough to get it to work at all. Because offsite backups are cool, but upgrades without downtime are even cooler… ;) Everyone else SourceHut is 100% free and open source software, and the community is invited to participate in its development. Let’s take a moment to acknowledge the work of the volunteers who use and depend on SourceHut and sent along patches to improve it over the past few months. First of all, as always our build image maintainers have been hard at work. Taavi Väänänen has been keeping an eye on the Debian builds, fixing some issues with recent upstream Debian changes and shipping a Trixie image. ~arusekk was kind enough to send along an Alpine 3.22 patch as well. As Drew mentioned earlier, ~meroje also sent us a small improvement for our updated YAML parser, which makes round-trips a bit prettier when your build manifests are edited by e.g. git.sr.ht prior to submission. Thomas Chauchefoin has once again generously sent us a number of security bug reports, most of which had patches attached. Thanks Thomas! If you’re running your own SourceHut instance, make sure you’re subscribed to the sr.ht-admins mailing list to get notified about these issues. Simon Martin has also continued his work on various SourceHut improvements – chronicled in weekly updates on his blog . Simon fixed a few small bugs and added minor features all over the place, such as showing the branch name on man.sr.ht footers to help contributors figure out where to find the code for your wikis, or fixing race conditions that prevent lists.sr.ht from threading messages properly. Notably Simon also added some heuristics for auto-detecting new patch versions and superseding the old versions automatically – very convenient! In related lists.sr.ht news, Robin Jarry also stopped by this quarter to continue some maintenance work on the lists.sr.ht-ingress daemon. Big thanks to everyone to worked to make SourceHut better this quarter! Thanks for reading our update – we welcome you to discuss it on sr.ht-discuss if you have comments or feedback. See you again in Q4!

I’m pleased to announce that, as part of our broader plans to migrate SourceHut to Europe, and after many months of hard work, SourceHut has begun to accept subscription payments in Euro today – one of our oldest and most highly demanded feature requests. The pricing is similar to the US dollar pricing at €2, €5, and €10 per month (slightly more expensive than USD due to the Euro’s purchasing power), 1 with two months free if you pay annually. We have added support for SEPA direct debits, iDEAL, and Bancontact. As before, payment remains optional for most SourceHut features (and many features still do not even require an account). Financial support is available for those who cannot afford the lowest subscription tier. SourceHut remains committed to never pricing anyone out: hundreds of people have applied for and received financial aid for any number of reasons. Please apply for support if your situation requires it. On the other hand, all users who have the means to support SourceHut are kindly asked to do so, provided that the public alpha has reached a level of maturity equal to their needs. SourceHut depends entirely on our users for support, which keeps us accountable to them first and foremost. We have never accepted private investors and we cannot be pressured to compromise on our values, on user privacy, or on our commitment to publishing 100% open source software, thanks to the generous support of our community. We are pleased to integrate deeper into the European community, and upgrading the billing system to support the Euro is the next big step in that direction. We are now running almost all of our infrastructure in Europe, 2 and moving our revenue to Euro and our business operations to the Netherlands is another step towards completing our move from the United States. Thank you as always to everyone our community for your generous support – to those who pay for their subscription, but also to the hundreds of people who collaborate with us on our open source codebase, volunteer maintainers of many of SourceHut’s subsystems, and all of the people who host their projects here, or contribute to those projects. FAQ Which payment methods are available? We accept payments in Euro via: Credit card SEPA direct debit (IBAN) iDEAL Bancontact Will you support more currencies? Yes. The work to make our system multi-currency generalizes to other currencies. We intend to support GBP and JPY soon and we will add others as well. If you would like to request that we prioritize your currency of choice, please email sr.ht-discuss . Will you support more payment methods? It depends. We have added all of the Euro-based payment methods that we care to, but as we introduce other currencies we will endeavour to support the payment methods most suitable for those currencies. If you have a payment method you prefer, please email sr.ht-discuss to propose it. Note that we have already evaluated and elected not to support PayPal and cryptocurrency. Will you accept cash or bank transfers again? Once upon a time, we accepted annual payments in cash at each year’s FOSDEM, and we would accept manual bank transfers as well. Both methods have the advantage of not requiring nonfree JavaScript to support. We have not ruled out accepting payments in this manner again in the future, but for right now we have no concrete plans to do so. I pay in USD now, can I switch to Euro? Not yet: presently Euro is only accepted for new subscriptions. We will be making it possible to change your payment currency in the near future. Workaround: if your subscription term ends within the next couple of months, you may cancel your paid subscription and sign up again in Euro when it runs out. Why is SourceHut moving to Europe? Drew, speaking for himself: The most straightforward explanation for why SourceHut is moving to Europe is that, since I moved to the Netherlands, all of SourceHut’s staff live in Europe. And because we don’t use the cloud – we colocate servers that we assembled and own ourselves – it also just made sense to move our infrastructure to Europe, to be located close to the sysadmins responsible for it. This simple explanation obscures the reasons why I personally moved to Europe. Indeed I moved myself, and SourceHut, to Europe in part for strategic business reasons: the European open source community is stronger than the US, and moreover I believe that we have a competitive advantage in subjecting ourselves to tough European laws and regulations regarding privacy and consumer protections – our users would demand this level of attention to their needs regardless. There are personal reasons, as well, for moving myself to Europe, thus starting a process that would naturally lead to SourceHut’s business and operations moving with me. When I moved out of the United States, back in 2021, I was moving away from a political and social landscape that I no longer felt comfortable being a part of. I expanded on these feelings in detail on my personal blog at the time . To my dismay, the years since have shown that my foresight in this matter was prudent. I sleep better knowing that my business isn’t being coerced into complicity with the US Gestapo, or implementing policies that would be letting down our queer and trans users, women, or people of color who use SourceHut. We feel that we can provide a safer service to our community from Europe, moving user data and SourceHut staff out of the reach of US institutions. What are your plans for winding down the US legal entity? The process is gradual and conservative, as our revenue and therefore our ability to provide useful services to our customers depended, until now, on our US legal entity being able to process payments. We have adopted a cautious approach to make sure that we continue to have the resources we need to provide reliable service. All customers who signed up prior to today are still having their payments processed through the US system. Once we establish confidence in the European system we’ve built, and normalized our processes and procedures at low volume, we will prepare to migrate all existing users to the European payment processing system. This is expected to take place by the end of the year. We have one more server in the US, which is used for off-site backups. Backup data on this server is encrypted in Europe before being sent off-site. We will likely de-provision this server in the foreseeable future, but it is not a particularly high priority. Once these steps are complete it is our intention to dis-incorporate the US legal entity and cease operations in the United States entirely. This is, in part, an intentional inflation adjustment given that we have not updated our prices in 7 years.  ↩︎ All that remains is one server used for encrypted off-site backups.  ↩︎

Hello everyone! We’re back with another quarterly update for you on how things are going at SourceHut. It’s been a busy few months, and we’re happy to share what we’re working on with you all. Drew’s updates As readers of my personal blog might have noticed when I posted my LLM rant in mid-March, my assessment in the Q1 update that the LLM scrapers were “more or less under control” was a bit optimistic. Since then, I think that we have actually gotten it under control after deploying the nuclear option ( Anubis ). A few weeks after this blog post, I moved us from Anubis to go-away , which is more configurable and allows us to reduce the user impact of Anubis (e.g. by offering challenges that don’t require JavaScript, or support text-mode browsers better). We have rolled this out on several services now, and unfortunately I think they’re going to remain necessary for a while yet – presumably until the bubble pops, I guess. Thankfully, the success of Anubis and go-away in mitigating the LLM bots problem has finally allowed me to spend most of my time on our real priorities, which is why I’m getting ready to ship a huge batch of billing upgrades next week. I’ve been hard at work (1) upgrading our Stripe integration to the latest Stripe APIs, and (2) moving most of it into meta.sr.ht’s GraphQL API. You can expect to see major user-facing improvements in the billing UX from next week onwards. It’s kind of difficult to overstate how much work has gone into the billing system upgrades. The billing system was very simple and unsophisticated prior to all of this work, and the new system is much more capable, flexible, and maintainable – but getting there is a lot of work. Most of it is not obvious to users, since we have to move slowly and carefully to avoid recklessly breaking our revenue sources (or nasty bugs like overcharging users), but this batch of changes will bring the first big group of obvious user-facing improvements. I also encourage you to play with the GraphQL API a bit when this ships next week – the new billing system is designed with an API-first mindset and so a lot of its knobs and dials are available to you to play with. After this big patch series lands in production, I have some more internal (non-user-facing) work to do to move our payment processing into the EU, but it should be relatively soon that we’re able to add the single longest-awaited feature of the billing upgrades: support for more payment methods, like iDEAL and SEPA transfers. This effort saw its first signs of life recently – last week I collected a payment via iDEAL in my development environment! Billing (and bot control) have been my two major focus areas this quarter, but I have done a little bit here and there, so I’ll briefly summarize those other accomplishments. A small improvement to the GraphQL APIs includes the addition of better standardized errors, so that you can distinguish error conditions on the client side – for example, readily distinguishing “not found” from “access denied” in your GraphQL client. In more colorful news, I have installed a server in my bike shed at home. This may seem odd to include here, but I thought I’d mention it because I have used it to replicate an infrastructure environment comparable to SourceHut production, with ceph, libvirt, and k8s all making an appearance. I’ve moved many of the services I rely on privately in my personal infrastructure constellation from The Cloud (VPS providers) to the shed, such as NextCloud, Immich, and Miniflux. I’m using this setup mainly to learn more about ceph and k8s in an environment that has realistic production constraints (I actually need this stuff to work!) but which I can more confidently experiment on without breaking anything for SourceHut users. I hope to apply this experience to help Conrad with his efforts to roll out upgrades to the SourceHut fleet – which I’ll let him share with you now. Conrad’s updates This time around I’ll start with the rather tangible items: first, I am happy to announce that the pages.sr.ht API has gained basic support for ACLs. It’s not really well documented yet - and only works for custom domains - but it is possible to allow other users to publish your site. We are using this ourselves to collectively publish docs.sourcehut.org . Pending some better documentation, we hope that this feature may make it easier for teams using SourceHut until there is proper groups support. Sort of descending into the infrastructure realm, but still tangible: we made the repository that hosts our Kubernetes infrastructure public . After the DDoS , we became a bit cautious about publishing details about our infrastructure. However, we think it’s time to be more transparent about our operations again. This repository is only the stuff that runs in Kubernetes, which isn’t that much yet. But it will become much more, and we will evaluate publishing other bits and pieces over time. Speaking of Kubernetes, this is may be a good time to lay out the current plans. Much preparation work has happened, and there are currently two distinct milestones we hope to reach soonish. One is to route all incoming sr.ht traffic through the Kubernetes ingress controller. In the background, much or even most of the traffic might still be handled by the service VMs. But it would provide much better visibility into sr.ht as a whole, and would give us easy knobs to turn when experimenting with running services in Kubernetes. To achieve this milestone we need to change all service’s DNS records to point to the ingress nodes, but that means they’ll also have to handle incoming SSH connections. Hence, the main thing missing is the unified SSH dispatch handler that is currently in review. For sr.ht, we will probably end up employing an SSH ingress solution that leverages the power of CephFS, but we’ll take care to keep things working for smaller instances (or those simply not using Ceph). The second milestone would be running one actual service completely in Kubernetes. I was initially entertaining the idea that it could be meta, but for various reasons it will probably be something else. Regardless of the service, there are a few things missing. First, the service’s Redict instance will have to be migrated from the on-VM instance to the new network-wide sentinel-based cluster. This way, VMs and pods can share a cache, which is important for some services. Then, of course, there need to be containers. Those are conceptually straightforward, but some minor things will still need adjustment (like the currently chosen-at-random Prometheus port for observing the work queue flushing). Also, a first attempt to add Containerfiles sparked some ideas about our repository layouts, so we may take the occasion to clean those up a bit. Most of my other work is also still in-flight. We’ve added another server to our Ceph cluster, but the big Ceph upgrade is still to come. And I am evaluating Go-based database migration frameworks, so we can throw out even more Python code (alembic). Everyone else I’d like to set out a spot here in our quarterly updates going forward to talk about what all of you have been doing for SourceHut! We’re a free software project after all, and we regularly accept contributions from our users. First, I’d like to thank our tireless builds.sr.ht image maintainers – most of our builds.sr.ht images are maintained by members of their respective communities (i.e. Fedora images are maintained by Fedora users, not SourceHut). Haowen Liu, Maxwell G, Hugo Osvaldo Barrera, CismonX, and Cayetano Santos all shipped updates to, respectively, Ubuntu, Fedora, OpenBSD, FreeBSD, and Guix this quarter, with Cayetano stepping up as the new Guix maintainer – and we’re seeing off Unwox with a big thanks for all of their work on Guix support to date. In addition to his work on builds.sr.ht, Hugo Osvaldo Barrera added a long-awaited feature allowing users to rename more resources on SourceHut and set up the appropriate redirects, such as projects on the project hub. Long time contributor наб also made an appearance in this quarter to land a bunch of small improvements, such as improving to diff rendering on git.sr.ht and lists.sr.ht, and building a more sophisticated git-over-HTTP implementation – nice to have you back, наб! Eli Schwartz also returned to revisit his favorite subject, namely making SourceHut more deterministic in support of downstream distributions that download and checksum resources from SourceHut. I’d also like to thank Simon Martin for making their first contributions to SourceHut this quarter, landing numerous small improvements to lists.sr.ht, such as access control improvements which allow users to update the status of their own patchsets. Be sure to read Simon’s blog post about the experience! There were many other small contributions from our community this month – thank you to everyone who helped to make SourceHut better! Thanks for reading our update – we welcome you to discuss it on sr.ht-discuss if you have comments or feedback. See you again in Q3!

As you may have noticed, SourceHut has deployed Anubis to parts of our services to protect ourselves from aggressive LLM crawlers. 1 Much ink has been spilled on the subject of the LLM problem elsewhere, and we needn’t revisit that here. I do want to take this opportunity, however, to clarify how SourceHut views this kind of scraper behavior more generally, and how we feel that the data entrusted to us by our users ought to be used. Up until this point, we have offered some quiet assurances to this effect in a few places, notably our terms of service and robots.txt file. Quoting the former: You may use automated tools to obtain public information from the services for the purposes of archival or open-access research. You may not use this data for recruiting, solicitation, or profit. This has been part of our terms of service since they were originally written in 2018. With the benefit of hindsight, I might propose a different wording to better reflect our intentions – but we try not to update the terms of service too often because we have to send all users an email letting them know we’ve done so. I have a proposed edit pending to include in the next batch of changes to the terms which reads as follows: You may use automated tools to access public SourceHut data in bulk (i.e. crawlers, robots, spiders, etc) provided that: Your software obeys the rules set forth in robots.txt Your software uses a User-Agent header which clearly identifies your software and its operators, including your contact information Your software requests data at a rate which does not negatively affect the performance of our services for other users You may only collect this data for one or more of the following purposes: Search engine indexing Open-access research Archival You may not use automated tools to collect SourceHut data for solicitation, profit, training machine learning models, or any other purpose not enumerated here without explicit permission from SourceHut staff. This text, or something similar, will be included in our next update to the terms of service, which will probably ship around the time we finish setting up our new European billing system. A careful observer can see our views on scrapers elaborated on in our robots.txt file as well. It begins as follows: # Our policy # # Allowed: # - Search engine indexers # - Archival services (e.g. IA) # # Disallowed: # - Marketing or SEO crawlers # - Anything used to feed a machine learning model # - Bots which are too agressive by default. This is subjective, if you annoy # our sysadmins you'll be blocked. # # If you do not respect robots.txt or you deliberately circumvent it we will # block your subnets and leave a bag of flaming dog shit on your mother's front # porch. One can infer from the tone of the last sentence that attempting to enforce robots.txt is a frustrating, thankless task for our sysadmins. To add to these resources, I’d like to elaborate a bit more on our views on scrapers for you today. Scrapers have been a thorn in the side of sysadmins for a very long time, but it’s particularly important as LLM scrapers seize the entire Internet to feed into expensive, inefficient machine learning models – ignoring the copyright ( or copyleft , as it were) of the data as they go. The serious costs and widespread performance issues and outages caused by reckless scrapers has been on everyone’s mind in the sysadmin community as of late, and has been the subject of much discussion online. Aside from the much-appreciated responses of incredulity towards LLM operators, and support and compassion for sysadmins from much of the community, a significant minority views this problem as less important than we believe it to be. Many of their arguments reduce to victim blaming – that it’s not that hard to handle this volume of traffic, that we should be optimized to better deal with it, that we need more caching or to improve our performance, or that we should pay a racketeer like CloudFlare to make the problem go away. Some suggest that sysadmins should be reaching out to LLM companies to offer them more efficient ways to access our data to address the problem. Of course, not all software is necessarily able to be as resource efficient as Joe Naysayer’s static website. Moreover, LLM companies are not particularly interested in the more expensive route of building software integrations for each data source they scrape when they could go the cheap route of making us all pay for the overhead; nor should us sysadmins have to spend the finite time and resources at our disposal (often much more modest than the resources available to these LLM companies) negotiating with terrorists and building bespoke solutions for them . More important than any of these concerns is to address the underlying assumption: that these companies are entitled to this data. This assumption has varied roots, as benign as misplaced Libertarian ideals and as unhinged as the Rationalism cult belief that AGI is around the corner and everyone ought to be participating as best they can for the benefit of untold numbers of unborn future-humans. It is our view that these companies are not entitled to the data we provide, nor is anyone else. The intended audience for the publicly available data on SourceHut is users of and contributors to open source software who are accessing the data for those purposes. Indeed some profitable use of public SourceHut data is permitted, as one is entitled to by the Open Source Definition , but we do not wish to provide our data in bulk for any business, megacorp or startup alike, who wants to feed it into an LLM or do anything else with it which does not directly serve our mission, which is to improve open source software. We would not come to a special arrangement to share this data with any of these companies, either, even in the unlikely event that they offered to pay for it. We are funded by paid subscriptions, not by selling our user’s data. It is not ours to sell – something GitHub, with its own LLM products, would do well to consider more carefully. The data we have been entrusted with belongs to our users, and is dedicated to the commons, and we take our role as the stewards of this data seriously. It is our duty to ensure that it is used in the service of our users best interests. We have always put them first . We’re also researching go-away , a new option which may be effective with a reduced user impact (notably by not necessarily requiring JavaScript).  ↩︎

Today, I’d like to show off our new documentation site dedicated to SourceHut’s (and hence sr.ht ’s) GraphQL APIs: docs.sourcehut.org . Our work to move from the legacy REST API to a GraphQL API is still ongoing, and certain features we are aiming for (such as federation ) are still missing. Nevertheless, it is already the best - and preferred - way to programatically interact with SourceHut instances. If you’ve never used GraphQL before, there will be a slight learning curve. But we aim to make our documentation as helpful as possible, for beginners and experts alike. At the core of each GraphQL API, there is a schema that defines the data types and operations that the API makes available. This schema can be considered a machine readable API documentation. The schema language also has first-class support for comments, to make it more accessible to humans as well. Yet, even with comments, the presentation of a raw schema is not exactly the most ergonomic way for a human to learn about an API. However, given that all information is contained in the schema in a machine-readable way, it is quite easy to transform it into a form that is much more suitable for human consumption. Such tools are called GraphQL documentation generators. There are plenty of them out there, and we evaluated several. But none really checked all the boxes we were aiming for: Entire API output on a single page, so Ctrl + f is useful Little to no JavaScript Good support for custom directives Not too opinionated about the CSS/HTML structure And so, I couldn’t resist: I wrote my own . You almost can’t call it a documentation generator, because it does nothing but throwing a specific object at a template you have to supply. But this also makes it powerful: the template has access to pretty much any aspect of the provided schema. These templates are key here, so if you are interested, you can find the templates for our documentation at git.sr.ht/~bitfehler/docs.sourcehut.org . Given that we probably would have ended up writing a bunch of templates anyway (to properly customize the output of other tools), the result is pretty satisfying. Gockel is an extremely simple tool, and it will be easy to add features should the need arise. The underlying schema object that it exposes contains a lot of information, so there are many avenues to explore for improving the documentation. Even though we ended up not using it, I would like to give credit to spectaql - it was the most promising candidate out of the existing tools I evaluated, and the current layout of docs.sourcehut.org is certainly inspired by its default theme. For now, I hope docs.sourcehut.org will help you navigate SourceHut’s API. We are just getting started here, so there are likely some rough edges. If you find any bugs or have any ideas for improvement, please let us know .

Hello all! We’re back with another “What’s cooking”, after another too-long hiatus since September . We did promise to resume monthly updates, but in hindsight that seems a bit ambitious given everything on our plate. For now we’re going to aim for the more modest ambition of publishing updates quarterly. Drew’s updates My work has been proceeding as expected since September, with significant disruptions due to the LLM bot crisis, which to my consternation has occupied a substantial amount of time I would have rather spent on more important priorities. Nonetheless, I am happy to report that we have more or less got the bot crisis under control – though it consumed nearly all of my attention for several weeks, I can now occasionally enjoy up to several days in a row without one of our services being knocked out by LLM scrapers. In any case, I finished the git webhook upgrades I mentioned in September, though for now there is no option for a webhook recipient to intervene to prevent a push from proceeding. I’ve taken advantage of this and other GraphQL-native webhooks besides to make inroads towards updating our internal users of the legacy API, notably by moving the project hub to the new API and porting over the first webhooks from the legacy system to GraphQL webhooks. A matter of greater interest to most users is the upgrades I’ve been gradually rolling out for the billing system. There are numerous improvements in the pipeline, including support for more payment methods (e.g. iDEAL, SEPA bank transfers, etc) and more currencies (in particular Euro). But, this is a time-consuming process of gradual improvements and migrations. The billing system has been largely untouched since it was originally written in 2018, and there are thousands of paying users whose payments we’re processing every day, and we need those payments to flow correctly to keep the lights on. Each improvement is as small as possible and is carefully reviewed and monitored as it’s rolled out to ensure that the system works correctly without interruption. This work is part of the bigger picture work to move SourceHut entirely from the US to the EU, in particular by processing payments through our European entity and in accordance with European laws and regulations. We’re getting there! I hope to continue this work through the rest of March, with the ambitious goal of accepting our first payments in Euro in Q1 (but, more realistically, early Q2… it’s a lot of work). In addition to billing work, my main focus for the next quarter will be paying down more tech debt, and in particular finishing the deprecation of the legacy API. In the past couple of weeks we have made more aggressive moves to get rid of it, removing the legacy API documentation from man.sr.ht and disabling some features which are not still being used by anyone. I have some flashy end-user-facing features coming up soon as well – but let’s make it a surprise. Conrad’s updates Before I start, I’ll let you in on a little secret. Like many infrastructure engineers, I have evolved to have two questionable superpowers. One is stumbling across very weird issues that take days to debug only to be fixed eventually by a single-line code change. The second is to perform major refactorings, often coördinated across multiple codebases, resulting in plenty of code changes but no discernable difference in functionality of any of the touched components. Both these superpowers are greatly under-appreciated by managers (“Last week I stared at packet traces, then disabled a single sysctl on all servers, that’s all, really…”). Fortunately, there are no managers at SourceHut, because I have plenty of the second category for you! All our Python packages are finally PEP440-compliant! 🎉 This took so long and added zero value for our users, yet it was inevitable. And it did have the pleasant side effect of finally unbreaking builds for submitted patches again. Another epic, but mostly invisible undertaking was the recent “shared asset refactoring”. This one, however, brought significant quality-of-life improvements for developers. It also drastically changed the process for setting up a dev environment. The generic instructions have already been updated, but I am also working on a more detailed write-up. Piggy-backing on this big change we also renamed all executables to have a consistent naming scheme, which should help developers and instance admins alike. And yet more from the “big changes you didn’t notice” department: we finally consolidated all SSH key handling in meta.sr.ht. Previously, services requiring SSH access (like git and hg) would have their own copy of the SSH key table, which was prone to stale data and other inconsistencies. This unlocked the removal of various chunks of legacy API surface, with even more cleanup yet to come. Specifically, the entire SSH dispatching - which currently has very similar implementations in various services - will get a face-lift, improving code re-use and reducing complexity. Okay, I’ll give you a break - here is something very tangible: in lists.sr.ht, we removed the instance-wide configuration that blocks emails with HTML parts, putting list owners in full control to make their own choices about blocking such parts. As for what’s cooking right now: besides the above-mentioned SSH dispatching I am working on refactoring log file locations for all services and providing log rotation policies with our packages. We are also preparing to extend our Ceph cluster, and we’ll have to retry botched Alpine upgrades on two hosts which had an unfortunate combination of packages installed, triggering some very sneaky breakage in Alpine’s ceph packages. So, you know, lot’s of epic stuff that no one will notice. Unless it breaks something.