Follow

when you think about it, the idea that software should scale is actually really weird. "sure this garden is nice, but how nice can it be if it doesn't grow to cover the entire surface of the earth?"

(just thinking about runyourown.social — the "More on scale" passage in particular, and how it's weird that Darius has to present "the notion that software does not have to scale" as some kind of tendentious heresy instead of, like, a default so obvious that it goes unstated)

@aparrish this weirdness kinda feeds my "most software isn't engineered" take -- it seems to me that very little of the software industry designs with limits in mind

@aparrish The related idea that a modern corporation must always seek growth at all costs.

So do cancer cells.

@aparrish That's pretty much how I feel when I suggest that the Web shouldn't be so relient on JavaScript, and that both and web native have their place.

@aparrish i think it betrays the real purpose of writing software for many people, which is that it's only really meaningful as a vehicle for power

@aparrish /me unpacks old internet uncle mode

Actually, scale used to mean that it supports more users/things *on the same hardware*. It was a question of how well optimized the software was. I still support this idea of scalability as a measure of good engineering (more or less; separate discussion).

Scaling to multiple cores (vertically) or machines (horizontally) has often been used to avoid optimization. Which is kinda dumb, unless you can just throw money at your problem.

@jens
> Scaling to multiple cores (vertically) or machines (horizontally) has often been used to avoid optimization.

Often, but not quite.
Sometimes a task done in parallel with multiple low-power cores is completed faster than when done with one high-power core. Like compression, for instance. If you still don't use pigz, try it.

A lot of times, a task run by multiple machines provides the fault-tolerance and possibility to safely maintain one node while the other is working.

@aparrish

@drq @aparrish Of course some tasks benefit from parallelization. But that's rarely the case when people throw around the term "scale".

Let's say you have N cores. You may complete a task in (T/N)+D time instead of T time by spreading it to those N cores. The D is a delta you introduce by managing the parallelization, it's often small enough, but...

... well, first off, this is wall time, not CPU time. That may be all you care about, sure. But then you may want to...

@drq @aparrish ... scale to more tasks, M. Now your total wall time is M*((T/N)+D), or M(T/N) + MD. It becomes particularly visible when M=N that parallelization can hurt, because instead of just using a single core per task (each in time T in parallel), your extra parallelization introduces an overhead of MD.

It may still be the better choice for reasons of CPU utilisation, or because your single task wall time matters, etc.

Anyway, optimizing that...

@drq @aparrish ... initial single core T is always going to get you benefits without such a cost.

@aparrish Part of the design of Cwtch is that it davka DOESN'T scale well -- it becomes less efficient the more groups use a specific Cwtch server, and also it's very easy for groups to move servers, leading to loads naturally spreading.

@aparrish
(Servers are also -- assuming cryptography isn't broken -- untrusted with any data other than "some information is being exchanged, probably".)

@nightpool https://openprivacy.ca as an organization --
Sarah Jamie Lewis, Erinn Atwater, Dan Ballard mostly as the individual people coding.
Code is on their Gogs instance: https://git.openprivacy.ca/cwtch.im/ui
https://git.openprivacy.ca/cwtch.im/cwtch
(and other repos. Somewhere there thee's in implementation of OV-Net anonymous voting that they've used.)

@nightpool For the people themselves, most of their public online presence by volume is.. on Twitter.., but they probably have other places, too -- Sarah Jamie Lewis has an interesting research-notes/thought-dump blog at https://fieldnotes.resistant.tech/ .

@aparrish I think it's a natural thought for me that software should scale because I sort of grew up with Olympiad in Informatics in the back of my mind and the awareness of asymptotic complexity
@aparrish and now I sysadmin this very Olympiad and maintain its software
and I find that if you let people make their own little contests and suddenly you have 1500 contests then suddenly every page takes 15 seconds of server CPU time to load

@aparrish dueling gardening philosophies: kudzu vs rosebush

@phooky proposal to call twitter, fb, etc. "invasive social networks"

@aparrish love it!

would you then consider finger, Usenet, irc indigenous social networks?

@phooky haha I don't know if I'm ready to set up a whole for real taxonomy here, whoops

@aparrish @phooky

I like the double-meaning: invasive in the sense of invasive surgery.

@dynamic @aparrish @phooky somehow too benign. Invasive species still very much real organisms rather than parasitic?

@phooky @aparrish I take it you've not seen the rosebush that ate my neighbor's porch.

@aparrish You're losing me a little 'cause I kind of *do* want a garden the size of the planet?

@aparrish This is a great insight!

As a SW dev myself, I feel it's part of being competent for SW devs to understand how well their code scales. But the only actual requirement is that it scales "appropriately."

In some cases "appropriately" might not be very much at all.

Or even, if you're worried about centralization or and things, "appropriately" might actually mean "not too much."

@aparrish I have been toying a lot with the metaphor that the internet is the most ambitious terraforming project undertaken by civilization so this comparison resonates

@aparrish this is one of the key reasons software has become so bad.

(the same diseased thinking infects other products, but it happens slower).

@aparrish All I can see in this thread is the different type of Factorio players. Just one more ore patch...

@aparrish i'm just going to print it out and tape it to my monitor

BAD SUGGESTION NO BAD DON'T DO THIS 

@nightpool @aparrish Screenshot dunking.

BAD SUGGESTION NO BAD DON'T DO THIS 

@gaditb @aparrish screenshot dunking but in the style of https://printedinternet.tumblr.com/

BAD SUGGESTION NO BAD DON'T DO THIS 

@nightpool @aparrish silkscreen -shot

@aparrish Maybe that phrase is thinking more like air and water. Or perhaps grain.

@aparrish Nevertheless, it's always bothered me. Similar to how things are deemed failures when the only have a few million users.

@aparrish@friend.camp I used to overengineer my software because I was worried about being able to scale it up, now I overengineer it because it's fun

@aparrish

thank you

+ more colonial legacies
+ also capitalism's neverending search for new markets

it's about the humans (and their minds) more than any intrinsic qualities of software

@aparrish The problem is that Silicon Valley (and capitalism in general) holds the same criteria of success as cancer.

@aparrish Depends on what you mean by "scale".

Scale along what metric? Scale depending on what metric?

@drq @aparrish well, the tech industry use of the word "scale" is similarly devoid of meaning. So, the point stands.

@rysiek @drq @aparrish as in my other comment, it used to default to "scale with user/usages". Now it defaults to hardware, either cores or machines.

It's a little sad, really, that this default has shifted.

@aparrish If the software is meant for a computationally demanding task, for instance, like rendering an image or a video, and it doesn't scale along my system resources (like, it only uses one CPU core to work), this is a bad piece of software, because it leaves a lot of its own performance on the table.

@aparrish If the task is comparatively lightweight - like shoveling text strings around and the software demands a lot of resources per user per task instance, I would also say that's a bad piece of software because it scales badly along the user count, costing (me and the environment) too much to operate.

Which is one of pet peeves I have with Mastodon, for example. All it does it sends and receives text. Why does it need so much resources? Because it's thrown together using RoR, which is famously voracious.

@aparrish But the same Mastodon, supposedly scales well along multiple nodes in a cluster if you want to run a large fault-tolerant infrastructure or something.

Speaking of which, centralized web services always boasted how they "scale better". But over time we see that they really don't: Facebook and Twitter don't really scale along their sheer number of users, which causes problems with moderation fake news, political polarization, etc. etc. On the other hand, I see decentralized systems (like E-mail and Fediverse) scale awesomly by individual servers being largely detached from one another. I mean, E-mail is still with us since the 70s, and is used by people of all walks of life. Fediverse spans across all the social and political spectrum.

@aparrish Same with hardware. There's seemingly a threshold to the number of compute devices working in parallel efficiently. It's called Amdah's law. You can only throw so much CPU cores at a problem until you start hitting diminishing returns and adding more resources to the system does almost nothing.

In other words, "scaling" is complex, and demands context. It means nothing without it.

@aparrish @aral

Would it be nice to have a garden where I can chill? Sure!
Where I can invite all my friends and everyone has a good time? Hell, yea!
Where I can invite the whole world? Why would I do that!

@aparrish IMHO it's false to compare physical with immaterial.

Moreover, scalability from my point of view is to extend the software to solve a problem the original author not even imagine for. Serving more users is just a subset.

When the robustness, reliability, stability, and simplicity is software craft, the scalability and wording is software art.

Sign in to participate in the conversation
Friend Camp

Hometown is adapted from Mastodon, a decentralized social network with no ads, no corporate surveillance, and ethical design.

<svg xmlns="http://www.w3.org/2000/svg" id="hometownlogo" x="0px" y="0px" viewBox="25 40 50 20" width="100%" height="100%"><g><path d="M55.9,53.9H35.3c-0.7,0-1.3,0.6-1.3,1.3s0.6,1.3,1.3,1.3h20.6c0.7,0,1.3-0.6,1.3-1.3S56.6,53.9,55.9,53.9z"/><path d="M55.9,58.2H35.3c-0.7,0-1.3,0.6-1.3,1.3s0.6,1.3,1.3,1.3h20.6c0.7,0,1.3-0.6,1.3-1.3S56.6,58.2,55.9,58.2z"/><path d="M55.9,62.6H35.3c-0.7,0-1.3,0.6-1.3,1.3s0.6,1.3,1.3,1.3h20.6c0.7,0,1.3-0.6,1.3-1.3S56.6,62.6,55.9,62.6z"/><path d="M64.8,53.9c-0.7,0-1.3,0.6-1.3,1.3v8.8c0,0.7,0.6,1.3,1.3,1.3s1.3-0.6,1.3-1.3v-8.8C66,54.4,65.4,53.9,64.8,53.9z"/><path d="M60.4,53.9c-0.7,0-1.3,0.6-1.3,1.3v8.8c0,0.7,0.6,1.3,1.3,1.3s1.3-0.6,1.3-1.3v-8.8C61.6,54.4,61.1,53.9,60.4,53.9z"/><path d="M63.7,48.3c1.3-0.7,2-2.5,2-5.6c0-3.6-0.9-7.8-3.3-7.8s-3.3,4.2-3.3,7.8c0,3.1,0.7,4.9,2,5.6v2.4c0,0.7,0.6,1.3,1.3,1.3 s1.3-0.6,1.3-1.3V48.3z M62.4,37.8c0.4,0.8,0.8,2.5,0.8,4.9c0,2.5-0.5,3.4-0.8,3.4s-0.8-0.9-0.8-3.4C61.7,40.3,62.1,38.6,62.4,37.8 z"/><path d="M57,42.7c0-0.1-0.1-0.1-0.1-0.2l-3.2-4.1c-0.2-0.3-0.6-0.5-1-0.5h-1.6v-1.9c0-0.7-0.6-1.3-1.3-1.3s-1.3,0.6-1.3,1.3V38 h-3.9h-1.1h-5.2c-0.4,0-0.7,0.2-1,0.5l-3.2,4.1c0,0.1-0.1,0.1-0.1,0.2c0,0-0.1,0.1-0.1,0.1C34,43,34,43.2,34,43.3v7.4 c0,0.7,0.6,1.3,1.3,1.3h5.2h7.4h8c0.7,0,1.3-0.6,1.3-1.3v-7.4c0-0.2,0-0.3-0.1-0.4C57,42.8,57,42.8,57,42.7z M41.7,49.5h-5.2v-4.9 h10.2v4.9H41.7z M48.5,42.1l-1.2-1.6h4.8l1.2,1.6H48.5z M44.1,40.5l1.2,1.6h-7.5l1.2-1.6H44.1z M49.2,44.6h5.5v4.9h-5.5V44.6z"/></g></svg>