256 bit computing?

[zip accident]

This is probably over-simplified, but why can't computers be simple, with only one type, 256 bit chars for everything.

We'd need a new processor, with a bunch of 256 bit registers, but that has to be possible now, right?

[gilesgoat]

This is probably over-simplified, but why can't computers be simple, with only one type, 256 bit chars for everything.

We'd need a new processor, with a bunch of 256 bit registers, but that has to be possible now, right?

They always existed since long time ...

But it's called "microcode" inside a CPU, and/or VLIW ( Very Long Instruction Word ) where "various fields of that word control different HW parts", GPUs ARE made like that, they have "a smegload of FP registers" and "VLIW instructions".

If you look at the REAL "asm" of a shader it's all things like ( invented , don't remember out of my head the real mnemonics but to give the idea )

LOADFPINTOREG 0::0.331283,1:1.7756,2:0:3:0:4:0:5:1:..... etc. 32:-1.5764
LOADFPINTOREG 33::0.331283,34:1.7756,35:0:36:0:37:0:38:1:..... etc. 63:-1.5764
FMULADDSUBDIV 0,1:3,,4,5:6 sub:7:5:7 add:21:3:8 div 8:7:3

The fact is "does not really get simpler" .. gets more complex .. also takes a lot of silicon space I think.

Some GPUs probably have MORE than 256 FP "general registers" and not sure how long the instructions are but probably around 256 bits too.

So yeah they do exist to an extent and "microcode" also has stuff like 256 bits "instructions" ( or better to say "control signals ?" ) or even more bits.

[edit] - also correct me if I am wrong, this is a bit like to ask "why we can't have ONE universal CPU" and if I am not ( too ? ) wrong again "we kinda have it" because most complers generate "pseudo ASM" for "a virtual CPU that really does not exist" and then you have the "translator" step that "translates" that code into "real asm for the target CPU" ?

But probably "most compilers" don't even know about really what CPU they are targetting ( apart of course the "translator" stage ? ).

[KlaxonCow]

The 8087 floating point unit, launched in 1980, had eight 80-bit "extended precision" registers.

With the 80486DX, this floating point "co-processor" stopped being a separate chip and was integrated into the CPU.

With the Pentium, MMX was introduced in 1997. The eight 64-bit MMX registers are mapped onto the x87 floating point registers, reusing them in a slightly different way. You could have a single 64-bit value, two 32-bit values, four 16-bits values or eight 8-bit values packed into the 64-bit register and then issue commands that would work on all of these values simultaneously.

Then, in 1999, Intel added the SSE registers. These ones are actually eight new 128-bit registers (not mapped onto floating point or MMX registers). SSE1 just had four "single precision" (32-bit) floating point numbers. SSE2 added two 64-bit "double precision" values, two 64-bit integers, four 32-bit integers, eight 16-bit integers or sixteen 8-bit integers.

By 2011, the AVX extensions were added. These up the SSE registers to 256 bits. AVX-512, from 2016, doubles this again to 512-bit registers and ups the number of them from 16 to 32.

So, CPUs have been capable of dealing with larger quantities than their official "bitage" all along.

A reason not to up the general purpose registers beyond 64-bits is that, well, 64-bit pointers can address 16 exabytes of memory (or, in units you might comprehend, that's 16 million terabytes). This is a ridiculously large address space. There's no point going bigger than that for pointers, as you'll just be using up more memory for a pointer, but there's no conceivable way you've actually got more than 16 exabytes of RAM in your machine to ever warrant this.

And, as the MMX / SSE / AVX instructions suggest, performing instructions on 256-bit quantities ain't really that useful. Better to pack a bunch of more reasonably sized values - 64-bit, 32-bit, 16-bit, 8-bit - into that 256 / 512 bit register and do a bunch of values simultaneously. We don't typically need a single value with more bits, and would prefer to just perform the same instruction on multiple smaller values all in one instruction.

Indeed, if you had 256 bit CPU registers then you'd still be doing one instruction at a time, but with much bigger numbers. But range typically is not the problem, and it's performance that matters, so a single instruction that treats that 256 bit register as four 64-bit floating point numbers, say, is really what you want, as you're getting through the data faster, because you're doing four of them in one go.

It's the "natural limits" thing I mentioned before. Upping the bitage is good for small sizes - 8-bit to 16-bit, 16-bit to 32-bit, 32-bit to 64-bit - but it's diminishing returns, as do you ever really need to have a range more than 2^64 = 9,223,372,036,854,775,807 = 9 quintillion? It's not bigger numbers we want, but being able to process more of them in one go.

[zip accident]

I am not exactly set on 256 bits though it would possibly be nice. I think I remember hearing "who could possibly need more than 64k of memory?"

Anyhow, how about only having one data-type that always used all of a register, I mean you could have arrays, and pointers, but the type would always be the integer/char, you could perhaps use two for floating point, but that would be down to programming, not part of the language or processor.

[KlaxonCow]

I think I remember hearing "who could possibly need more than 64k of memory?"

It was 640KB that was more than enough.

The thing is that doubling the number of bits is exponential. 8-bits is 0-255. 16 bits is 0-65535. 32 bits is over 4 billion. 64 bits is over 9 quintillion.

I don't remember the exact comparison for that, whether that's more than the number of atoms in either planet Earth or the Solar System, but it's along those lines. The point being that even if you could somehow reduce things to storing a byte in a single atom somehow, then you still couldn't possibly physically manufacture a "memory stick" that large.

Certainly, if you went to 128 bits, then the max integer becomes 2^128 = 340,282,366,920,938,463,463,374,607,431,768,211,456 or over 340 duodecillion (more than 10^39), and we're definitely in absurd "could not possibly be even physically manufactured" territory there.

The point is that doubling the number of bits squares the maximum integer. So 256 bits would be 340,282,366,920,938,463,463,374,607,431,768,211,456 multiplied by 340,282,366,920,938,463,463,374,607,431,768,211,456 (minus one, as we start counting from zero). For small bitages, you can invoke the "640KB is more than enough for anyone" but it rapidly becomes, due to this all being an exponential thing, a more and more valid point - that memory sizes could never physically become that large.That the whole visible universe itself does not have "storage" that gargantuan, in terms of numbers of atoms.

Anyhow, how about only having one data-type that always used all of a register, I mean you could have arrays, and pointers, but the type would always be the integer/char, you could perhaps use two for floating point, but that would be down to programming, not part of the language or processor.

That's not a daft concept, because early computers basically were exactly like that.

There have been 9-bit and 36-bit machines. And a lot of these earlier machines did only have a single "machine word" and that's it.

But it's a trade-off / compromise thing. If you've only got a 256 bit machine word, then ASCII text - 7 bits - now wastes 249 bits per character. Or you cram 32 characters into those 256 bits... but you'd need a way to get individual 8-bits into and out of that machine word to parse and manipulate those characters.

You hopefully get the gist. A bigger machine word is better for some things, sure, but then it makes other things more awkward, unwieldy and wasteful. If instructions are a fixed size of 256 bits, then code becomes bigger - yet there's no way your CPU has over 340 duodecillion instructions, so you're always wasting the vast majority of that machine instruction. Pointers would have an address space so large, that it's not physically possible to ever manufacture that much RAM inside this physical universe. Et cetera.

And the thing is, processing 256 bits in one go, just means "bigger numbers" - a bigger range - but you're still only doing one value per instruction. As I say, the MMX / SSE / AVX stuff hints at it being more useful, in practice, to have that single instruction process multiple data (all packed into those 256 bits) to get through work faster.

Let's say we've got an array of 256 8-bit values and it takes one instruction per clock to process it. Then that's 256 cycles to process the array. If the array were 256 256-bit values and it still takes one instruction per clock to process it, then that's still 256 cycles. This doesn't help speed / performance, it just makes the range of values bigger (and, as we've covered, every time you double the bits, you're squaring the range, so this shoots off to "not practical for anything in real life" territory really fast).

But if you've got an array of 256 8-bits values but you're packing those 8-bits into a 256-bit word and you've got instructions that can process the individual 8-bits inside the bigger 256-bit machine word in one go, then now it's 8 cycles. That is actually making the CPU do more work in one go, and speeding things up. And, you know, that's the logic of the "SIMD" (single instruction multiple data) extensions - like MMX, SSE, AVX - in a nutshell. That's what those are doing.

You could do floating point in software. I mean, early CPUs didn't have floating point hardware - it wasn't in the x86 architecture until the 486DX, for example. But the real point of an FPU is that it's all "hardware accelerated". The FPU does all the mantissa and exponent stuff in hardware. Software can certainly do it, because with early CPUs, that's all you could do. But it'd take many, many instructions to do what the FPU hardware just does implicitly.

This again wouldn't really help things. Having an FPU and a built-in floating point type in the hardware is a form of "hardware acceleration". It's, like, the original GPU co-processor, so to speak. You can trade that away, but you're slowing things down for no real gain but the simplicity of having only one type.

It's worth noting that, at the binary / CPU level, there are no data types. No restrictions. You can read / write sequences of bits with integer instructions, floating point instructions. You can grab one 256-bit value as individual 8-bits bytes from memory, if you want. "Data types" are an abstraction we put onto the binary to make sure the accesses make sense. There's nothing actually in memory that records what "types" things are. We add this on in our compilers - to help us detect bugs - it's not actually there in the CPU itself.

Basically, you could do this. But making the machine word bigger and bigger has limits. It introduces wastage and problems. Unless you're splitting that 256 bit number down into smaller chunks (a la SIMD instructions) then you're not processing more values, only bigger values (and how big do you really need your numbers to be?).

In essence, "bigger is not always better". Bigger is better up to a point, but then any bigger just becomes unwieldy and wasteful, and much too awkward. That further scaling up really isn't winning you anything but more headaches. You want to do more useful work in shorter time with those bits, not just do bigger work with more bits in the same amount of time.


("Bigger is better" is the essence of superstimulation.

Scientists did a study where they noticed that, for a particular type of bird, the mother showed more care for the bigger eggs than the smaller eggs. The essence of this (slightly cruel) bit of evolution is that the bigger eggs are growing well. Therefore more healthy, therefore the babies are more hardy and have better chance of survival. So an instinct has evolved in the mother birds to prefer the bigger eggs.

And, as an experiment, they painted a basketball to look like the bird's egg. And, yeah, the mother bird got obsessive about nurturing and protecting this gargantuan egg, that couldn't possibly be real. But the thing is that her instincts are just "bigger is better", and this thing is monstrously big, therefore the supreme best of all. And she was obsessing over this massive fake egg, to the detriment of her real eggs. It was not a good thing.

But the underlying point is that evolution has bred into her this "bigger is better" instinct, but at no point do real eggs in reality ever get so massive, so there is no corresponding "okay, that's too big now" instinct to balance it up. So the egg can just get bigger and bigger and bigger, and this just stimulates - or "super stimulates" - the mother bird's instincts, until she's obsessive about this big fake egg.

Indeed, something similar could be seen to be happening in humans, to our detriment. Back in caveman times, when our evolutionary instincts were being forged, then the availability of food was not guaranteed. Hunters and gatherers could come home empty handed. There could be stretches of days or even weeks where you were starving.

That's where our instincts were forged. So we have instincts towards sweet things - a sweet tooth - and for fats, because that's our instincts directing us to the high energy foods.

But this is to our detriment today, as McDonald's and other food manufacturers exploit this by overloading their foods with sugars and fats. But we don't live in that same environment of food scarcity - at least, not in the Western world - but food abundance, so if you're always going for the sweet and fatty foods, you're overloading yourself with calories.

And you get obese. And a leading killer of humanity is heart attacks, strokes, diabetes, etc. because people are overloading on high calorie foods and getting fat.

The instinct now very much works against us. The once vital instinct to seek out high calorie foods to survive is now to our detriment, as folks can't help - following their instincts - to grossly overload on food (and the body - again, to assist survival - turns excess calories into fat. To store the energy away for later. To tide us over a period of "no food" later on - but in a world of food abundance, that never happens. So you just get fatter and fatter).

This is "superstimulation". Where you've just got a simplistic instinct of "bigger is better" or "more is better" that's not tempered by limits and common sense.

Humans are definitely vulnerable to this flaw. Obese people effectively slowly killing themselves, multi-billionaires with more cash than could ever be practical. The world too easily turns to shit, because we only have the "bigger is better" instinct, and it was never - in the environment that our evolutionary instincts grew - tempered by over-abundance. That simply never happened, so there is no corresponding "okay, that's more than enough now" instinct to limit it.

Scale is actually a problem unto itself. When a company has 10 employees then you can wing things organisationally, but if you've got 10,000 employees then, no, you've got to start putting in procedures and protocols and managers and things, or it'll collapse under its own weight.

The "mythical man month". One person does this much work. So ten people will do ten times that amount of work? Well, no, if they're interdependent on each other, then it's going to be much more complicated than that. Communication and organisation becomes vital.

Having too many people on this planet is really the root cause of an awful lot of our global problems. If we only had millions, not billions, of people then deforestation wouldn't be such an issue - nature could grow back new trees faster than we chop them down. Carbon emissions wouldn't be so bad, as nature could balance it and we'd have taken centuries and centuries longer to reach this point. And nature does have natural means of balancing out carbon, but we're just going way too fast with our CO2 emissions for it to keep up. Agricultural land - for crops and animals - wouldn't take up anywhere near the acreage that it does.

You get the gist. Scale is the essence of many human problems. That we could totally get away with it, if we weren't obsessed with "bigger is better" and making shit ever more massive without end. Economists might do a better job of things, if they ever realised that "infinite growth" isn't a real thing and were taught about "the growth ogive". Our economic systems have this assumption of ever-more growth - a recession is defined as two quarters without growth - but this is not physically sustainable, yet there it is, as a fundamental economic assumption: keep growing. Seemingly forever. Which can't happen. So, at some point - and one could argue on many fronts, we've hit those limits - this fantasy is going to clash horridly with the finite universe we actually exist inside.)

[Dio]

Memory bandwidth is a real issue. People don't use 64b numbers or even 32b ones everywhere. It's much cheaper than 8x as much cache memory.

I honestly can't think of a use for a 256b number outside of crypto, where it still isn't big enough.

[KlaxonCow]

Memory bandwidth is a real issue. People don't use 64b numbers or even 32b ones everywhere. It's much cheaper than 8x as much cache memory.

Indeed. And that's the other thing worth mentioning, that if the bitage is bigger than you need correspondingly more cache memory.

And with the speed difference between how fast CPUs can be made to run, and how fast our memory interfaces are, then you definitely want that cache. Without it, then CPUs might be capable of running at 5GHz, with multiple cores, or whatever but it'd be constantly stalled, waiting for memory, and effectively could only run at memory speeds (well, there are things a CPU can do that doesn't involve memory access... but it's pretty bloody frequent in any arbitrary code to be hitting memory all the time).

I honestly can't think of a use for a 256b number outside of crypto, where it still isn't big enough.

Yeah, me neither.

Certainly, even if you could think of a use - such as, I don't know, very big astronomical calculations or something - then it'd still be hard to justify a 256-bit CPU, especially for that use case, over just using "big integers" where you need to.

As it's always possible to string a bunch of integers together to create a bigger number. Look for overflow / underflow, and then propagate it along, so that you can create values of arbitrarily long length.

(This happened all the time on the 8-bits. In fact, the 6502 only had "adc" - add with carry - and "sbc" - subtract with carry - and you just set the carry to zero for the first byte. It was auto-assumed that you'd be doing "big integers" in the instructions themselves, as they always add / subtract the carry bit, and it's up to you to make sure it's zero / one before you start on that first byte.)

So even if you were doing infeasibly large astronomical calculations or cryptography or something, then it's not a common enough case to justify making the CPU 256 bits specifically for that, over just, you know, using "big integers" for those particular cases.

[zip accident]

Memory bandwidth is a real issue. People don't use 64b numbers or even 32b ones everywhere. It's much cheaper than 8x as much cache memory.

Indeed. And that's the other thing worth mentioning, that if the bitage is bigger than you need correspondingly more cache memory.

And with the speed difference between how fast CPUs can be made to run, and how fast our memory interfaces are, then you definitely want that cache. Without it, then CPUs might be capable of running at 5GHz, with multiple cores, or whatever but it'd be constantly stalled, waiting for memory, and effectively could only run at memory speeds (well, there are things a CPU can do that doesn't involve memory access... but it's pretty bloody frequent in any arbitrary code to be hitting memory all the time).

Cache might be an issue I'll grqnt.

I honestly can't think of a use for a 256b number outside of crypto, where it still isn't big enough.

Yeah, me neither.

Certainly, even if you could think of a use - such as, I don't know, very big astronomical calculations or something - then it'd still be hard to justify a 256-bit CPU, especially for that use case, over just using "big integers" where you need to.

As it's always possible to string a bunch of integers together to create a bigger number. Look for overflow / underflow, and then propagate it along, so that you can create values of arbitrarily long length.

(This happened all the time on the 8-bits. In fact, the 6502 only had "adc" - add with carry - and "sbc" - subtract with carry - and you just set the carry to zero for the first byte. It was auto-assumed that you'd be doing "big integers" in the instructions themselves, as they always add / subtract the carry bit, and it's up to you to make sure it's zero / one before you start on that first byte.)

So even if you were doing infeasibly large astronomical calculations or cryptography or something, then it's not a common enough case to justify making the CPU 256 bits specifically for that, over just, you know, using "big integers" for those particular cases.

I'm aware of one case outside those quoted, which is zooming in on the Mandelbrot set (possibly other fractals too). They need to be fixed point numbers, not floats or ints, but length is really important there.

Ascii being 7 bits really isn't a refutation, Unicode has replaced it and is already 64 bits and creaking at the seams.

I am mainly thinking we've already gone from 8 bit, to 16 bit, ... 64 bit why not skip a stage and go straight to 256 bits leaving out 128 bits.

[Dio]

Because we don't even use 64-bit very much.

Frankly, the only reason we went that way is because we need more than 32 address bits, and it's then a lot simpler to have a ubiquitous data type that's able to represent the size of anything. Do we ever go past 64-bit virtual addressing? ... probably not? That's the only thing I can think of that would drive >64 bit data types.

I think you're getting confused describing unicode as '64-bit'. The largest code point size used in any glyph system is 32-bit and this has 'lots of empty space', and UTF-32 hasn't really caught on because even UTF-16 has big space overheads. Yes, you combine multiple UTF points to give a single code. But you might as well argue that we need 256 bits because some common English words won't fit in 8 characters.

[KlaxonCow]

I'm aware of one case outside those quoted, which is zooming in on the Mandelbrot set (possibly other fractals too). They need to be fixed point numbers, not floats or ints, but length is really important there.

But as I mentioned, there's the "common case vs. rare case" thing.

Creating a whole new 128-bit CPU just for the Mandelbrot set? How often are we needing to do that, versus how much time, money and effort it takes to make a CPU?

Hopefully, you get my point. Sure, we can rack our brains to think up some particular case where 256-bits would be awesome. But one or two specific cases, here and there, doesn't justify the non-trivial effort of creating a whole new CPU architecture.

Indeed, somewhat the whole point of a CPU is to be general purpose.

Otherwise, might as well make dedicated specific-purpose circuits for everything - those would always beat the general purpose CPU in that specific task, as they can be exactly designed for it, but the drawback, of course, is that that's all they can do. They're specific purpose. Non-reprogrammable.

The whole underlying concept of a general purpose CPU is to be "Turing complete" and reprogrammable to do, well, absolutely anything.

Because then Intel can churn out loads of these CPUs and then we load different software onto them, as needed. Rather than having, you know, a "Microsoft Excel CPU", "a Pong CPU", "a Mandelbrot set CPU" and every single application is its own specific-purpose dedicated circuit. We don't do this because that would be ridiculously expensive and time-consuming, if you had to design a completely brand new dedicated CPU for every single application.

So you can say "256 bits would work here, and it would also work here" - give some concrete and possibly very good examples of where 256 bits would be totally awesome - but it doesn't justify a whole new CPU.

CPUs ain't cheap nor easy to design and build from scratch. So it needs to be general purpose. Universal. Good for a wide range of applications. Optimised (generally) for the common case.

Indeed, the vast majority of code only uses a handful of CPU instructions (this is one of the arguments behind RISC). A heck of a lot of code - at the machine level - is just "mov" instructions, setting registers to specific intermediate or memory-based values. Then you have your ALU instructions - "add", "sub", "inc", "dec", "and", "or", "xor" - to do basic arithmetic and logic (most code only uses basic arithmetic - even when code does do complex math, there's probably still vastly more code that's just "inc / dec", "add / sub" to just keep a count for loops and stuff. Worth noting that I didn't even include "mul" or "div" there, because for "house-keeping" arithmetic, it'll be bit shifts most of the time, as common data types are powers of two in size). Then you've got your branches, including the unconditional jump. Add in "call" and "ret" for subroutines, and I've probably covered 90% of all code right there.

I mean, you might still need those other 10% of instructions, possibly, but rarely so, in the grand scheme of things.

In a certain sense, you're not wrong. If we lived in "time and money are no objects" universe, where we never have to care about time or money, then we actually probably should have dedicated circuits for everything. Dedicated hardware would always out-perform the general purpose CPU for that specific task. The problem is that that's all it could do. That one specific task and nothing else. Forcing you to, yet again, start from scratch with a brand new dedicated circuit for some other task.

Ascii being 7 bits really isn't a refutation, Unicode has replaced it and is already 64 bits and creaking at the seams.

Let me just quickly check in the developer console.

Yeah, as I thought: "Content-Type: text/html;charset=UTF-8".

This web page is sent as UTF-8 (which is not at all unusual on the web, which is why I had the feeling it was worth checking whether the very page I'm on was UTF-8).

As Dio rightly notes, UNICODE defines "code points" - that this number is this character, that this other number is this other character, etc. - but it (quite deliberately) doesn't define the format of its delivery.

And UTF-8 is 8-bit. In fact, it's identical to ASCII in the lower 7 bits. It only differs when the high bit is set, and then we have multiple bytes to escape our way to the higher UNICODE code points.

I'm also on Linux, which also uses UTF-8 as its default character set.

Windows differs, in that they - fair play to Microsoft - embraced UNICODE early. But, ah, they might be paying the price for jumping so early, as they moved to 16-bit "wide characters"... and then UNICODE went beyond 64K characters and ruined it all. So now you've got "wide characters" and yet still need multiple bytes escapes as well, so it's all a bit of a mess. But I won't blame them too harshly for that, as their heart was in the right place - and it did all make sense at the time they did it - but 16-bits was not enough.

But, anyway, if you don't like that byte-based example, then we could go with the colour components - red, green, blue, alpha - of a "true colour" pixel instead. That's also byte-sized.

Or, indeed, as we're talking about a 256-bit CPU, then Windows 16-bit "wide characters" would also be wasting 240 bits in a "one character per machine word" unpacked format.

As me and Dio were saying, it's not easy to think of any common data types we generally use actually needing 256 bits. So most of memory would be full of wasted zero padding bits.

We mentioned cache sizes. But, like, cache is memory. This is just a general "data's unnecessarily taking up much more space than it needs to" in RAM (and likely on storage media too, because though you really should pack and compress data for storage, lots of files and formats are really just "memory dumps" to disk).

I am mainly thinking we've already gone from 8 bit, to 16 bit, ... 64 bit why not skip a stage and go straight to 256 bits leaving out 128 bits.

But, as I covered, this is exponential.

The address space of 8-bits (with byte-addressable memory) is 256 bytes. Obviously, that's pathetically small, so we need more than that. 16-bit address space is 64KB. Well, that was good for the old 8-bit machines, but that was soon broken. Okay, 32-bit address space is 4GB. That took longer for us to breach - as we've only just fairly recently done it - but we need more than that.

64-bit address space is 16 exabytes (or, in more familiar units, 16 million terabytes). Even bearing "640KB is more than enough for anyone" in mind, I think computers that have RAM sizes that exceed 16 million terabytes are still a very long way away.

Hopefully you get the gist of why it being exponential matters. We're easily breaking the RAM limits at smaller sizes, but a doubling of bits is a squaring of the address space. It very, very rapidly grows to gargantuan size.

I guess the answer to "why not skip a stage and go straight to 256 bits leaving out 128 bits" is that there will probably never be a general need for a 128-bit CPU. Dio's right that we're not even using 64-bits most of the time.

Doubling the number of bits is squaring the range. This gets very exponential very rapidly.