The main point that people are missing is that experienced engineers don’t want to work with people who think like the author of this article.
Protocol Buffers are not wrong, they simply have constraints, advantages, and disadvantages.
No language, binary format, text format, etc is free from advantages and disadvantages. All of them have different use cases.
If you are building a system where your data can be described by protobufs, it may be a good choice. If your data structures don’t mash up well and you have to manipulate them heavily, protobufs may be a bad choice.
Instead of pointing out use cases where a different serialization format may be better than protobufs and use cases where protobufs are better, and why, the author is spouting dogma about how protobufs are bad for every use case.
Be wary of working with developers who prefer to argue about why they hate certain technologies instead of providing useful data and ways to solve problems. You don’t always have to solve a problem just because you are aware of it but don’t go shouting from the rooftops that a technology sucks for every use case under the planet when that’s obviously not the case. The author’s opinion is more of: I don’t like protobufs. Not: protobufs are wrong
The author isn’t trying to say either of those things, really.
I think their real point was something like: many companies that have an Enterprise Service Bus architecture for their pile of polyglot microservices, have a dogma for the encoding of the data flowing over the Bus. And Protobufs, though good for some use-cases, are a particularly bad dogma to be stuck with. That is, when they don’t work for a use-case, they really don’t work for that use-case—and an ESB means having to try to shove a wire format into pretty much every use-case. So “when it’s bad, it’s really bad” is a bad property for a format aimed specifically at being an enterprise’s ESB-interchange-format dogma to have.
(Contrast to, say, JSON-RPC, which I’d place on the opposite end of the scale being described here. JSON-RPC is mediocre at best in terms of data fidelity, performance, etc., but it’s inoffensive—when it’s bad, it’s no worse than when it’s good. That makes it a good choice of ESB dogma... even though, in engineering terms, it sucks!)
No, the author is showing off how smart they are, and in an unprofessional and nasty manner to boot.
It's kind of you to gift the article with some genuinely thoughtful conclusions, but you're doing all the work there, not the original.
As the grandparent says, it's easy to poke holes in something, especially when disregarding important requirements that influenced its design. The fact that the OP can't point to any implementation of the "right" way to do things is telling.
I mean, all the arguments to support my conclusions, are their arguments. If someone spends 10 pages giving you facts and anecdotes and other types of data, and then uses them to support a "dumb" conclusion, they've still done useful work. They don't realize what exactly it is their data proved, but they did do the work required to prove something. Those facts and anecdotes support a conclusion, whether or not they feed it to you at the end/in the abstract.
And, that being said, I think they're right about the conclusion, too. A shorter way to say "Protocol Buffers are a bad choice of common ESB-bus format, even though being an ESB-bus format is the primary thing they're for and what everyone tries to use them for" is "Protocol Buffers are a wrong design." If something doesn't work when used to do the thing it's advertised to do, then it's broken, even if it can do something else.
The only difference between "Protobuffers Are Wrong" and my conclusion is that I'm making the implicit context of their argument explicit. Read between the lines of their argument—they are talking about the use of protocol buffers (specifically, gRPC) in an ESB-bus common-format scenario. None of their arguments make sense if they aren't.
What makes you say that Protobufs are bad thing? I'm finding the opposite - where you end up with everyone's favourite json parser, not able to handle UTF8 here and there, or some handle multi-line json, other handle comments, etc, etc. Also certainly slower, and certainly when you process data back you have to serialize, convert, check, etc.
For me protobuf+grpc gives you enough building block, not to shoot yourself in the foot - but rather start from something ready. You get on top of that census metrics, bi-directional communication, and yes there are limitations - it's HTTP2 only (right? correct me if I'm wrong), but maybe that's what you need.
Instead, I see folks - going back to sock(), using them directly, and then reinventing wheels of wheels of wheels.
Then discovery comes - how do I discover what's there and there? I've got used to the nice (forgot the itnernal name) where I can ask the "grpc" server (well stubby, whatever it was called) give me your endpoints - and it gives me the endppoints. Then I can talk to it.
Yes, it complicates build systems (need to generate damn protos), but you can also do custom protobuf generator plugins that generate custom tailored access, where you get performance gains, sacrificing features you don't need, like project perfetto - https://android.googlesource.com/platform/external/perfetto/...
It doesn't complicate just build systems, because of the weaknesses of it's type system and code generation it requires wrappers. Those get desynced and take maintenance.
This is the main point where the whole reason to use Protobuf falls apart - it doesn't work well enough to skip manual parsing.
And if you do manual parsing you might well go all the way in and use JSON with a schema and a reasonable library to handle these. You gain very little convenience wise and only a bunch of line bytes at best. Which typically does not matter for use cases of Protobuf that are not Google sized.
With certain build systems, this is a solved problem, or at least solved well, but it's a buy-in, so no great solution for MSBuild (for example), but works well in bazel.
>experienced engineers don’t want to work with people who think like the author of this article
- has opinions on interface design
- isn't afraid to be wrong publicly
- is brash on a personal blog
I dunno, this is mostly positive. I'd have to see how well they'd adapt to the much different context and goals of one of our design reviews, but this isn't an immediate red flag. There's a million ways to be bad at a job. I'll give them enough rope to hang themselves in a serious conversation rather than invent the idea that someone would be an unworkable perfectionist professionally just from being a type purist on the internet.
Brash on a personal blog is tricky. In this post's case, I'd worry that the blogger has a hard time separating technical deficiencies from professional incompetence. "Designed by amateurs" is an over-the-top and dubious claim.
As an experienced engineer I can hear the pain behind that dubiuos claim and no, I wouldn't want to work with someone who behaves like this all the time but I wouldn't mind someone who once in a while gets fed up show some emotions. It's all about how you handle the aftermath, ie. can you apologize to people you accidentally hurt along the way.
Their goals were not to create the most academically sound type system. They were focused on engineering challenges. Read the comments here ti understand the authors' priorities and obstacles.
I see protobuf as being designed to be serialized fast and with a low footprint by not too complex assembly or C or Go. All the trade-offs are correct with that mindset. Maybe the author is looking at this from an academic type-theory perspective, and that's why he can't see the "why" for each one of the trade-offs.
You're just not going to beat these formats in serialization speed with anything (just mmap the file, use. Good luck beating it, certainly not with protobuf). Seriously, use hdf5 for machine learning and you can restart experiments and ... well they start. They don't spend the first minute or 2-3 reading their data back in.
Protobufs focus on:
* standard (meaning it's perhaps a bad standard but it's standard) (also meaning of course you don't get to choose formats, or perhaps I should say your company will lose a lot if it lets developers choose alternatives to protobuf anywhere) (this is where the frustration comes from)
* streaming support (meaning you can write out and read in protobufs without having to keep the whole thing in memory)
* extendable (including sort-of kind-of backwards-forward-compatibility. Meaning old code can read in a new version protobuf, change something and write out a proto that still has the fields it didn't understand)
* language agnostic
* composability (meaning cat protobuf1 protobuf2 > resultbuf means resultbuf is deserializable)
* an attempt to efficiently store integers (meaning it's integer encoding format is bloody complex, but "space efficient", except not quite so efficient it doesn't need compression at which point, why bother ?)
And in this protobuf will beat the above formats (except cap'n proto).
None of the alternatives to protobuf existed when it was invented though. The only alternative in existence was ASN.1. And it beat that, by a LOT.
All of those significantly post-date protocol buffers, though. It would have been great if they had existed before PB, so we could have used them at Google. But, we had to invent something because nothing that existed at the time was suitable, and the migration cost to use something else now would be astonishing. I'll readily admit that a new company would likely be better off standardizing on one of these. However, standardizing on Protocol Buffers is still way better than having a mish mash of different formats, even if all of those formats are individually better.
I'd probably pick Cap'n Proto or Flat Buffers if speed were paramount, in a grass is greener sort of way. I haven't used either of those technologies, though, just read about them. I'm also cool with plain JSON, which is beautiful from an ease-of-getting-started and universality perspective. I also think GraphQL is super compelling, and there's something to be said for records-as-in-SQL.
Mainly, I just think that interchange formats should be boring, simple, and ideally not incredibly slow. Protocol buffers at least meets those bars, even if it doesn't meet the consistency bar that the article desires. The whole "stop being a hipster and just use X" meme comes to mind. Probably realistically X is JSON in this day and age.
The problem with JSON for internal formats is that there’s often only one consumer and producer, so documenting the format rarely happens. Later, when you want to reimplement one side, you learn that there is no “one place” where you parse the JSON, but that you hand bits and pieces of it to completely unrelated areas of code.
Figuring out these as hoc formats is nigh impossible, so you end up just looking at the data over the wire and writing code that parses what you see. Except then you have broken code that doesn’t parse the less-common variants that have extra fields, don’t have extra fields, sometimes have their fields as stringified ints instead of just binary ints, etc.
Virtually every time I’ve seen JSON used as an internal interchange format, reverse engineering that format a few years later.has become a massive, error-prone, tedious, and time-consuming task. Save yourself from this ahead of time and pick formats that require a predefined structure, like protobufs.
This is the classic static vs dynamic typing question. Suffice to say, I don't think there is sufficient science or anecdata on the issue to really give a solid answer.
I am a proponent of both static (Rust) and dynamic (Ruby) typing in programming languages. They both have their place, and if there are bugs you can always fix the code.
The one place where I don’t think dynamic types have their place is in internal interchange formats. These can live forever, and they can change subtly over time. It’s the living forever part that changes the calculus.
Don't forget integer type defined by implementation. Very easy to shoot yourself in the foot in when consumers/producers are not written in the same language, even more if it's a mix of dynamic and static languages.
Nothing beats a well defined and commented schema.
Protobufs also run anywhere, there are libraries of it ported to almost any machine.
When my team was deciding on a binary serialization and RPC communication format, we needed something that ran on a Cortext m3 with no malloc, and every major phone platform (including WP at the time) and desktop OS.
Protobuf did not support maps for a long time. In practice using a list of key/value pairs works fine for most use cases. Sending the message over the wire is obviously always O(n); if you need fast inserts/lookups in code, then it usually makes sense to convert to and from the appropriate in-memory data structure for that purpose.
That said, I think maps are a fine feature for a serialization format to have. But, I haven't gotten around to adding them in Cap'n Proto because I have yet to hit a use case where I wasn't happy with a list of key/value pairs.
For what it's worth, I like them primarily because I don't mind using proto-generated objects as pieces passed around the stack (at least at higher levels of the stack). I lose consistency if I can do that for some objects, but am required to convert objects if they would need a mapping type.
It follows sorta the same reason that Clojure became a fairly widely used lisp – mapping types are extremely common in real world programs. Having them supported first class makes a lot of common tasks quicker to get through.
No, you're right. The canonical way to store a map in capnp is to use a List<Pair<key, value>>. Personally it doesn't bother me that much (mostly from an aesthetic standpoint) because it is not meant as an in-memory, working data structure.
What does bother me is that in the above generic type, "key" and "value" must be pointer types (i.e. non-scalar). The reasons are not unreasonable but it's definitely obnoxious in practice.
Arent you doing exactly what you are accusing the author of? One could say - "People are not wrong, they simply have constraints, advantages, and disadvantages". Drawing a conclusion about whether all experienced engineers want to work with the author based on a single article or opinion appears just as myopic.
Protocol buffers are definitely not perfect. As of 2014 in Java the biggest issue was that you could not have different versions of protobuf [easily] running in the same VM due to the fact that protobuf generated version-dependent implementation code for IDL. So if one library depended version X of protobufs and another library depended on version Y, you had a problem because Java will only load one protobuf jar file. (Maybe it's been fixed but it was a problem then.)
That said...I used protobufs to encode the log of Tungsten Replicator, a replicator for MySQL. We did not have a single version related error while I worked on it over a period of years. That was essential given the fact that replication logs are an on-disk format that must remain stable across version upgrades. Our logs in aggregate contained billions of transactions--you can't not read an older log file just because you are running a new software version. The claim that protobufs do not handle versioning is nonsense.
Java only allows one version of any class within a single class loader. That's just how it works. So the problem was at the very least a limitation of the protobufs Java implementation.
Speaking of which, the generated code was incredibly convoluted with what seemed like tangled dependencies on the underlying protobufs library. I used to dread debugging it--sometimes I would have to go into it to figure out what the upper layers were doing wrong. Luckily it did not happen very often. And I don't recall ever hitting a bug in protobufs itself for all that the code was virtually unreadable.
I'm an outsider, but maybe it's a problem of their interaction?
Most likely, the vast majority of Protobuf use is with code generated to a single, canonical location (which leads to the problem described above).
But hypothetically, it would be possible for code to generate and refer to two different versions. That doesn't sound impossible. Maybe just improbably given the way things are typically set up.
If we suppose that his conclusion is using boolean logic, then what you're saying is a strawman because of his last claim; namely, protobufs are bad if "[...] && !Google":
> They're clearly written by amateurs, unbelievably ad-hoc, mired in gotchas, tricky to compile, and solve a problem that nobody but Google really has.
This dovetails with other arguments that I've seen recently that are becoming more frequent:
Have we entered a new world where the lessons of companies working at massive scales are not only generally superfluous for smaller scales, but are actively harmful?
> Have we entered a new world where the lessons of companies working at massive scales are not only generally superfluous for smaller scales, but are actively harmful?
I think so, yeah.
Microservices turn out to have a lot of negative consequences, and their positives work best when you have dozens or hundreds of developers. If you've got a handful of developers... not so great.
Your argument really depends on the use case. If you have discrete, well-factored operations microservices can make even small systems easier to deploy and manage. For example, you do the front-end API in Java (easier to build secure, debuggable systems with good RDBMS access) and backend analytic services in Python (easier to scrape data out of XML/JSON). Splitting them up into 2 or more microservices can simplify development, CI/CD, and deployment.
Whenever I see large numbers of microservices anywhere my null hypothesis is that some organizational disfunction is leading teams to factor applications into unnecessarily small pieces.
No. But you have to examine use cases carefully including the assumptions. I think this has always been the case but people (in my experience at least) get a little dazzled by the massive scale of companies like Facebook and try to apply their solutions to problems for which they are simply not applicable.
Maybe that's the main point, for you. I don't think I'm missing anything :)
You can take your logic and apply it to anything.
For example killing innocent babies:
"Killing babies has constraints, advantage and disadvantages. No other action in the universe is free from having advantages and disadvantages. They all have different use cases...
I don't get why the author is 'spouting dogma' about how killing innocent babies is bad.
Be wary of people who argue about killing babies etc etc."
This relativism you're expressing can be seen as ridiculous when applied to killing babies - it can actually seem to make sense when you use it to justify your own biases - which is what you're doing. Of course doing that has it's advantages and disadvantages, etc etc no opinion is better or worse everything is relative :)
Protocol Buffers are not wrong, they simply have constraints, advantages, and disadvantages.
No language, binary format, text format, etc is free from advantages and disadvantages. All of them have different use cases.
If you are building a system where your data can be described by protobufs, it may be a good choice. If your data structures don’t mash up well and you have to manipulate them heavily, protobufs may be a bad choice.
Instead of pointing out use cases where a different serialization format may be better than protobufs and use cases where protobufs are better, and why, the author is spouting dogma about how protobufs are bad for every use case.
Be wary of working with developers who prefer to argue about why they hate certain technologies instead of providing useful data and ways to solve problems. You don’t always have to solve a problem just because you are aware of it but don’t go shouting from the rooftops that a technology sucks for every use case under the planet when that’s obviously not the case. The author’s opinion is more of: I don’t like protobufs. Not: protobufs are wrong