Not sure why this got so many upvotes, also the landing page is not great, its better to look at the paper (see link below).
Seems to be a columnar storage format that addresses some shortcomings in parquet. Thing is, though, that of all these formats the real winning feature is compatibility, which is (obviously) very hard to improve on, as anything new immediately loses.
Parquet is unfortunately very good just by virtue of being first, and so widely supported. The most widely used parquet version is the oldest version from 2013 (as per the paper itself), so parquet itself couldn't even supplant parquet. If you want to improve on it, you need to bring some serious results, which I don't think f3 does.
Also, my main gripe with parquet (single table per file) is not even addressed, so, also the name is a bit hyped up.
Also also, it seems to go out of its own way to include a compiled wasm binary for decoding, yet requires flatbuffers to parse that blob? Kind of defeats the purpose.
Its main result seems to be improved random access which, although certainly welcome, is not the point of columnar storage, as columnar storage was invented to exchange random access for something else: fast analytics. F3 seems to sacrifice fast analytics for the wasm decoder. I don't get it.
Maybe I'm being too cynical. Can someone help me out here?
This bit is quite genius, rather than depend on a language-specific SDK/lib for working with the formats you can fallback to exported WASM methods if none exist:
> "Each self-describing F3 file includes both the data and meta-data, as well as WebAssembly (Wasm) binaries to decode the data. Embedding the decoders in each file requires minimal storage (kilobytes) and ensures compatibility on any platform in case native decoders are unavailable. "
WASM has strong tried and proven sandboxing. We basically can build on nearly 30 years of experience. The decoders don't need a lot of access, they can basically be pure functions.
If this will pan out security-wise I don't know. I'm more worried that it will be so slow that no one will use it. Interesting idea, though, and I can see applications outside of the "big data" realm this apparently targets.
OOM killing in WebAssembly is trivial, since it’s all in a growable linear memory. All the runtimes I’m aware of have a simple maximum memory setting, and they’ll trap any allocation requests after that point.
Attack is not just on file format itself. Based on the function signature it's possible for a single decoder to generate infinite bytestream - makes a lot of headache to reader implementation - implementing STRLEN is no longer trivial question.
Either engines should put some limit (e.g. VARCHAR(2000) to enforce length to be limited to 2000, but there are some other engines supporting unlimited BLOBs), or decoder should give a hint what is the maximum length it will yield. Unfortunately current research level project does not have such considerations implemented yet...
For images, it makes sense: people dealing with 16k x 16k PNGs are uncommon. Give them an error message that tells them the setting to bump. But what should be the threshold for "big data"? I'm sure it will follow Zipf's Law, but the tail will be fatter.
WASM implementations are fairly mature now, but if there was e.g. an image file format with embedded WASM that needed to execute before you could view it, it would become the new low-hanging-fruit target for 0-click RCEs - whether it's exploiting the WASM engine itself or some other attack surface that's influenceable via it (See also, the FORCEDENTRY JBIG2 exploit).
That exploit targeted an integer overflow in a bespoke Apple sandboxing mechanism. Bespoke sandboxing mechanisms have weird bugs.
Not that Wasm engines don't have bugs, but the whole point is to have an extremely solid, well-specified and efficient implementation of a widely accepted bytecode format. We can scope down the capabilities given to any program to a minimal set.
Bugs are near-inevitable, and mitigations are the last line of defence. Scripting engines are excellent for bypassing mitigations (iiuc in the case of the FORCEDENTRY exploit, it was used for adjusting ASLR'd offsets).
As a random example that's an area of personal interest to me, I know of 3 distinct methods of achieving userland ROP execution of the Nintendo Switch 2, and all three rely on the (ab)use of a scripting engine (even if they aren't a vulnerability in the scripting engine itself).
But seriously, if your format requires extensibility to the point that it embeds a bytecode, especially a Turing-complete bytecode, what format are you going to choose? Just design a new one? That's how you end up with a scripting engine with three ROP exploits.
> WASM has strong tried and proven sandboxing. We basically can build on nearly 30 years of experience. The decoders don't need a lot of access, they can basically be pure functions.
I've heard that kind of sentiment many times before. It's not a good (thought-terminating) mindset to have for any secure software.
There are several WASM implementations, WASM is just a format. "Pure functions" are pure at a superficial level. Many people say that they don't mutate global state, but they do ... it's just hidden. The decoders "not needing a lot of access" doesn't matter if the WASM engine is pwned through arbitrary code execution inside the environment, or if it's contorted to bypass the access control you are mentioning through various side-effects.
> The decoders don't need a lot of access, they can basically be pure functions
They don't currently either do they? It's the tight coupling of the interface layer no? I'm not sure this would be faster, or more secure so reliability might be the best usecase?
All WASM can do is transfer bag of bytes between module runtime and host. So yes, so yeah it can just decode into a buffer. Even you use wasm components to give it I/O, you can still make these go to buffer.
I don't understand how that's supposed to work. What does the decoder decode into? That's gonna depend entirely on the kind of data, right? For some formats, it's gonna be a stream of bytes; for others, a 2D plane of pixels; others again will need vertexes, 2D planes of pixels and UV maps; for some, an object graph will make more sense.
It sounds neat, but feels like it might fall apart with higher-complexity formats. What does an embedded decoder for a PDF look like? I guess since they are tightly-coupled to the file bytes themselves, the author of the file gets to choose what formats make sense, but not all formats have a one-true-decode-step.
Despite the name seemingly implying otherwise, F3 is an alternative to columnar storage formats like Parquet; the goal is not to support every conceivable encoding of every file type such as a PDF. Think of the use cases being more like "What if you used a specialized compressor and need a custom block decompression algorithm" or "Decode internal format into Arrow output" or something like that.
This isn't using WASM to solve the "how can I make my file format compatible with more programming languages?" problem. This is trying to solve the "how can I add new encodings to my file format without making everyone update their code?" problem. The former would rightly be solved with C bindings that anyone can link with if they want. The latter might not seem like a big deal, but it's been the main blocker advancing the parquet format. Most people end up not caring about new advanced encodings and just write parquet files with the most compatible feature set.
I would call it clever. I'm not sure I'd call it genius.
When I'm working with data I'm working in a specific set of languages. Usually one. Yeah, other people might be working in other languages, but no individual author really needs a language-agnostic way of accessing data beyond compile time. Add to that the likely runtime boundaries that may need to be crossed instead of e.g. inlined by the compiler because it's in-language and dealing with known offsets or tags (depends on the data format of course). To the other commenter's point, am I going to have to sandbox all data access code just to be sure it's not able to do something unexpected? There's a lot of complexity here. And the inherent risk is going to slow down the operation that should be the simplest and fastest: interpreting bytes.
>no individual author really needs a language-agnostic way of accessing data beyond compile time.
That's so untrue! People need language-agnostic ways to access data all the time, and people work with data accessing them from multiple languages all the time!
If I have parquet files I can load them in duckdb, in pandas and polars, process them with various independent tools, and loads of other things...
A big problem with parquet, which this aims to replace, is that it's hard to add new encodings because everyone wants to stay compatible with old readers. Embedding the decoders in the file as WASM solves this problem since in theory, old readers will be able to read new files by just using the provided WASM to decode a column whose format the reader doesn't recognize.
So this is really about making a file that is forwards compatible in a way that lets you push the standards more than existing formats.
That would be why it chose a VM that is explicitly designed for sandboxing rather than native executable code or similar, the risk can be minimized by reducing the surface area available to that executable code to almost nothing.
I mean I disable javascript embedded in pdf and feel like it would have been better to not have that feature. It would spare people from the invoice.pdf email attachment viruses because most people had assumed pdf isn't going to be as bad as an exe.
TrueType and OpenType fonts include code executed by a VM to even render them. This wasn't a viable source of attacks so far, due to the properly limited nature of the VMs.
Maybe I would pick the eBPF VM instead, with all its limiting and verifying mechanics.
> This security update resolves a publicly disclosed vulnerability in Microsoft Windows. The vulnerability could allow remote code execution if a user opens a specially crafted document or visits a malicious Web page that embeds TrueType font files.
> This security update is rated Critical for all supported releases of Microsoft Windows. For more information, see the subsection, Affected and Non-Affected Software, in this section.
> The security update addresses the vulnerability by modifying the way that a Windows kernel-mode driver handles TrueType font files. For more information about the vulnerability, see the Frequently Asked Questions (FAQ) subsection for the specific vulnerability entry under the next section, Vulnerability Information.
There are many documented, exploited-in-the-wild font-file attacks (one example in 1]). Apple is re-writing their font interpreter specifically to improve security. [2]
Sure, but when the standard says "read this file and execute the instructions you find at the beginning" that is more dangerous than "this is a file with data and your program needs to figure out how to read it".
Ok if you want to be pedantic, the standard says, "if you can't read this file, go to the offset and then execute the code you find" which isn't functionally different from what I said.
There’s a big difference in the expected use of a file. If the file is attacker provided, and the fallback path is being used, the attacker can embed whatever WASM payload they want into the file since the file will be “opened” by “execute this offset into the file”.
Compare that to JSON. The parser NEVER needs to execute arbitrary instructions. Parser might have bugs, but it avoids a whole class of issues.
Putting aside the WASM sandboxing (I’m not familiar enough with it to understand how sandboxing works) there’s a DoS vector at least. Even regexes have had many DoS issues, and I can’t imagine WASM being easier to sandbox for DoS risk.
Many can, even if they have JITs, e.g. Wasmtime. Failing that, it's not that hard to add bytecode instrumentation that will count instructions and terminate early. Some execution platforms that utilize Wasm just inject bytecode instrumentation into guest programs before sending them to the Wasm engine. It's relatively easy to do and not that much overhead.
I mean json might not be the best example since for a long time people would run json through a javascript engine to parse it but I can see your point.
gotcha, so the vulnerability will be in some common libraries that attackers force some wasm fallback path with custom wasm instructions that when executed does something nefarious.
.exe has bindings to OS ABI and system calls, WASM doesn’t have this by default, it’s up to the VM to provide whatever environment the WASM executable needs, ideally there should be no system calls, no stdio, just instructions on how to interpret the file format.
Double-clicking an ".exe" (or running it via a shell) is not the same as "bag of bytes", it's "send these bytes to an executable environment".
Doing `head foo.exe` is quite different than `run foo.exe`
If I encode executable instructions in "image.png" and then send them to an interpreter that runs those instructions, the file extension doesn't matter.
As appealing as this is, it will never gain traction without some backwards compatibility with Parquet and wide adoption of query engines to implement that backwards compatible path.
I don’t know what are people commenting on. I see a README with little to no information about what this is, what problems it solves, just links to its Flatbuffer description and a directory full of source code.
Some folks described it as genius. I guess it's my turn to play the role of an annoying HN skeptic: I find it somewhat silly. Data compression formats are secondary to what you're planning to do with the data once decoded. An audio file is completely different than an SVG image. An embedded VM that decompresses video to raw pixels doesn't magically let you play that video in a text editor, so there's no radically new kind of interoperability. Each new format still needs to be handled in a format-specific way.
I guess one use case is that I come up with a video compression scheme that's better than H.265, but not all platforms support it, so I embed a decoder that would allow me to play it back on legacy hardware. But that also shows the weakness of the idea: it's unlikely that legacy hardware will perform well doing software-only decode for video formats from the future. If we rolled this idea out in the 1990s, it would not have allowed watching Netflix on an i386.
In the same vein, I doubt this would have allowed me to open Word 2021 files in Word 97. There's no 1-to-1 mapping between the data structures. So if this kind of compat isn't slam-dunk, what's the goal?
The downsides are clear. First, it's probably a maintenance nightmare: if your decoder has a bug that needs fixing, how do you patch all the files that already embed it? And then, there's size overhead and security risks. We're adding a considerable attack surface to every format parser. It's more opportunities for remote code execution, resource exhaustion attacks, and so on. Again, this is not always wrong, but what's the benefit?
I don't think you have encountered the problems that this class of formats solves. Try looking up columnar storage formats, the pros and cons are pretty well defined these days. It is not meant for video decoding, indeed.
One nice thing about some modern formats is that there are tools that read them at extraordinarily high effective speed. For example, DuckDB can do all manner of nifty optimizations while reading its own native format or Parquet. And I’m not sure that those optimizations can be effectively applied to a format that needs a WASM blob to be run to understand it. By the time you run a non-SIMD or even a SIMD-optimized pass over app the data, if that pass doesn’t understand your query, you may have already lost.
I admit I only skimmed the beginning of the paper, and maybe the format is less general than it sounds.
I see many replies criticizing F3 as an operational data format, like Parquet. Of course it can't be made as fast in the general case, or as compatible to the existing infrastructure.
OTOH F3 would be easy to decode into almost any of today's accepted formats, and likely to any of tomorrow's data formats. That's where being self-describing and self-unpacking would be important.
Hm. I can kinda see it replacing self-extracting EXEs, but a lot of why you choose specific file formats is for specific features they offer - any self-describing system can fall into "there are too many competing features and nobody handles them all" exactly as easily as any other format.
Like, can this file be efficiently mmap'd? Maybe if it emulates tar internally, but you don't know until you run it. Can it be seeked to specific bytes to only decompress part? It only supports a pre-release version of ISO-36898533 seeking, and your file library dropped support for it 6 years ago. If I rewrite 1MB in the middle, can it only change those pages on disk (and maybe an index), or do I have to rewrite the whole thing? Well the wasm blob supports 97 different APIs for it (there are 35 copies of one with different names), so it's larger than the data (but nobody paid attention to that), so you have 19 options that you recognize, but your CPU's native WASM accelerator only handles two or three so you've still got to specialize your code heavily.
At least with "*.tar.gz" you have some idea of what's possible.
This reminds me of Alan Kay's OOPSLA 1997 presentation "The Computer Revolution Hasn’t Happened Yet" when he describes the Air Force / Burroughs 220 file format from 1961 where the file/tape contained both the data and the procedures to read/write/print them: https://youtu.be/oKg1hTOQXoY?t=1355
If I am archiving PBs of data for 10+ years, I don't want to rely on a WASM interpreter being available and performant in the future just to read a file. I want a dead-simple, heavily documented byte specification like Parquet.
Additionally, putting the decoding logic inside an WASM binary introduces an active execution layer into what should be a cold storage.
Nice! The world can always use a better data format.
I think you might get some traction if you post the advantages over parquet and other files directly on the readme, so that if someone goes to https://github.com/future-file-format/f3 the see why they should try it.
Mention the advantages and post metrics. Cherry pick the metrics! There's probably a good use case for this but, from the current readme, it's not clear who should use this and why.
My concern is, if decode fails I need to debug WASM added by some other party maybe containing random bugs. Maybe a library of standard decoders maintained and tested by the project could help, but then not sure if it kills the advantage of the flexibility it provides.
But Wasm has deterministic execution, so if decode fails for you, it should have failed for them. I.e. it's not a problem that your system has introduced; they should be able to reproduce the failure independent of any client.
It doesn't explain what the project does (a file format for what? Name dropping other things I haven't heard of isn't useful)
There are no examples. It links to a flatbuffer schema which is at least well commented, but is full of deep implementation details.
The point is that within 2-3 minutes I'm not convinced why I care and still don't know enough about what this is to even think back to if if I encounter a scenario in the future where it would be useful.
> designed with efficiency, interoperability, and extensibility in mind. It provides a data organization that rectifies the layout shortcomings of the last-generation formats like Parquet,
This is all marketing speak that says nothing.
> maintaining good interoperability and extensibility (a.k.a future-proof) via embedded Wasm decoders
What does this even mean? Providing a decoder is no guarantee of futureproofness.
To save a click it's a file format for columnar data specifically (like Parquet), which they very generically named Future-proof File Format. Most of this could fit in the title instead of just "F3" that is a bit of bait.
Has nimble/velox had any better luck lately? I forget what stories someone shared, but, it seemed to have such big intent, then real trouble actually getting released. I want to say someone was saying the lawyers ended up not letting a lot of the work get released. Nimble is the one competitor benchmarked against here that beats them, and is also extensible (to some degree?), so I'd love to know how things have gone for the past 6-12 months for nimble/velox.
https://news.ycombinator.com/item?id=39995112https://github.com/facebookincubator/nimble/https://materializedview.io/p/nimble-and-lance-parquet-kille...
The description mentions shortcomings of the previous file types like parquet, but it isn't really evident to me what those shortcomings are, or if the use cases for parquet and F3 have really that much of an overlap to make this comparison valid in the first place.
I love the idea, and I developed something similar of myself in the past (https://github.com/golddranks/kobuta), but... this reeks of slop. With Rust code, edition="2021" is a dead giveaway.
Not sure why this got so many upvotes, also the landing page is not great, its better to look at the paper (see link below).
Seems to be a columnar storage format that addresses some shortcomings in parquet. Thing is, though, that of all these formats the real winning feature is compatibility, which is (obviously) very hard to improve on, as anything new immediately loses.
Parquet is unfortunately very good just by virtue of being first, and so widely supported. The most widely used parquet version is the oldest version from 2013 (as per the paper itself), so parquet itself couldn't even supplant parquet. If you want to improve on it, you need to bring some serious results, which I don't think f3 does.
Also, my main gripe with parquet (single table per file) is not even addressed, so, also the name is a bit hyped up.
Also also, it seems to go out of its own way to include a compiled wasm binary for decoding, yet requires flatbuffers to parse that blob? Kind of defeats the purpose.
Its main result seems to be improved random access which, although certainly welcome, is not the point of columnar storage, as columnar storage was invented to exchange random access for something else: fast analytics. F3 seems to sacrifice fast analytics for the wasm decoder. I don't get it.
Maybe I'm being too cynical. Can someone help me out here?
https://dl.acm.org/doi/epdf/10.1145/3749163
This bit is quite genius, rather than depend on a language-specific SDK/lib for working with the formats you can fallback to exported WASM methods if none exist:
So attackers don't have to craft specially corrupted files? They can just include the code to perform the attack in the data file itself?
WASM has strong tried and proven sandboxing. We basically can build on nearly 30 years of experience. The decoders don't need a lot of access, they can basically be pure functions.
If this will pan out security-wise I don't know. I'm more worried that it will be so slow that no one will use it. Interesting idea, though, and I can see applications outside of the "big data" realm this apparently targets.
How do you prevent compression bomb attacks when files can define their own compression functions?
You could have some kind of OOM killer, but that will be a "footgun" that people who are actually doing "big data" will constantly shoot.
This pretty much kills any ingestion pipeline where the source is untrusted.
It seems like the WASM is simply a fallback if no other decoder is available. If the data source is untrusted, simply don’t run the WASM decoders.
“Some code is untrusted” does not mean code should never be executed. There are more use cases with trusted sources than untrusted.
So I define the data type to be "asdklfjaslkdfjiolsadfjoiusadfoiasfoikasjfdoisadf" and give you a decoder for it.
OOM killing in WebAssembly is trivial, since it’s all in a growable linear memory. All the runtimes I’m aware of have a simple maximum memory setting, and they’ll trap any allocation requests after that point.
Attack is not just on file format itself. Based on the function signature it's possible for a single decoder to generate infinite bytestream - makes a lot of headache to reader implementation - implementing STRLEN is no longer trivial question.
Either engines should put some limit (e.g. VARCHAR(2000) to enforce length to be limited to 2000, but there are some other engines supporting unlimited BLOBs), or decoder should give a hint what is the maximum length it will yield. Unfortunately current research level project does not have such considerations implemented yet...
For images, it makes sense: people dealing with 16k x 16k PNGs are uncommon. Give them an error message that tells them the setting to bump. But what should be the threshold for "big data"? I'm sure it will follow Zipf's Law, but the tail will be fatter.
And many of them have built-in gas metering, so you can time out the decode if it runs too many instructions.
Denial-of-service is bad, but it's not in the same ballpark, the same sport, the same planet, or the same universe of bad as RCE.
WASM implementations are fairly mature now, but if there was e.g. an image file format with embedded WASM that needed to execute before you could view it, it would become the new low-hanging-fruit target for 0-click RCEs - whether it's exploiting the WASM engine itself or some other attack surface that's influenceable via it (See also, the FORCEDENTRY JBIG2 exploit).
That exploit targeted an integer overflow in a bespoke Apple sandboxing mechanism. Bespoke sandboxing mechanisms have weird bugs.
Not that Wasm engines don't have bugs, but the whole point is to have an extremely solid, well-specified and efficient implementation of a widely accepted bytecode format. We can scope down the capabilities given to any program to a minimal set.
Bugs are near-inevitable, and mitigations are the last line of defence. Scripting engines are excellent for bypassing mitigations (iiuc in the case of the FORCEDENTRY exploit, it was used for adjusting ASLR'd offsets).
As a random example that's an area of personal interest to me, I know of 3 distinct methods of achieving userland ROP execution of the Nintendo Switch 2, and all three rely on the (ab)use of a scripting engine (even if they aren't a vulnerability in the scripting engine itself).
Well don't accept code from anyone ever then.
But seriously, if your format requires extensibility to the point that it embeds a bytecode, especially a Turing-complete bytecode, what format are you going to choose? Just design a new one? That's how you end up with a scripting engine with three ROP exploits.
> WASM has strong tried and proven sandboxing. We basically can build on nearly 30 years of experience. The decoders don't need a lot of access, they can basically be pure functions.
I've heard that kind of sentiment many times before. It's not a good (thought-terminating) mindset to have for any secure software.
There are several WASM implementations, WASM is just a format. "Pure functions" are pure at a superficial level. Many people say that they don't mutate global state, but they do ... it's just hidden. The decoders "not needing a lot of access" doesn't matter if the WASM engine is pwned through arbitrary code execution inside the environment, or if it's contorted to bypass the access control you are mentioning through various side-effects.
> The decoders don't need a lot of access, they can basically be pure functions
They don't currently either do they? It's the tight coupling of the interface layer no? I'm not sure this would be faster, or more secure so reliability might be the best usecase?
Yes...my first thought. No way in hell anyone actually trusts this.
(And as if we didn't trust the compiler enough already!)
Meh, it's not that bad. Pretty simple to block inline wasm and to use well known external decoders.
Does WASM have built-in I/O? If not, all that a decoder would be able to do is to decode into a buffer.
All WASM can do is transfer bag of bytes between module runtime and host. So yes, so yeah it can just decode into a buffer. Even you use wasm components to give it I/O, you can still make these go to buffer.
But the WASM runs in the sandbox! It only has access to some files, your display, inputs, ... nothing insecure at all!
WASM runs in a confined memory space allocated for the program. There is no I/O or host address space access.
You need to run a WASI environment for that.
I don't understand how that's supposed to work. What does the decoder decode into? That's gonna depend entirely on the kind of data, right? For some formats, it's gonna be a stream of bytes; for others, a 2D plane of pixels; others again will need vertexes, 2D planes of pixels and UV maps; for some, an object graph will make more sense.
It appears as though the WASM decode returns two values -- one indicating the data type as a primitive value, and a second value being the data buffer
Then there is a helper in this case to de-serialize, "primitive_array_from_buffers()"
https://github.com/future-file-format/F3/blob/bd92506447dc13...
It sounds neat, but feels like it might fall apart with higher-complexity formats. What does an embedded decoder for a PDF look like? I guess since they are tightly-coupled to the file bytes themselves, the author of the file gets to choose what formats make sense, but not all formats have a one-true-decode-step.
Despite the name seemingly implying otherwise, F3 is an alternative to columnar storage formats like Parquet; the goal is not to support every conceivable encoding of every file type such as a PDF. Think of the use cases being more like "What if you used a specialized compressor and need a custom block decompression algorithm" or "Decode internal format into Arrow output" or something like that.
Applets redux.
How is wasm better than C bindings?
Many languages don't have ergonomic experiences for working with C ABI's without explicit wrapper code.
Hell, Node.js didn't even get this ability until LAST MONTH:
https://nodejs.org/en/blog/release/v26.1.0
You'd have to write a second library to interface the C ABI with Node via NAPI just to consume it.
WASM is platform independent.
What do you mean by C bindings? C bindings to what?
C bindings to a C implementation
This isn't using WASM to solve the "how can I make my file format compatible with more programming languages?" problem. This is trying to solve the "how can I add new encodings to my file format without making everyone update their code?" problem. The former would rightly be solved with C bindings that anyone can link with if they want. The latter might not seem like a big deal, but it's been the main blocker advancing the parquet format. Most people end up not caring about new advanced encodings and just write parquet files with the most compatible feature set.
C bindings are not platform independent, nor do they come with a runtime and a sandbox, among other things. Apples to oranges.
I would call it clever. I'm not sure I'd call it genius.
When I'm working with data I'm working in a specific set of languages. Usually one. Yeah, other people might be working in other languages, but no individual author really needs a language-agnostic way of accessing data beyond compile time. Add to that the likely runtime boundaries that may need to be crossed instead of e.g. inlined by the compiler because it's in-language and dealing with known offsets or tags (depends on the data format of course). To the other commenter's point, am I going to have to sandbox all data access code just to be sure it's not able to do something unexpected? There's a lot of complexity here. And the inherent risk is going to slow down the operation that should be the simplest and fastest: interpreting bytes.
>no individual author really needs a language-agnostic way of accessing data beyond compile time.
That's so untrue! People need language-agnostic ways to access data all the time, and people work with data accessing them from multiple languages all the time!
If I have parquet files I can load them in duckdb, in pandas and polars, process them with various independent tools, and loads of other things...
A big problem with parquet, which this aims to replace, is that it's hard to add new encodings because everyone wants to stay compatible with old readers. Embedding the decoders in the file as WASM solves this problem since in theory, old readers will be able to read new files by just using the provided WASM to decode a column whose format the reader doesn't recognize.
So this is really about making a file that is forwards compatible in a way that lets you push the standards more than existing formats.
except you need flatbuffers to access that blob
Is embedding executable code into a file a security risk? My assumption is a yes
That would be why it chose a VM that is explicitly designed for sandboxing rather than native executable code or similar, the risk can be minimized by reducing the surface area available to that executable code to almost nothing.
> Is embedding executable code into a file a security risk?
Yes, which is why nobody uses PDFs.
I mean I disable javascript embedded in pdf and feel like it would have been better to not have that feature. It would spare people from the invoice.pdf email attachment viruses because most people had assumed pdf isn't going to be as bad as an exe.
which is why no sane pdf viewer implements executable features*
TrueType and OpenType fonts include code executed by a VM to even render them. This wasn't a viable source of attacks so far, due to the properly limited nature of the VMs.
Maybe I would pick the eBPF VM instead, with all its limiting and verifying mechanics.
https://learn.microsoft.com/en-us/security-updates/SecurityB...
> This security update resolves a publicly disclosed vulnerability in Microsoft Windows. The vulnerability could allow remote code execution if a user opens a specially crafted document or visits a malicious Web page that embeds TrueType font files.
> This security update is rated Critical for all supported releases of Microsoft Windows. For more information, see the subsection, Affected and Non-Affected Software, in this section.
> The security update addresses the vulnerability by modifying the way that a Windows kernel-mode driver handles TrueType font files. For more information about the vulnerability, see the Frequently Asked Questions (FAQ) subsection for the specific vulnerability entry under the next section, Vulnerability Information.
There are many documented, exploited-in-the-wild font-file attacks (one example in 1]). Apple is re-writing their font interpreter specifically to improve security. [2]
[1] https://www.bleepingcomputer.com/news/security/facebook-disc...
[2] https://blakecrosley.com/blog/truetype-hinting-swift-migrati...
There is no concept of "executable" vs "non-executable" content in a file.
A file is a bag of bytes. You can send those bytes to different things, like a text editor's content-stream, or as the input to a WASM interpreter.
What you decide to do with the bytes in a file is your own prerogative. Each byte is whatever you make of it.
Sure, but when the standard says "read this file and execute the instructions you find at the beginning" that is more dangerous than "this is a file with data and your program needs to figure out how to read it".
I guess it's a good thing that the F3 standard does not say "read this file and execute the instructions you find at the beginning", then?
The WASM encoders/decoders are embedded resources that exist as byte offsets in the file metadata, not header info.
Ok if you want to be pedantic, the standard says, "if you can't read this file, go to the offset and then execute the code you find" which isn't functionally different from what I said.
There’s a big difference in the expected use of a file. If the file is attacker provided, and the fallback path is being used, the attacker can embed whatever WASM payload they want into the file since the file will be “opened” by “execute this offset into the file”.
Compare that to JSON. The parser NEVER needs to execute arbitrary instructions. Parser might have bugs, but it avoids a whole class of issues.
WASM physically cannot interact with the underlying host or perform I/O -- you need a WASI environment for that.
Putting aside the WASM sandboxing (I’m not familiar enough with it to understand how sandboxing works) there’s a DoS vector at least. Even regexes have had many DoS issues, and I can’t imagine WASM being easier to sandbox for DoS risk.
There exist Wasm interpreters capable of limiting the number of instructions executed.
Many can, even if they have JITs, e.g. Wasmtime. Failing that, it's not that hard to add bytecode instrumentation that will count instructions and terminate early. Some execution platforms that utilize Wasm just inject bytecode instrumentation into guest programs before sending them to the Wasm engine. It's relatively easy to do and not that much overhead.
I mean json might not be the best example since for a long time people would run json through a javascript engine to parse it but I can see your point.
gotcha, so the vulnerability will be in some common libraries that attackers force some wasm fallback path with custom wasm instructions that when executed does something nefarious.
I'd say at worst it's setup for poor security
I mean can't we say the same thing about sending around a .exe though?
.exe has bindings to OS ABI and system calls, WASM doesn’t have this by default, it’s up to the VM to provide whatever environment the WASM executable needs, ideally there should be no system calls, no stdio, just instructions on how to interpret the file format.
Double-clicking an ".exe" (or running it via a shell) is not the same as "bag of bytes", it's "send these bytes to an executable environment".
Doing `head foo.exe` is quite different than `run foo.exe`
If I encode executable instructions in "image.png" and then send them to an interpreter that runs those instructions, the file extension doesn't matter.
exactly
As appealing as this is, it will never gain traction without some backwards compatibility with Parquet and wide adoption of query engines to implement that backwards compatible path.
I don’t know what are people commenting on. I see a README with little to no information about what this is, what problems it solves, just links to its Flatbuffer description and a directory full of source code.
What context am I missing?
There is a linked paper: https://dl.acm.org/doi/epdf/10.1145/3749163
Some folks described it as genius. I guess it's my turn to play the role of an annoying HN skeptic: I find it somewhat silly. Data compression formats are secondary to what you're planning to do with the data once decoded. An audio file is completely different than an SVG image. An embedded VM that decompresses video to raw pixels doesn't magically let you play that video in a text editor, so there's no radically new kind of interoperability. Each new format still needs to be handled in a format-specific way.
I guess one use case is that I come up with a video compression scheme that's better than H.265, but not all platforms support it, so I embed a decoder that would allow me to play it back on legacy hardware. But that also shows the weakness of the idea: it's unlikely that legacy hardware will perform well doing software-only decode for video formats from the future. If we rolled this idea out in the 1990s, it would not have allowed watching Netflix on an i386.
In the same vein, I doubt this would have allowed me to open Word 2021 files in Word 97. There's no 1-to-1 mapping between the data structures. So if this kind of compat isn't slam-dunk, what's the goal?
The downsides are clear. First, it's probably a maintenance nightmare: if your decoder has a bug that needs fixing, how do you patch all the files that already embed it? And then, there's size overhead and security risks. We're adding a considerable attack surface to every format parser. It's more opportunities for remote code execution, resource exhaustion attacks, and so on. Again, this is not always wrong, but what's the benefit?
I don't think you have encountered the problems that this class of formats solves. Try looking up columnar storage formats, the pros and cons are pretty well defined these days. It is not meant for video decoding, indeed.
One nice thing about some modern formats is that there are tools that read them at extraordinarily high effective speed. For example, DuckDB can do all manner of nifty optimizations while reading its own native format or Parquet. And I’m not sure that those optimizations can be effectively applied to a format that needs a WASM blob to be run to understand it. By the time you run a non-SIMD or even a SIMD-optimized pass over app the data, if that pass doesn’t understand your query, you may have already lost.
I admit I only skimmed the beginning of the paper, and maybe the format is less general than it sounds.
My understanding is it’s a fallback mechanism
F3 seems to be a reasonable archival data format.
I see many replies criticizing F3 as an operational data format, like Parquet. Of course it can't be made as fast in the general case, or as compatible to the existing infrastructure.
OTOH F3 would be easy to decode into almost any of today's accepted formats, and likely to any of tomorrow's data formats. That's where being self-describing and self-unpacking would be important.
This could use a bit more "why".
Shortcomings of Parquet are mentioned as overcome by this, which ones? Certainly not wide tool support...
Why should one leave Parquet or ORC for this structure?
I also had no idea what they were talking about, but there's good points about how hardware oblivious and somewhat global is Parquet around metadata.
I found this post interesting, - https://medium.com/@reliabledataengineering/f3-the-future-pr...
The ‘why’ is referenced in the bibliography at the end of the readme; this repo is not meant to be consumed standalone. Start with the paper instead:
https://doi.org/10.1145/3749163
Yeah it seems like most of this can be handled by some more dev hours to Parquet
Paper mentions Parquet, ORC, Nimble, Lance, TSFile, Bullion, and BtrBlocks.
Hm. I can kinda see it replacing self-extracting EXEs, but a lot of why you choose specific file formats is for specific features they offer - any self-describing system can fall into "there are too many competing features and nobody handles them all" exactly as easily as any other format.
Like, can this file be efficiently mmap'd? Maybe if it emulates tar internally, but you don't know until you run it. Can it be seeked to specific bytes to only decompress part? It only supports a pre-release version of ISO-36898533 seeking, and your file library dropped support for it 6 years ago. If I rewrite 1MB in the middle, can it only change those pages on disk (and maybe an index), or do I have to rewrite the whole thing? Well the wasm blob supports 97 different APIs for it (there are 35 copies of one with different names), so it's larger than the data (but nobody paid attention to that), so you have 19 options that you recognize, but your CPU's native WASM accelerator only handles two or three so you've still got to specialize your code heavily.
At least with "*.tar.gz" you have some idea of what's possible.
This reminds me of Alan Kay's OOPSLA 1997 presentation "The Computer Revolution Hasn’t Happened Yet" when he describes the Air Force / Burroughs 220 file format from 1961 where the file/tape contained both the data and the procedures to read/write/print them: https://youtu.be/oKg1hTOQXoY?t=1355
If I am archiving PBs of data for 10+ years, I don't want to rely on a WASM interpreter being available and performant in the future just to read a file. I want a dead-simple, heavily documented byte specification like Parquet.
Additionally, putting the decoding logic inside an WASM binary introduces an active execution layer into what should be a cold storage.
You don't want to run a custom 10-year-old data parsing function every time you read a single data record?
Nice! The world can always use a better data format.
I think you might get some traction if you post the advantages over parquet and other files directly on the readme, so that if someone goes to https://github.com/future-file-format/f3 the see why they should try it.
Mention the advantages and post metrics. Cherry pick the metrics! There's probably a good use case for this but, from the current readme, it's not clear who should use this and why.
My concern is, if decode fails I need to debug WASM added by some other party maybe containing random bugs. Maybe a library of standard decoders maintained and tested by the project could help, but then not sure if it kills the advantage of the flexibility it provides.
But Wasm has deterministic execution, so if decode fails for you, it should have failed for them. I.e. it's not a problem that your system has introduced; they should be able to reproduce the failure independent of any client.
File format for what? Text, graphics, compiled code?
For columnar data storage I think. The description references Parquet and they appear to benchmark against Parquet, Vortex, and Lance.
The future.
I was afraid that it is for Marty McFlys only.
Not quite vaporware, but few commits, PRs, history, actual examples etc. It's pretty thin.
A Wasm decoder takes encoded bytes and returns an iterator of Arrow Buffers. In case you were wondering.
Great! I'll use it.
In the "future."
Nimble? Lance? Also in the future. Maybe.
I'll use Parquet in the present.
Probably not a good idea to name your project “future” anything, if you expect that future to become the present.
Also, f3 is already “fight-flash-fraud”.
This project README is not particularly useful:
It doesn't explain what the project does (a file format for what? Name dropping other things I haven't heard of isn't useful)
There are no examples. It links to a flatbuffer schema which is at least well commented, but is full of deep implementation details.
The point is that within 2-3 minutes I'm not convinced why I care and still don't know enough about what this is to even think back to if if I encounter a scenario in the future where it would be useful.
> designed with efficiency, interoperability, and extensibility in mind. It provides a data organization that rectifies the layout shortcomings of the last-generation formats like Parquet,
This is all marketing speak that says nothing.
> maintaining good interoperability and extensibility (a.k.a future-proof) via embedded Wasm decoders What does this even mean? Providing a decoder is no guarantee of futureproofness.
Tabular data, it wants to replace Parquet
Don't know why but I had to think of https://xkcd.com/2116/
No commits in 8 months?
Yeah this was a research project, it doesn't look like this is getting any adoption
>via embedded Wasm decoders
runs screaming
To save a click it's a file format for columnar data specifically (like Parquet), which they very generically named Future-proof File Format. Most of this could fit in the title instead of just "F3" that is a bit of bait.
The wasm decoder thing was also done in Anyblox. https://github.com/AnyBlox https://gienieczko.com/anyblox-paper
Has nimble/velox had any better luck lately? I forget what stories someone shared, but, it seemed to have such big intent, then real trouble actually getting released. I want to say someone was saying the lawyers ended up not letting a lot of the work get released. Nimble is the one competitor benchmarked against here that beats them, and is also extensible (to some degree?), so I'd love to know how things have gone for the past 6-12 months for nimble/velox. https://news.ycombinator.com/item?id=39995112 https://github.com/facebookincubator/nimble/ https://materializedview.io/p/nimble-and-lance-parquet-kille...
The description mentions shortcomings of the previous file types like parquet, but it isn't really evident to me what those shortcomings are, or if the use cases for parquet and F3 have really that much of an overlap to make this comparison valid in the first place.
I love the idea, and I developed something similar of myself in the past (https://github.com/golddranks/kobuta), but... this reeks of slop. With Rust code, edition="2021" is a dead giveaway.
Is what?
https://xkcd.com/927/
A more descriptive title would be helpful OP:
F3: Open-source data file format for the future
Previous discussion:
2025 https://news.ycombinator.com/item?id=45437759