I saw Node-RED recently, and now this. While I think these are cool (and I would rather Erlang than NodeJS), is there some code based system for these kinds of things?
I know I can just write code, but I mean a DSL or some library specifically oriented around solving exactly these kinds of event driven sequencing/action problems.
Not to my knowledge, the closest thing to a DSL for Node-RED is its flows.json format for storing flow configurations.
It's a very simple array-of-object format with the graph/flow definitions defined by node ids and wires. It is fairly simple to create a minimalist interpretor for that format.
Also NodeRED is a very stable piece of software that has been around for over ten years. Yet I can take a flows.json file from today and have it run on the first releases of Node-RED, i.e., the format is backward compatible yet easily extendable.
> event driven sequencing/action problems.
Flow based programming[1] isn't strictly event driven, it's implemented that way in NodeJS because NodeJS does not have the concept of messages, only events.
Erlang has message passing between independent processes and therefore, IMHO, better suited to FBP.
Firstly, thanks for the link and information on more of the roots of this kind of programming, I couldn't find this when I looked.
Regarding NodeRED. I don't dispute that it's stable, but my experience actually running any nodejs based software is that it's all heavily designed for running out of a container and a massive PITA to maintain otherwise. I already have enough hassle running zigbee2mqtt on my FreeBSD home server which wouldn't work in a linux jail because I couldn't figure out how to expose the zigbee dongle to it.
Secondly, the reason I wanted to run one of these was not because I wanted to use it to automate my house, but instead because I wanted to understand how it works so I could replicate it in my own weird way.
I suggest one of Benthos' descendants, either Bento or RedPanda Connect.
Configuration driven, with a transformation DSL, and good documentation (I prefer the Bento documentation over RedPanda Connect's documentation). And Benthos explicitly refused to ack a consumed message upstream unless it had been acknowledged downstream, so it wouldn't drop messages on the floor unless you explicitly told it to.
> Node-RED is a amazing[*] tool for creating flows that describe concurrent processing, it is just a shame the NodeJS is single threaded. So why not use something that is multi-process from the ground up?
That's great! But now that we are doing this, it kind of makes me wish that it was not multi-processing and Erlang, but a more mainstream language with a better library ecosystem and focused on multi-threading instead, Rust comes to mind but there could be a better one.
Is there a visual flow programming language based on Rust that is relatively mature?
> wish that it was not multi-processing and Erlang
Hehe - honestly I chose Erlang because I love the syntax and secondly because it is so niche. Flow based programming is niche, visual FBP is nicher, Node-RED is even nicher and Erlang is niche - it's niche all the way down ;)
Have a look at Elixir which is Erlang with Ruby syntax and also has a large community.
Also the intention is to be using the flow editor and not the lanuage underneath, so the end user won't even know that it's Erlang or NodeJS - I aim t stay compatible with Node-RED so that flows are actually interchangeable.
So to an certain extent it doesn't matter which language is used.
I did chose Erlang also because it is well suited to flow based programming being message based and flow based programming is all about passing immutable objects amongst independent processes .... that's Erlang in a nutshell.
> makes me wish that it was not multi-processing and Erlang, but a more mainstream language with a better library ecosystem and focused on multi-threading instead, Rust comes to mind but there could be a better one.
I am not quite following, why would we drop multi-processing with isolated tiny heaps with a few KBs each and move to multi-threading with a large multi-megabyte stacks per thread and all sharing and writing to the same heap? It seems like a step backward.
> Rust comes to mind but there could be a better one.
I can see the excitement, but at the same time this is becoming a meme. I mean the drive-by comments of "why don't your rewrite it in Rust" on every repo or "show HN" post. It's kind of like nodejs from years past: "why don't your write in nodejs, it's webscale!". The problem is, after a while it starts to have an inverse dose response: it makes people run away from Rust just based on that kind of stuff.
>I am not quite following, why would we drop multi-processing with isolated tiny heaps with a few KBs each and move to multi-threading with a large multi-megabyte stacks per thread and all sharing and writing to the same heap? It seems like a step backward.
Performance typically, hard to work in parallel on large amount of data performantly without multiple thread sharing a heap and you typically don't need large amount of threads because you don't actually have that many real cores to run them on.
Lots of little share nothing process are great conceptually but that does create significant overhead.
> Lots of little share nothing process are great conceptually but that does create significant overhead.
It doesn't, really? I have clusters running 1M+ Erlang processes comfortably per node.
> you typically don't need large amount of threads
Exactly, that's why Erlang only spawns just the right amount of threads. One scheduler thread per CPU, then a bunch of long running CPU task threads (same number as CPUs as well), plus some to do IO (10-20) and that's it.
Erlang is not as performant for heavy computational loads, this is bared out in many benchmarks, thats not what it's good at.
Message passing share nothing adds overhead when trying to reference data between processes because it must be copied, how would you do multithreaded processing of a large amount of data without a shared heap in a performant way? Only one thread can work on a heap at a time, so what do you do? Chop up the data and copy it around then piece it back together afterward? Thats overhead vs just working on a single heap in a lockless way. Far as I can tell the main Erlang image processing libraries just call out to C libraries that says something of that kind of work.
Yes Erlang indirects computation to a OS thread pool, multiplexing all those little Erlang process on real threads creates scheduling overhead. Those threads cannot work on the same data at the same time unless they call out to a libraries written in another language like C to do the heavy lifting.
.Net does similar things for say web server implementations, it uses a thread pool to execute many concurrent requests and if you use async it can yield those threads back to the pool while say waiting on a DB call to complete, you would not create a thread per http connection so the 4mb stack size is not an issue just like its not with Erlangs thread pool.
> Erlang is not as performant for heavy computational loads
Well, sure don't use it for heavy computational loads if it doesn't work for you. It works great for our use case and I think Node-RED is also not for heavy computational workload, but rather for event programming, where lots of events happen concurrently.
> how would you do multithreaded processing of a large amount of data without a shared heap in a performant way?
That's a specific workload, it's not universal, for sure. I haven't worked on cases there is a single large data structure and all million clients have to manipulate it concurrently. Not saying it's implausible, I can see a game server with an arena maybe being like that, but there are other systems that work like that.
>Well, sure don't use it for heavy computational loads if it doesn't work for you.
>why would we drop multi-processing with isolated tiny heaps with a few KBs each and move to multi-threading with a large multi-megabyte stacks per thread and all sharing and writing to the same heap?
This is the question I was answering and I gave you an example then you proceeded to say there is no overhead Erlangs process model, which now you admit exists.
>That's a specific workload, it's not universal, for sure. I haven't worked on cases there is a single large data structure and all million clients have to manipulate it concurrently.
Thats not the workload, simple as a single client doing heavy processing of large image or video, multithreaded shared heap wins due the the overhead of Erlang's processing model, thats why you give it up which is answering your original question again.
Geez this is tough crowd, first Erlang has no overhead, then its well that nots whats its good at, which I said. Then its well those are specific workloads, yeah serving little messages to a million clients is a specific workload too. I like Erlang, it's very good at specific things but it sacrifices quite a bit of performance to do that like many good abstractions. That can help overall reliability and performance on certain workloads but other modern languages following the shared heap multi-threading model will probably outperform it if care is taken in the design because they more closely match the underlying OS and hardware execution model of today with less indirection and overhead and they can also do other workloads that Erlang will never be good at with its share nothing green thread/process design.
> This is the question I was answering and I gave you an example then you proceeded to say there is no overhead Erlangs process model, which now you admit exists
But that's not what NodeRed-Erlang does. The author is confirming it one one of the sibling comments. You invented some new type of workload, were all clients have to access the same heap concurrently and telling the author to rewrite everything in Rust. But that's your use-case, not the the one we're discussion. My use case is not that either.
> Geez this is tough crowd, first Erlang has no overhead, then its well that nots whats its good at, which I said. Then its well those are specific workloads, yeah serving little messages to a million clients is a specific workload too.
The application in this case fits Erlang perfectly. You're the one who introduced a new type of a use case - when you need to concurrently process the same video or audio and arguing that Erlang is not good for it. Yeah, we agree, it's not good for that, use something else, Rust, C etc. It's like we're talking about trains and you're saying, don't use trains, they can't fly, what if you have to cross the ocean or something like.
> You invented some new type of workload, were all clients have to access the same heap concurrently and telling the author to rewrite everything in Rust. But that's your use-case, not the the one we're discussion. My use case is not that either.
Image processing or video or audio or parsing or working with large amount of data in a multithreaded way is not a new kind of workload its a very common one.
It was stated by the OP that they wished it would have been done in a more standard multithreaded language like rust. You asked why, I gave you an example, you said it wasn't true that there is no overhead to Erlang, I gave further details, you then pivoted to thats not the use case. How do you know what oersted's use case is?
>It's like we're talking about trains and you're saying, don't use trains, they can't fly, what if you have to cross the ocean or something like.
It's more like you asked when would you not use a train? I answered when you need to go across an ocean and planes can do both what trains and boats do just at higher cost but at greater speed.
Erlang is great at specific things, standard multithreaded shared heap runtimes are good at everything they just might take more effort and care to do the things Erlang can do, but they will perform better than Erlang. Erlang can technically do anything since its Turing complete but it will perform unacceptably at many standard workloads just like a train could go across an ocean if put on a barge and towed.
> I think Node-RED is also not for heavy computational workload
That is correct, Node-RED is designed for routing many thoughts of smallish data packets through some computational units. Each computational unit is basically a node with a message queue and each message is handled and then passed on.
This means that if a node takes too long, its queue will potentially overflow causing failure. However if there are fewer messages than nodes can take longer - there is nothing ingrained in Node-RED that prevents having long running processes and fewer messages.
As long as message queues don't overflow or messages get too large causing massive memory usage, then Node-RED will happily just work.
NodeREDs many use cases are controlling many small devices (home automation) and (I)IoT - collecting datapoints from 100s or 1000s of devices and routing that around to whereever the data is needed.
Nobody choses erlang for heavy computational loads, but for ease of use and a simple mental model. You do heavy compute on it mostly if you don't want to be bothered. Erlang's multi-process thing, it's not to be efficient at compute, it's to be efficient at crash recovery and distributed compute. It was there before multi-core multithread were a thing.
> it kind of makes me wish that it was not multi-processing and Erlang, but a more mainstream language with a better library ecosystem and focused on multi-threading instead, Rust comes to mind but there could be a better one
There are 3 or 4 Go FBP packages. It seems to be completely capable, but FBP itself doesn't seem to have reached enough critical mass for them to be popular.
.Net uses real OS threads and that 4mb is the stack not the heap which must be allocated on thread creation. The heap is shared in the process amongst all the threads whereas Erlang is separate lightweight green thread/processes with dynamic heap and stack per process.
.Net originates in the Windows world where new processes are expensive but threads are much cheaper however they still use native stacks for iterop and hardware protection, stack must be contiguous so it is fixed at thread creation this is typical for all native os threads.
Erlang is a VM and runs on top of of processes and threads but exposes its own process model to the language that does not map 1:1 and therefore does not have a fixed stack size allowing many light weight processes. It pays an indirection penalty in performance for this but allows some nice stuff.
Java started with green threads and abandoned them, both .Net and Java use asynchronous concepts instead to get high concurrency now days cooperatively releasing threads and rentering them. There was talk of trying green threads out in .Net and bringing them back in Java for ergonomics compared to async, these green threads would not have such large fixed stacks and many more could be made similar to Erlang processes.
In the end, .NET is going hybrid route. Circa .NET 10/11 Roslyn will stop emitting state machines and something state-machine-like will be produced by the compiler itself iff the asynchronous execution suspends, and will be otherwise practically free.
Mind you, short-lived operations are much cheaper with stackless co-routine model like in C#/F# or Rust than spawning an entire, even if virtual, thread/process.
I get why they went the async route due to performance implications and such, just like the old cooperative multi-tasking days everything was much lighter weight but one misbehaved process not yielding can block the whole OS.
Nice to see further improvement coming along in .Net however I conceptually prefer the everything is a lightweight process model to this function coloring async virus that invades the codebase now days, wouldn't be so bad if it had been easy to do sync over async from the beginning without caveat in .Net so you didn't need to have dual sync and async api all the way down the stack.
Usually dual APIs are offered when they do distinctly different things. Async calls on Socket go through epoll/kqueue via SocketAsyncEngine, which has observably different behavior to regular sockets. There is no such duality in high-level APIs like HttpClient's `.GetStringAsync`.
You are not free from having to worry about it with virtual threads either. If you block a carrier thread with such a low-level API call, the runtime will have a bad time. Or you will always end up paying for a context switch (if you maintain a separate pool of block-able threads).
Green threading UX is better for "regular" code that never bothers to take advantage of concurrency features, but if you do want them, you are quickly back to usually more ceremonious structured concurrency abstractions which, lo and behold, can reimplement the same task system with a worker pool but on top of virtual threads.
If you start to care about cancellation, suddenly you are back to coloring problem regardless. I agree with you that having to write many await's in quick succession is tiresome and unnecessary. It's partially a fault of library authors who name methods "GetSomeLongNameAsync" instead of "Get" or "Query". But other than this it's not as invasive as people make it out to be, and encourages writing highly concurrent code - reducing latency 2x can require moving one or two awaits around. Not so much in Java/Erlang/Go.
I haven't found this is the case many times, many api will have two versions of the api with XxxAsync versions all the way down, maybe less so now days but it took a while to get async entry points added everywhere so you could use it properly, such as ActionResult.ExecuteResultAsync and IAsyncEnumerable as a couple examples.
I suppose the difference of opinion comes down to whether you consider hiding side effects a good or a bad thing. And whether you want to have easy composition and cancellation (Go struggles with passing context around a lot, in .NET it's much less of an issue if at all, depending on codebase).
If you don't want to join on some other virtual thread or goroutine completing immediately (and with Goroutines it's additionally painful since you need to communicate completion separately), it's more difficult and costlier than doing so with tasks/futures.
CancellationTokens can be passed in synchronous functions just fine, I have done cancellable sync apis before cancellation tokens existed usually its a .Cancel() on the thing doing the work just like Command.Cancel() in the Sql client or Connection.Cancel() in a network client. I agree CancellationTokens are better and always use them now and thats not the problem with async.
The problem is the need to color functions with async and if you have existing client code that call existing synchronous methods it not easy to move their internal implementations to async form so that you can remain backward compatible without copy/pasting the same layers of code with different function coloring to async.
I was just thinking of making something like this last night but using FLENG instead of Erlang ;) I'm really glad more people are taking inspiration from Node-RED, I will definitely be playing with this!
FLENG[1] - Wow, "low level concurrent logic programming language descended from Prolog" - sounds like an easy swap out for Erlang since they have the same heritage. ;)
I know of Py-Red[1] which is an attempt to do the same but with Python. Using Node-RED as a frontend and something else as a backend, no I haven't heard of anything similar.
Main problems with visual flow based programming is the tooling: there aren't any good tools for doing visual comparison nor visual version control. Things like GitHub aren't able to render flow code as seen in the editor.
For Node-RED that means that the Json that defines a flow can be compared but that soon becomes meaningless because semantic changes get mixed with visual changes, i.e., that the x,y coordinates of a node have changed - is meaningless for the logic of the code.
This means that code sharing and collaborative development is difficult purely visually but only because tooling is missing. Compared to textual programming before things like SourceForge or Git and we had the same issues as we do with visual coding now.
Node-RED has a bunch of features to make large programs maintainable. Things like link nodes that allow jumping between flows also allow code reuse. There also subflows which encapsulate repeated code and can be referenced from anywhere.
That's also why I chose Node-RED for this because it seems to be the most mature and best maintained low-code visual flow-based programming tool. There are others such as n8n (which seems to be the other biggie) but they lack the generalness of Node-RED. I can create web sites with Node-RED while and the same time controlling my LEDs via Raspberry installation. Node-RED is extremely flexible.
For function block programming used in control systems in PLCs there are tools like copia.io that are basically GitHub for version control and can do visual diff of the diagrams
Speaking of controlling LEDs, there's also https://www.atomvm.net/doc/main/index.html. From a very quick scan it looks like it would need some fiddling (especially around the filesystem) to get it to work, but being able to scale down to smaller devices than a full pi would be very, very interesting.
> With AtomVM, you can write your IoT applications in a functional programming language, using a modern actor-based concurrency model, making them vastly easier to write and understand!
> It also supports WiFi networking on devices that support it, such as the Espressif ESP32. All of this on a device that can cost as little as $2!
If ErlangRED made it to a $2 chip, I'd be very happy!
How stable are node and edge IDs and their ordering? Could you sort the JSON before saving so the output is more stable and diff-able?
Another thought is, how would you code up a flow like this in your preferred language if you were just writing code in that language (you didn't care about the visual editting). Call most of the same functions, but do it directly. Ok, think about that, then write a transformation from the NODE language to output source in your preferred language.
I have some sense that coding this type of control flow logic doesn't work well with text line based diffs because of the complexity of the problem, less the syntax.
> Could you sort the JSON before saving so the output is more stable and diff-able?
Jsons are definitely diff-able but it's the difference between a change that changes the logic of the flow or a change that is purely visual and has no affect on the logic of the flow that is hard to distinguish by just diffing the JSONs.
Remember the Json contains location of nodes and labels of nodes as well as logic of nodes - if a label of a node changes it doesn't affect the flow logic or how it works.
So if the value of an 'x' changed, that means the node moved along the x-axis. Did the value of 'func' (function value for a function node) change, then that is a logically change. Now what change is important if you only looking at the JSON. So you need to remember that 'x' and 'y' are visual, 'func', 'tmpl', 'rules' are logical. And what is important for each node isn't the same across all possible nodes ...
> this type of control flow logic doesn't work well with text line
Exactly that's why places like GitHub would need a lot work to support visual coding in proper way. Diffs would have to become like image comparisons that line-by-line diffs.
And that is partly why visual coding has taken off - the tooling isn't available.
I wonder what would happen if they structured the JSON so that control-relevant information was just saved in a different place than visual/label-relevant information? EG, seperate form from function and thus more easily determine which has any changes in a diff?
There really isn't a good way to compare visual code textually. It similar to attempting to compare 3D scenes in 2D - textual code and visual code are a dimension apart in their representation.
I created a visual comparison that combines both visual and semantic textual changes for NodeRED[1]. For me that is the only solution: visually comparing visual code.
One star and it gave me the inspiration to create this ;)
The people who use Node-RED aren't the same that can develop those kinds of solutions. Those that can port Node-RED to Python don't need to use a low-code visual environment.
It isn't like Emacs users: Emacs user can also extend Emacs and probably even fix a bug in the lisp code of Emacs. Node-RED users use Node-RED becausae they don't want to program nor learn to program nor understand how Node-RED works.
Hence my attempt to take the whole concept and bring to a collection of developers who might appreciate visual FBP but who could also code extensions to the editor.
My goal is to bring visual FBP to a broader audience, be that developers or non-developers, be that Erlang coders or NodeJS coders, no matter - the ideas of FBP are important to understand, especially in our data-driven digital world.
and the biggest problem is that a strong bound is screen/display size --- any code chunk which won't fit on a single screen becomes hard to follow (which direction does one scroll next?) and if one does the obvious of declaring modules, then one runs into the wall of text which one was presumably trying to escape from --- just each word is wrapped up in pretty boxes and decorated with lines.
My take on this has always been that we don't have an answer for the question:
>What does an algorithm look like?
That said, I work in this style when I can, often using:
This is related to visual programming, not flow based programming.
I modified a python hosted lisp interpreter to read JSON flavored lisp - JLISP. It's much easier to get a web frontend to output JSON then lisp. Then I made a simple Low Code UI built around this language. The editor has operations (invocations of lisp functions) organized left to right like a timeline in CAD software, then you can click on an individual operation to edit the arguments.
Unreal Blueprint is probably the most popular. Its a custom system over a fairly customized C++ framework.
Visual scripting can be highly productive. The major issue is these visual scripts usually do not map cleanly back to text so you're throwing away decades of tooling. They usually don't merge well either.
This is neat - maybe move the example section higher up; it looks great and gives a much better idea of what the system looks like visually, which to me is key in node-based envs.
The thing which makes me be annoyed with Yours and the Node-RED site, is that you people do not define terms, before using them. Like "flow". What is this? A program?
You may want to consider Elixir instead. It has an easier syntax and is (don't quote me on this) equivalent in function and purpose with Erlang. Plus you get lots of other goodies like LiveView (notebooks) and a good web stack (Phoenix).
I prefer Erlang because it makes the uniqueness of its paradigm clear. Tail recursion, function matching and, bang, everything is a message!
Elixir makes everything seem like Ruby code and many of those Erlang concepts are hidden away, which is also ok but also takes much away.
My aim with ErlangRED is that both are supported, there is a repository for Elixir code[1] that is included in ErlangRED - so I don't take sides, rather I would like to take advantage of both paradigms.
I was an early adopter of NodeRed. Earlier, it worked exceptionally well.
Now? Not so much.
Well, that's not exactly true. Base NodeRed works as well as before. But the libraries of modules to interface with all sorts of websites, APIs, hardware, and other stuff is rotten to the core.
Most plugins/js modules either just don't work, or 'successfully fail'. The easier fail case is where the module can't be installed due to ancient (6mo or older JS, sigh) modules.
I abandoned NR because its basically a hyper-speed bitrot due to terrible library module versioning. And I didn't want to reinvent the wheel on every system I wanted to touch.
I wrote an article about this[1] and you are definitely right: a lot of packages are rotting away a bit however, because NodeRED does a lot to stay backward compatible, older packages still work. The oldest ones I'm actively using are five or more years old, i.e., haven't been touched nor updated - still work.
What I tried to say in the article is the same as you say: base NodeRED, the core works really well and has great features - no questions. And even if packages die, the core will still remain usable and that makes a difference.
Its a bit like saying Linux is a pile of broken bits because the ls command has been updated in ten years: Linux will always work and those commands that are based on the core will continue to work becausae the Linux kernel will largely remain backward compatible. Packages fail when they have external dependencies that change but code that is solely based on the core will continue to work.
Not surprising. Whenever you have a project like NR or HA you have a ton of barely working glue code written by people who just want to get it working and move on without any sort of commitment to maintenance. It allows these projects to rapidly expand their support but then that support quickly rots unless the core team assumes responsibility for maintenance. I really want to mess with home automation again but this sort of low software quality and resulting instability and maintenance hassles makes it not worth the effort.
I don't think this is easily to solve, in general. Similar orchestrators (e.g. n8n) have the same issues because there are a lot of components dependencies that change with time and there is no real cohesion between the core and all kind of plugins. Probably a future "contracts/manifests" linking orchestrators with components could help.
I think it's a fundamental flaw for such project that will never get out of the niche regions and will only have a few fanatic users and collaborators to use the JS/npm ecosystem which is known, as you say, for having breaking changes every release of every module and modules having updates every day because of grifting for stars/relevance/whatever on github/npm. I would like to see this based on something that's stable for very long times because module devs are not being bothered by the github issues 'is this project dead? there have been no updates for 7 hours?!?!?!? whats going on', with people responding with alternatives as happens every day with at least 1 js module. Scheme, Lisp, Perl, a Prolog etc maybe.
I am a big fan of visual programming, and also think visual and digital circuit design tools overlap quite nicely with functional programming languages.
The mess most people complain about in visual languages, also happens on text ones, the biggest difference is that visual ones make it quite clear the code lacks modularity.
Get to package stuff into reusable ICs more often.
> visual ones make it quite clear the code lacks modularity.
NodeRED has the concept of subflows to encapsulate reusable code. There are a number of features that NodeRED has that make it more usable for long term projects but it's a steep learning curve.
I've been using Node-RED everyday for the last three years to find the edge, something that cannot be done but there has always been a visual solution. I'm starting to think Node-RED might be turing complete ;)
But I have to admit, I did build a lot of tooling around NodeRED to make that happen - visual flow comparison[1] was something that was missing, introspection of message flow[2] and visual code management[3] - all these things I need to maintain an overview of what I am doing.
Visual programming helps automate flows in circuit design and I feel adoption would be higher as well since most circuit designers for some reason prefer GUI
> No Artificial Intelligence was harmed in the creation of this codebase. This codebase is old skool search engine (ddg), stackoverflow, blog posts and RTFM technology.
Good. To me, that also means that no human intelligence was degraded as a result. It sounds like a quality badge.
1. You don't explain what flow based programming is or when it should be used. It looks like it represents some type of data pipeline, but I'm not sure.
2. Who is this repo for? PL enthusiasts, tool builders of type $X?
3. What are operators?
4. I understand that this is supposed to execute on an Erlang cluster/VM... Could you build a JS implementation of the functions so that you could build interactive demos in a browser? The lack of multithreading in JS doesn't rally matter, you just want to be able to send 10 line examples from one function to another.
5. A fun challenge for documenting projects like this, explain it as you would to a non-technical family member. It forces you to define the other things in erlang in a very direct and simple way.
1. Ah sorry, indeed I forgot to add a link to J. Paul Morrison[1] the inventor of FBP - definitely add that.
2. The target audience is definitely Erlang developers who have not come across FBP before. But I didn't really ask myself that question since FBP is a hard sell at the best of times and selling Erlang with FBP is probably impossible! This is very niche ;)
3. Operators/Clients are hopefully those designing Erlang architectures. At the moment, that is really clear and it's something that has become clear for me in the last couple of months. When I started out, I was doing it as a finger exercise to see how far my Erlang skills would take me. Then I started creating test flows for testing node functionality - that turned into a testsuite[2] that can be used to test Node-RED functionality for correctness. At the moment, this is just a way of introducing the concepts of FBP to a new set of developers - Erlang developers.
4. I did actually create a in-browser-only Node-RED[3] which has some functionality and gives a feel of using Node-RED. There is also a live version of Erlang-RED[4] which is running on the BEAM.
5. Impossible :) To explain Node-RED and/or FBP to anyone outside of the IT - I keep using pipes and water hoses and sprinkles as metaphors!
6. I've made a couple of videos explaining my other visual coding idea[5] - didn't help ;) But yes, in the long term I definitely have to do something about the marketing because FBP does not sell itself...
Node-RED has a create frontend that just works and the APIs to the server are clearly defined. So it's relatively easy to swap out the backend - that's what I did.
Erlang has many features that make flow based programming (as implemented by Node-RED) simpler - e.g., message passing builtin and cheap processes and cheap concurrency.
NodeJS isn't truly concurrent, it is event-based single threaded - fast but not concurrent.
I use Node-RED every day! This is an exercise in applying the concept of flow based programming to another programming language. For me, flow based programming is like object oriented programming or functional programming or procedural ... etc. So the idea is to take the paradigm and apply to Erlang.
Clarification: I've been using everyday Node-RED for last three years and have built many things with it - including my mind map!
You should absolutely use whatever license you want but the actual price tag a nonstandard license adds is 'far fewer people will be able to try or contribute to your project'. Which might be totally ok for your purposes, just pointing it out since many people who dabble in 'license as messaging' for fun aren't aware of the impact/implications.
This has become clear since I started using the license, fortunately what I do is so niche that few contribute :)
I understand the implications but I also don't like big-tech to stealing my code and leaving me out in the cold. In sense, this license is a snub of the wall-gardens that big-tech has become. This software is something that they can't simple integrate into their product and sell on for profit. It stays out of the wall gardens.
Oh I'm definitely not arguing with you over what license you ought to have on your own project for your own reasons. You want a zany license, let your freak flag fly!
It's a cool license in that it triggered Google enough to bar the use of jslint and led IBM's lawyers to seek out a special exemption to allow them and their customers to use JSON for evil.
No company's PR will ever tell you what the soul of a company is like but their lawyers will indirectly tell you everything.
That’s a ridiculous take. The fact is “evil” is so ill defined that it would have been an unlimited liability. That’s a reflection of the world, not IBM.
Plenty (in not most) legal cases hinge upon non specifically defined terms hence lot of lawyering is not "this is definitely legal/illegal" but "what's the risk of this going wrong if we face a jury?"
Companies take tens of thousands of legal risks every day and they single out particular risks over others to try and indemnify themselves because they think that risk is serious.
Theyre not going to admit that theyre worried that the company is evil enough to qualify under a reasonable person's interpretation but thats what theyre thinking.
It's the same with your employment contract. The level of nasty bullshit they put in there ("lawyers made us!") is probably the most accurate meter of how horrible (or not) the company will be towards you as an employer.
About 15 years ago I made a custom MIT or BSD license that added:
"You agree to think carefully and always reflect about what you do and why you do it" or something to that effect. I thought I was so clever at the time.
Now I'm on the fence about adding philosophy to software licenses, even if only in a joking way. It can be fun and even thought provoking, but it could get in the way of genuine business adoption for basically no return (you never know if it helps influence anyone for the better, and most likely it doesn't).
Any license that isn't a word-for-word match to one of the "approved" licenses can potentially trigger a need for legal review, depending on the enterprise. Lawyers are expensive so quirky licenses can be a deal-killer for any customers who take licenses seriously.
If you want to be funny, put an easter egg in your code, don't mess with your license.
I don't write code for corporates, so my license is purely fictive. I cannot enforce my license but I can prevent corporates from taking my code and wrapping it into a product and selling it on for a profit. While not passing on a cent to me.
If a corporate wishes to use my code, then they are welcome to pay me a license fee or a one-off payment for a non-distributable license.
My inspiration was came from Douglas Crockford and the JSONlint[1] license.
Why not have a message? I mean if big-tech won't use my software because they legally think they might do evil with it, so be it.
Do I really want big-tech to wrap my software into a product and sell it for profit while not giving me a cent because what I did was share my code without strings attached?
I don't know. I would like to make this place just that little bit better and if if that's a license that makes folks think about what is evil, heck why not!
> Or they'll write to me and say, "how do I know if it's evil or not? I
don't think it's evil, but someone else might think it's evil, so I'm
not gonna use it."
> Great. It's working. My license works. I'm stopping the evildoers.
Or cautiously logical people who are probably doing good but don't have an absolute certainty that they are, which is probably the best way to live.
I saw Node-RED recently, and now this. While I think these are cool (and I would rather Erlang than NodeJS), is there some code based system for these kinds of things?
I know I can just write code, but I mean a DSL or some library specifically oriented around solving exactly these kinds of event driven sequencing/action problems.
Not to my knowledge, the closest thing to a DSL for Node-RED is its flows.json format for storing flow configurations.
It's a very simple array-of-object format with the graph/flow definitions defined by node ids and wires. It is fairly simple to create a minimalist interpretor for that format.
Also NodeRED is a very stable piece of software that has been around for over ten years. Yet I can take a flows.json file from today and have it run on the first releases of Node-RED, i.e., the format is backward compatible yet easily extendable.
> event driven sequencing/action problems.
Flow based programming[1] isn't strictly event driven, it's implemented that way in NodeJS because NodeJS does not have the concept of messages, only events.
Erlang has message passing between independent processes and therefore, IMHO, better suited to FBP.
[1] https://jpaulm.github.io/fbp/index.html
Firstly, thanks for the link and information on more of the roots of this kind of programming, I couldn't find this when I looked.
Regarding NodeRED. I don't dispute that it's stable, but my experience actually running any nodejs based software is that it's all heavily designed for running out of a container and a massive PITA to maintain otherwise. I already have enough hassle running zigbee2mqtt on my FreeBSD home server which wouldn't work in a linux jail because I couldn't figure out how to expose the zigbee dongle to it.
Secondly, the reason I wanted to run one of these was not because I wanted to use it to automate my house, but instead because I wanted to understand how it works so I could replicate it in my own weird way.
I suggest one of Benthos' descendants, either Bento or RedPanda Connect.
Configuration driven, with a transformation DSL, and good documentation (I prefer the Bento documentation over RedPanda Connect's documentation). And Benthos explicitly refused to ack a consumed message upstream unless it had been acknowledged downstream, so it wouldn't drop messages on the floor unless you explicitly told it to.
There's also flogo https://github.com/TIBCOSoftware/flogo
Wouldn’t all the ReativeX libraries (e.g. RxJS) count?
Is there any specific features you’d be looking for?
> Node-RED is a amazing[*] tool for creating flows that describe concurrent processing, it is just a shame the NodeJS is single threaded. So why not use something that is multi-process from the ground up?
That's great! But now that we are doing this, it kind of makes me wish that it was not multi-processing and Erlang, but a more mainstream language with a better library ecosystem and focused on multi-threading instead, Rust comes to mind but there could be a better one.
Is there a visual flow programming language based on Rust that is relatively mature?
> wish that it was not multi-processing and Erlang
Hehe - honestly I chose Erlang because I love the syntax and secondly because it is so niche. Flow based programming is niche, visual FBP is nicher, Node-RED is even nicher and Erlang is niche - it's niche all the way down ;)
Have a look at Elixir which is Erlang with Ruby syntax and also has a large community.
Also the intention is to be using the flow editor and not the lanuage underneath, so the end user won't even know that it's Erlang or NodeJS - I aim t stay compatible with Node-RED so that flows are actually interchangeable.
So to an certain extent it doesn't matter which language is used.
I did chose Erlang also because it is well suited to flow based programming being message based and flow based programming is all about passing immutable objects amongst independent processes .... that's Erlang in a nutshell.
> makes me wish that it was not multi-processing and Erlang, but a more mainstream language with a better library ecosystem and focused on multi-threading instead, Rust comes to mind but there could be a better one.
I am not quite following, why would we drop multi-processing with isolated tiny heaps with a few KBs each and move to multi-threading with a large multi-megabyte stacks per thread and all sharing and writing to the same heap? It seems like a step backward.
> Rust comes to mind but there could be a better one.
I can see the excitement, but at the same time this is becoming a meme. I mean the drive-by comments of "why don't your rewrite it in Rust" on every repo or "show HN" post. It's kind of like nodejs from years past: "why don't your write in nodejs, it's webscale!". The problem is, after a while it starts to have an inverse dose response: it makes people run away from Rust just based on that kind of stuff.
>I am not quite following, why would we drop multi-processing with isolated tiny heaps with a few KBs each and move to multi-threading with a large multi-megabyte stacks per thread and all sharing and writing to the same heap? It seems like a step backward.
Performance typically, hard to work in parallel on large amount of data performantly without multiple thread sharing a heap and you typically don't need large amount of threads because you don't actually have that many real cores to run them on.
Lots of little share nothing process are great conceptually but that does create significant overhead.
> Lots of little share nothing process are great conceptually but that does create significant overhead.
It doesn't, really? I have clusters running 1M+ Erlang processes comfortably per node.
> you typically don't need large amount of threads
Exactly, that's why Erlang only spawns just the right amount of threads. One scheduler thread per CPU, then a bunch of long running CPU task threads (same number as CPUs as well), plus some to do IO (10-20) and that's it.
Erlang is not as performant for heavy computational loads, this is bared out in many benchmarks, thats not what it's good at.
Message passing share nothing adds overhead when trying to reference data between processes because it must be copied, how would you do multithreaded processing of a large amount of data without a shared heap in a performant way? Only one thread can work on a heap at a time, so what do you do? Chop up the data and copy it around then piece it back together afterward? Thats overhead vs just working on a single heap in a lockless way. Far as I can tell the main Erlang image processing libraries just call out to C libraries that says something of that kind of work.
Yes Erlang indirects computation to a OS thread pool, multiplexing all those little Erlang process on real threads creates scheduling overhead. Those threads cannot work on the same data at the same time unless they call out to a libraries written in another language like C to do the heavy lifting.
.Net does similar things for say web server implementations, it uses a thread pool to execute many concurrent requests and if you use async it can yield those threads back to the pool while say waiting on a DB call to complete, you would not create a thread per http connection so the 4mb stack size is not an issue just like its not with Erlangs thread pool.
> Erlang is not as performant for heavy computational loads
Well, sure don't use it for heavy computational loads if it doesn't work for you. It works great for our use case and I think Node-RED is also not for heavy computational workload, but rather for event programming, where lots of events happen concurrently.
> how would you do multithreaded processing of a large amount of data without a shared heap in a performant way?
That's a specific workload, it's not universal, for sure. I haven't worked on cases there is a single large data structure and all million clients have to manipulate it concurrently. Not saying it's implausible, I can see a game server with an arena maybe being like that, but there are other systems that work like that.
>Well, sure don't use it for heavy computational loads if it doesn't work for you.
>why would we drop multi-processing with isolated tiny heaps with a few KBs each and move to multi-threading with a large multi-megabyte stacks per thread and all sharing and writing to the same heap?
This is the question I was answering and I gave you an example then you proceeded to say there is no overhead Erlangs process model, which now you admit exists.
>That's a specific workload, it's not universal, for sure. I haven't worked on cases there is a single large data structure and all million clients have to manipulate it concurrently.
Thats not the workload, simple as a single client doing heavy processing of large image or video, multithreaded shared heap wins due the the overhead of Erlang's processing model, thats why you give it up which is answering your original question again.
Geez this is tough crowd, first Erlang has no overhead, then its well that nots whats its good at, which I said. Then its well those are specific workloads, yeah serving little messages to a million clients is a specific workload too. I like Erlang, it's very good at specific things but it sacrifices quite a bit of performance to do that like many good abstractions. That can help overall reliability and performance on certain workloads but other modern languages following the shared heap multi-threading model will probably outperform it if care is taken in the design because they more closely match the underlying OS and hardware execution model of today with less indirection and overhead and they can also do other workloads that Erlang will never be good at with its share nothing green thread/process design.
> This is the question I was answering and I gave you an example then you proceeded to say there is no overhead Erlangs process model, which now you admit exists
But that's not what NodeRed-Erlang does. The author is confirming it one one of the sibling comments. You invented some new type of workload, were all clients have to access the same heap concurrently and telling the author to rewrite everything in Rust. But that's your use-case, not the the one we're discussion. My use case is not that either.
> Geez this is tough crowd, first Erlang has no overhead, then its well that nots whats its good at, which I said. Then its well those are specific workloads, yeah serving little messages to a million clients is a specific workload too.
The application in this case fits Erlang perfectly. You're the one who introduced a new type of a use case - when you need to concurrently process the same video or audio and arguing that Erlang is not good for it. Yeah, we agree, it's not good for that, use something else, Rust, C etc. It's like we're talking about trains and you're saying, don't use trains, they can't fly, what if you have to cross the ocean or something like.
> You invented some new type of workload, were all clients have to access the same heap concurrently and telling the author to rewrite everything in Rust. But that's your use-case, not the the one we're discussion. My use case is not that either.
Image processing or video or audio or parsing or working with large amount of data in a multithreaded way is not a new kind of workload its a very common one.
It was stated by the OP that they wished it would have been done in a more standard multithreaded language like rust. You asked why, I gave you an example, you said it wasn't true that there is no overhead to Erlang, I gave further details, you then pivoted to thats not the use case. How do you know what oersted's use case is?
>It's like we're talking about trains and you're saying, don't use trains, they can't fly, what if you have to cross the ocean or something like.
It's more like you asked when would you not use a train? I answered when you need to go across an ocean and planes can do both what trains and boats do just at higher cost but at greater speed.
Erlang is great at specific things, standard multithreaded shared heap runtimes are good at everything they just might take more effort and care to do the things Erlang can do, but they will perform better than Erlang. Erlang can technically do anything since its Turing complete but it will perform unacceptably at many standard workloads just like a train could go across an ocean if put on a barge and towed.
> I think Node-RED is also not for heavy computational workload
That is correct, Node-RED is designed for routing many thoughts of smallish data packets through some computational units. Each computational unit is basically a node with a message queue and each message is handled and then passed on.
This means that if a node takes too long, its queue will potentially overflow causing failure. However if there are fewer messages than nodes can take longer - there is nothing ingrained in Node-RED that prevents having long running processes and fewer messages.
As long as message queues don't overflow or messages get too large causing massive memory usage, then Node-RED will happily just work.
NodeREDs many use cases are controlling many small devices (home automation) and (I)IoT - collecting datapoints from 100s or 1000s of devices and routing that around to whereever the data is needed.
Nobody choses erlang for heavy computational loads, but for ease of use and a simple mental model. You do heavy compute on it mostly if you don't want to be bothered. Erlang's multi-process thing, it's not to be efficient at compute, it's to be efficient at crash recovery and distributed compute. It was there before multi-core multithread were a thing.
> a better library ecosystem
For what it's worth, Erlang's standard library is decently "batteries-included". The C interop also isn't terrible.[1]
[0]: https://www.erlang.org/doc/apps/stdlib/api-reference.html
[1]: https://www.erlang.org/doc/system/tutorial.html
> it kind of makes me wish that it was not multi-processing and Erlang, but a more mainstream language with a better library ecosystem and focused on multi-threading instead, Rust comes to mind but there could be a better one
Go sounds like a good candidate.
There are 3 or 4 Go FBP packages. It seems to be completely capable, but FBP itself doesn't seem to have reached enough critical mass for them to be popular.
https://github.com/flowbase/flowbase is one, and it has links to to some others.
seems like a jvm-based one would also fit that requirement. None that I can find
Note that this uses (I hope!) Erlang processes, which are not to be confused with OS processes.
Yep. I researched JVM and .NET thread sizes once, to compare/contrast with Erlang processes, and the difference is ludicrous.
Erlang: ~1k
64-bit .NET thread: 4MB
.Net uses real OS threads and that 4mb is the stack not the heap which must be allocated on thread creation. The heap is shared in the process amongst all the threads whereas Erlang is separate lightweight green thread/processes with dynamic heap and stack per process.
.Net originates in the Windows world where new processes are expensive but threads are much cheaper however they still use native stacks for iterop and hardware protection, stack must be contiguous so it is fixed at thread creation this is typical for all native os threads.
Erlang is a VM and runs on top of of processes and threads but exposes its own process model to the language that does not map 1:1 and therefore does not have a fixed stack size allowing many light weight processes. It pays an indirection penalty in performance for this but allows some nice stuff.
Java started with green threads and abandoned them, both .Net and Java use asynchronous concepts instead to get high concurrency now days cooperatively releasing threads and rentering them. There was talk of trying green threads out in .Net and bringing them back in Java for ergonomics compared to async, these green threads would not have such large fixed stacks and many more could be made similar to Erlang processes.
In the end, .NET is going hybrid route. Circa .NET 10/11 Roslyn will stop emitting state machines and something state-machine-like will be produced by the compiler itself iff the asynchronous execution suspends, and will be otherwise practically free.
https://github.com/dotnet/runtime/blob/main/docs/design/spec...
Mind you, short-lived operations are much cheaper with stackless co-routine model like in C#/F# or Rust than spawning an entire, even if virtual, thread/process.
I get why they went the async route due to performance implications and such, just like the old cooperative multi-tasking days everything was much lighter weight but one misbehaved process not yielding can block the whole OS.
Nice to see further improvement coming along in .Net however I conceptually prefer the everything is a lightweight process model to this function coloring async virus that invades the codebase now days, wouldn't be so bad if it had been easy to do sync over async from the beginning without caveat in .Net so you didn't need to have dual sync and async api all the way down the stack.
Usually dual APIs are offered when they do distinctly different things. Async calls on Socket go through epoll/kqueue via SocketAsyncEngine, which has observably different behavior to regular sockets. There is no such duality in high-level APIs like HttpClient's `.GetStringAsync`.
You are not free from having to worry about it with virtual threads either. If you block a carrier thread with such a low-level API call, the runtime will have a bad time. Or you will always end up paying for a context switch (if you maintain a separate pool of block-able threads).
Green threading UX is better for "regular" code that never bothers to take advantage of concurrency features, but if you do want them, you are quickly back to usually more ceremonious structured concurrency abstractions which, lo and behold, can reimplement the same task system with a worker pool but on top of virtual threads.
If you start to care about cancellation, suddenly you are back to coloring problem regardless. I agree with you that having to write many await's in quick succession is tiresome and unnecessary. It's partially a fault of library authors who name methods "GetSomeLongNameAsync" instead of "Get" or "Query". But other than this it's not as invasive as people make it out to be, and encourages writing highly concurrent code - reducing latency 2x can require moving one or two awaits around. Not so much in Java/Erlang/Go.
I haven't found this is the case many times, many api will have two versions of the api with XxxAsync versions all the way down, maybe less so now days but it took a while to get async entry points added everywhere so you could use it properly, such as ActionResult.ExecuteResultAsync and IAsyncEnumerable as a couple examples.
HttpClient one prime example and you can see the back and forth here internally from MS: https://github.com/dotnet/runtime/issues/32125
This snippet of code or something like it has been used in countless libraries to try and give a workable sync over async which should be baked in: https://github.com/aspnet/AspNetIdentity/blob/main/src/Micro...
I suppose the difference of opinion comes down to whether you consider hiding side effects a good or a bad thing. And whether you want to have easy composition and cancellation (Go struggles with passing context around a lot, in .NET it's much less of an issue if at all, depending on codebase).
If you don't want to join on some other virtual thread or goroutine completing immediately (and with Goroutines it's additionally painful since you need to communicate completion separately), it's more difficult and costlier than doing so with tasks/futures.
CancellationTokens can be passed in synchronous functions just fine, I have done cancellable sync apis before cancellation tokens existed usually its a .Cancel() on the thing doing the work just like Command.Cancel() in the Sql client or Connection.Cancel() in a network client. I agree CancellationTokens are better and always use them now and thats not the problem with async.
The problem is the need to color functions with async and if you have existing client code that call existing synchronous methods it not easy to move their internal implementations to async form so that you can remain backward compatible without copy/pasting the same layers of code with different function coloring to async.
Whoa!
The unit of comparison with BEAM processes are .NET tasks. The results are much less favorable to Erlang or Elixir in that case
https://gist.github.com/neon-sunset/8fcc31d6853ebcde3b45dc7a...
Yes Erlang processes all the way down!
I was just thinking of making something like this last night but using FLENG instead of Erlang ;) I'm really glad more people are taking inspiration from Node-RED, I will definitely be playing with this!
FLENG[1] - Wow, "low level concurrent logic programming language descended from Prolog" - sounds like an easy swap out for Erlang since they have the same heritage. ;)
[1] http://www.call-with-current-continuation.org/fleng/fleng.ht...
I've been interested in this for years.
Are there other similar projects that you're familiar with? Perhaps targeting other languages?
What are the major problems with this approach to programming? Are large programs hard to work with?
I know of Py-Red[1] which is an attempt to do the same but with Python. Using Node-RED as a frontend and something else as a backend, no I haven't heard of anything similar.
Main problems with visual flow based programming is the tooling: there aren't any good tools for doing visual comparison nor visual version control. Things like GitHub aren't able to render flow code as seen in the editor.
For Node-RED that means that the Json that defines a flow can be compared but that soon becomes meaningless because semantic changes get mixed with visual changes, i.e., that the x,y coordinates of a node have changed - is meaningless for the logic of the code.
This means that code sharing and collaborative development is difficult purely visually but only because tooling is missing. Compared to textual programming before things like SourceForge or Git and we had the same issues as we do with visual coding now.
Node-RED has a bunch of features to make large programs maintainable. Things like link nodes that allow jumping between flows also allow code reuse. There also subflows which encapsulate repeated code and can be referenced from anywhere.
That's also why I chose Node-RED for this because it seems to be the most mature and best maintained low-code visual flow-based programming tool. There are others such as n8n (which seems to be the other biggie) but they lack the generalness of Node-RED. I can create web sites with Node-RED while and the same time controlling my LEDs via Raspberry installation. Node-RED is extremely flexible.
[1]: https://github.com/mdkrieg/py-red
For function block programming used in control systems in PLCs there are tools like copia.io that are basically GitHub for version control and can do visual diff of the diagrams
Speaking of controlling LEDs, there's also https://www.atomvm.net/doc/main/index.html. From a very quick scan it looks like it would need some fiddling (especially around the filesystem) to get it to work, but being able to scale down to smaller devices than a full pi would be very, very interesting.
Wow! That does look interesting:
> With AtomVM, you can write your IoT applications in a functional programming language, using a modern actor-based concurrency model, making them vastly easier to write and understand!
> It also supports WiFi networking on devices that support it, such as the Espressif ESP32. All of this on a device that can cost as little as $2!
If ErlangRED made it to a $2 chip, I'd be very happy!
Shouldn't be too hard to get AtomVM onto chips now, according to rumour that's been worked on and made simpler.
This recent talk, https://www.elixirconf.eu/keynotes/the-atomvm-and-new-horizo..., probably showcases the process to some extent so it's a good idea to keep an eye on their YouTube channel for a recording.
re:version control
How stable are node and edge IDs and their ordering? Could you sort the JSON before saving so the output is more stable and diff-able?
Another thought is, how would you code up a flow like this in your preferred language if you were just writing code in that language (you didn't care about the visual editting). Call most of the same functions, but do it directly. Ok, think about that, then write a transformation from the NODE language to output source in your preferred language.
I have some sense that coding this type of control flow logic doesn't work well with text line based diffs because of the complexity of the problem, less the syntax.
> Could you sort the JSON before saving so the output is more stable and diff-able?
Jsons are definitely diff-able but it's the difference between a change that changes the logic of the flow or a change that is purely visual and has no affect on the logic of the flow that is hard to distinguish by just diffing the JSONs.
Remember the Json contains location of nodes and labels of nodes as well as logic of nodes - if a label of a node changes it doesn't affect the flow logic or how it works.
So if the value of an 'x' changed, that means the node moved along the x-axis. Did the value of 'func' (function value for a function node) change, then that is a logically change. Now what change is important if you only looking at the JSON. So you need to remember that 'x' and 'y' are visual, 'func', 'tmpl', 'rules' are logical. And what is important for each node isn't the same across all possible nodes ...
> this type of control flow logic doesn't work well with text line
Exactly that's why places like GitHub would need a lot work to support visual coding in proper way. Diffs would have to become like image comparisons that line-by-line diffs.
And that is partly why visual coding has taken off - the tooling isn't available.
I wonder what would happen if they structured the JSON so that control-relevant information was just saved in a different place than visual/label-relevant information? EG, seperate form from function and thus more easily determine which has any changes in a diff?
There really isn't a good way to compare visual code textually. It similar to attempting to compare 3D scenes in 2D - textual code and visual code are a dimension apart in their representation.
I created a visual comparison that combines both visual and semantic textual changes for NodeRED[1]. For me that is the only solution: visually comparing visual code.
[1] https://flows.nodered.org/node/@gregoriusrippenstein/node-re...
One star on Gighub, could still be good though. Any alternatives on Python that is equally used.
HomeAssistant build on Python have an Node-Red extension, would be nice to see a Python version with more seamless integration
One star and it gave me the inspiration to create this ;)
The people who use Node-RED aren't the same that can develop those kinds of solutions. Those that can port Node-RED to Python don't need to use a low-code visual environment.
It isn't like Emacs users: Emacs user can also extend Emacs and probably even fix a bug in the lisp code of Emacs. Node-RED users use Node-RED becausae they don't want to program nor learn to program nor understand how Node-RED works.
Hence my attempt to take the whole concept and bring to a collection of developers who might appreciate visual FBP but who could also code extensions to the editor.
My goal is to bring visual FBP to a broader audience, be that developers or non-developers, be that Erlang coders or NodeJS coders, no matter - the ideas of FBP are important to understand, especially in our data-driven digital world.
Well, there are pages on problems such as;
https://scriptsofanotherdimension.tumblr.com/
https://blueprintsfromhell.tumblr.com/
and the biggest problem is that a strong bound is screen/display size --- any code chunk which won't fit on a single screen becomes hard to follow (which direction does one scroll next?) and if one does the obvious of declaring modules, then one runs into the wall of text which one was presumably trying to escape from --- just each word is wrapped up in pretty boxes and decorated with lines.
My take on this has always been that we don't have an answer for the question:
>What does an algorithm look like?
That said, I work in this style when I can, often using:
https://www.blockscad3d.com/editor/
to rough out a design, or:
https://github.com/derkork/openscad-graph-editor
for more complex projects --- though that latter usually with a special-purpose library I've been working on: https://github.com/WillAdams/gcodepreview
This is related to visual programming, not flow based programming.
I modified a python hosted lisp interpreter to read JSON flavored lisp - JLISP. It's much easier to get a web frontend to output JSON then lisp. Then I made a simple Low Code UI built around this language. The editor has operations (invocations of lisp functions) organized left to right like a timeline in CAD software, then you can click on an individual operation to edit the arguments.
Here is a video walking through the system https://youtu.be/3Tf3lnuZcj8
Live edittable notebook (slow to load, runs python in WASM) https://marimo.io/p/@paddy-mullen/jlisp-in-buckaroo
I'll make a statically exported Jupyter notebook shortly
Unreal Blueprint is probably the most popular. Its a custom system over a fairly customized C++ framework.
Visual scripting can be highly productive. The major issue is these visual scripts usually do not map cleanly back to text so you're throwing away decades of tooling. They usually don't merge well either.
I see there's some suggestions here, you may want to give nodescript.dev a go, it's also nodejs based.
As a disclaimer, I'm somewhat affiliated with the project.
This is neat - maybe move the example section higher up; it looks great and gives a much better idea of what the system looks like visually, which to me is key in node-based envs.
One example would be the website I've created using Erlang-RED[1]
But it's early days so I've yet come up with good examples.
I would really like to get some examples from the Erlang community and implement those, so if anyone has any ideas please contact me!
[1] https://red-erik.org
The thing which makes me be annoyed with Yours and the Node-RED site, is that you people do not define terms, before using them. Like "flow". What is this? A program?
Please maintain a glossary.
Okay, It seems you do, https://flowfuse.com/node-red/terminology/, https://nodered.org/docs/user-guide/concepts, so please link to it.
Can someone recommend an Erlang course / book?
I keep running into learn you some erlang[1] which seems to be a good online resource. The author of which also wrote the "Erlang in Anger"[2] book.
Then there is the BEAM book[3] which provides an deep dive into Erlang and the BEAM virtual machine.
I'm not the best person to ask since I'm only starting out on Erlang! I much prefer to code visually :)
[1] https://learnyousomeerlang.com [2] https://www.heroku.com/blog/erlang-in-anger/ [3] https://blog.stenmans.org/theBeamBook/
Thanks for the suggestions! I looked at "learn you some erlang" and I like the style and humor of it.
You may want to consider Elixir instead. It has an easier syntax and is (don't quote me on this) equivalent in function and purpose with Erlang. Plus you get lots of other goodies like LiveView (notebooks) and a good web stack (Phoenix).
I prefer Erlang because it makes the uniqueness of its paradigm clear. Tail recursion, function matching and, bang, everything is a message!
Elixir makes everything seem like Ruby code and many of those Erlang concepts are hidden away, which is also ok but also takes much away.
My aim with ErlangRED is that both are supported, there is a repository for Elixir code[1] that is included in ErlangRED - so I don't take sides, rather I would like to take advantage of both paradigms.
[1] https://github.com/gorenje/erlang-red-elixir-helpers
I would move your sample images to the top of the README.
Don't bury the lede.
for a visual environment, I was expecting more images to show me what this tool is
There is a live demo[1] if you want to try it out!
For me, the focus is more on the Erlang part than the visual part since that's basically NodeRED.
[1] https://ered.fly.dev/node-red
thank you, very neat
the hivemind thinks alike
I was an early adopter of NodeRed. Earlier, it worked exceptionally well.
Now? Not so much.
Well, that's not exactly true. Base NodeRed works as well as before. But the libraries of modules to interface with all sorts of websites, APIs, hardware, and other stuff is rotten to the core.
Most plugins/js modules either just don't work, or 'successfully fail'. The easier fail case is where the module can't be installed due to ancient (6mo or older JS, sigh) modules.
I abandoned NR because its basically a hyper-speed bitrot due to terrible library module versioning. And I didn't want to reinvent the wheel on every system I wanted to touch.
I wrote an article about this[1] and you are definitely right: a lot of packages are rotting away a bit however, because NodeRED does a lot to stay backward compatible, older packages still work. The oldest ones I'm actively using are five or more years old, i.e., haven't been touched nor updated - still work.
What I tried to say in the article is the same as you say: base NodeRED, the core works really well and has great features - no questions. And even if packages die, the core will still remain usable and that makes a difference.
Its a bit like saying Linux is a pile of broken bits because the ls command has been updated in ten years: Linux will always work and those commands that are based on the core will continue to work becausae the Linux kernel will largely remain backward compatible. Packages fail when they have external dependencies that change but code that is solely based on the core will continue to work.
[1] https://blog.openmindmap.org/blog/crunchy-numbers
Not surprising. Whenever you have a project like NR or HA you have a ton of barely working glue code written by people who just want to get it working and move on without any sort of commitment to maintenance. It allows these projects to rapidly expand their support but then that support quickly rots unless the core team assumes responsibility for maintenance. I really want to mess with home automation again but this sort of low software quality and resulting instability and maintenance hassles makes it not worth the effort.
I don't think this is easily to solve, in general. Similar orchestrators (e.g. n8n) have the same issues because there are a lot of components dependencies that change with time and there is no real cohesion between the core and all kind of plugins. Probably a future "contracts/manifests" linking orchestrators with components could help.
I think it's a fundamental flaw for such project that will never get out of the niche regions and will only have a few fanatic users and collaborators to use the JS/npm ecosystem which is known, as you say, for having breaking changes every release of every module and modules having updates every day because of grifting for stars/relevance/whatever on github/npm. I would like to see this based on something that's stable for very long times because module devs are not being bothered by the github issues 'is this project dead? there have been no updates for 7 hours?!?!?!? whats going on', with people responding with alternatives as happens every day with at least 1 js module. Scheme, Lisp, Perl, a Prolog etc maybe.
I am a big fan of visual programming, and also think visual and digital circuit design tools overlap quite nicely with functional programming languages.
The mess most people complain about in visual languages, also happens on text ones, the biggest difference is that visual ones make it quite clear the code lacks modularity.
Get to package stuff into reusable ICs more often.
All the best for the project.
> visual ones make it quite clear the code lacks modularity.
NodeRED has the concept of subflows to encapsulate reusable code. There are a number of features that NodeRED has that make it more usable for long term projects but it's a steep learning curve.
I've been using Node-RED everyday for the last three years to find the edge, something that cannot be done but there has always been a visual solution. I'm starting to think Node-RED might be turing complete ;)
But I have to admit, I did build a lot of tooling around NodeRED to make that happen - visual flow comparison[1] was something that was missing, introspection of message flow[2] and visual code management[3] - all these things I need to maintain an overview of what I am doing.
[1] https://flows.nodered.org/node/@gregoriusrippenstein/node-re... [2] https://flows.nodered.org/node/@gregoriusrippenstein/node-re... [3] https://flowhub.org/
> All the best for the project.
Thank you!
Visual programming helps automate flows in circuit design and I feel adoption would be higher as well since most circuit designers for some reason prefer GUI
> No Artificial Intelligence was harmed in the creation of this codebase. This codebase is old skool search engine (ddg), stackoverflow, blog posts and RTFM technology.
Good. To me, that also means that no human intelligence was degraded as a result. It sounds like a quality badge.
Nice work! A couple of comments
1. You don't explain what flow based programming is or when it should be used. It looks like it represents some type of data pipeline, but I'm not sure.
2. Who is this repo for? PL enthusiasts, tool builders of type $X?
3. What are operators?
4. I understand that this is supposed to execute on an Erlang cluster/VM... Could you build a JS implementation of the functions so that you could build interactive demos in a browser? The lack of multithreading in JS doesn't rally matter, you just want to be able to send 10 line examples from one function to another.
5. A fun challenge for documenting projects like this, explain it as you would to a non-technical family member. It forces you to define the other things in erlang in a very direct and simple way.
6. Make a video explaining the project.
1. Ah sorry, indeed I forgot to add a link to J. Paul Morrison[1] the inventor of FBP - definitely add that.
2. The target audience is definitely Erlang developers who have not come across FBP before. But I didn't really ask myself that question since FBP is a hard sell at the best of times and selling Erlang with FBP is probably impossible! This is very niche ;)
3. Operators/Clients are hopefully those designing Erlang architectures. At the moment, that is really clear and it's something that has become clear for me in the last couple of months. When I started out, I was doing it as a finger exercise to see how far my Erlang skills would take me. Then I started creating test flows for testing node functionality - that turned into a testsuite[2] that can be used to test Node-RED functionality for correctness. At the moment, this is just a way of introducing the concepts of FBP to a new set of developers - Erlang developers.
4. I did actually create a in-browser-only Node-RED[3] which has some functionality and gives a feel of using Node-RED. There is also a live version of Erlang-RED[4] which is running on the BEAM.
5. Impossible :) To explain Node-RED and/or FBP to anyone outside of the IT - I keep using pipes and water hoses and sprinkles as metaphors!
6. I've made a couple of videos explaining my other visual coding idea[5] - didn't help ;) But yes, in the long term I definitely have to do something about the marketing because FBP does not sell itself...
[1] https://jpaulm.github.io/fbp/index.html
[2] https://github.com/gorenje/erlang-red-flow-testsuite
[3] https://cdn.flowhub.org/ (repo: https://github.com/gorenje/cdn.flowhub.org)
[4] https://ered.fly.dev/node-red
[5] https://flowhub.org
Ahh, you didn't create Node-RED editor. That's an external project. I didn't understand this at first, and probably explains a lot of my confusion
https://nodered.org/
No, that much talent I don't have :)
Node-RED has a create frontend that just works and the APIs to the server are clearly defined. So it's relatively easy to swap out the backend - that's what I did.
Erlang has many features that make flow based programming (as implemented by Node-RED) simpler - e.g., message passing builtin and cheap processes and cheap concurrency.
Why not just use Node-Red in that case?
NodeJS isn't truly concurrent, it is event-based single threaded - fast but not concurrent.
I use Node-RED every day! This is an exercise in applying the concept of flow based programming to another programming language. For me, flow based programming is like object oriented programming or functional programming or procedural ... etc. So the idea is to take the paradigm and apply to Erlang.
Clarification: I've been using everyday Node-RED for last three years and have built many things with it - including my mind map!
Congratulations for the launch! Also I never heard of that license before:
"LICENSE - DON'T DO EVIL" https://github.com/gorenje/erlang-red?tab=License-1-ov-file#...
Also I recommend you to put screenshots higher in the readme and also provide real world use case instead of fully abstract examples
Thanks for the suggestion, I’ll move the examples up and add a link to a running instance.
The license is a reminder that open source software isn’t free, in this case, I would like folks to think about what is evil. That’s the price tag.
The license isn’t enforceable and won’t be enforced.
You should absolutely use whatever license you want but the actual price tag a nonstandard license adds is 'far fewer people will be able to try or contribute to your project'. Which might be totally ok for your purposes, just pointing it out since many people who dabble in 'license as messaging' for fun aren't aware of the impact/implications.
This has become clear since I started using the license, fortunately what I do is so niche that few contribute :)
I understand the implications but I also don't like big-tech to stealing my code and leaving me out in the cold. In sense, this license is a snub of the wall-gardens that big-tech has become. This software is something that they can't simple integrate into their product and sell on for profit. It stays out of the wall gardens.
Oh I'm definitely not arguing with you over what license you ought to have on your own project for your own reasons. You want a zany license, let your freak flag fly!
Licenses are enforceable and will prevent many people from collaborating with you, due to the burden of legal review.
Hopefully karma takes care of it heh
It's a cool license in that it triggered Google enough to bar the use of jslint and led IBM's lawyers to seek out a special exemption to allow them and their customers to use JSON for evil.
No company's PR will ever tell you what the soul of a company is like but their lawyers will indirectly tell you everything.
That’s a ridiculous take. The fact is “evil” is so ill defined that it would have been an unlimited liability. That’s a reflection of the world, not IBM.
Plenty (in not most) legal cases hinge upon non specifically defined terms hence lot of lawyering is not "this is definitely legal/illegal" but "what's the risk of this going wrong if we face a jury?"
Companies take tens of thousands of legal risks every day and they single out particular risks over others to try and indemnify themselves because they think that risk is serious.
Theyre not going to admit that theyre worried that the company is evil enough to qualify under a reasonable person's interpretation but thats what theyre thinking.
It's the same with your employment contract. The level of nasty bullshit they put in there ("lawyers made us!") is probably the most accurate meter of how horrible (or not) the company will be towards you as an employer.
About 15 years ago I made a custom MIT or BSD license that added:
"You agree to think carefully and always reflect about what you do and why you do it" or something to that effect. I thought I was so clever at the time.
Now I'm on the fence about adding philosophy to software licenses, even if only in a joking way. It can be fun and even thought provoking, but it could get in the way of genuine business adoption for basically no return (you never know if it helps influence anyone for the better, and most likely it doesn't).
Any license that isn't a word-for-word match to one of the "approved" licenses can potentially trigger a need for legal review, depending on the enterprise. Lawyers are expensive so quirky licenses can be a deal-killer for any customers who take licenses seriously.
If you want to be funny, put an easter egg in your code, don't mess with your license.
Easter eggs in code that break things are evil!
I don't write code for corporates, so my license is purely fictive. I cannot enforce my license but I can prevent corporates from taking my code and wrapping it into a product and selling it on for a profit. While not passing on a cent to me.
If a corporate wishes to use my code, then they are welcome to pay me a license fee or a one-off payment for a non-distributable license.
My inspiration was came from Douglas Crockford and the JSONlint[1] license.
Why not have a message? I mean if big-tech won't use my software because they legally think they might do evil with it, so be it.
Do I really want big-tech to wrap my software into a product and sell it for profit while not giving me a cent because what I did was share my code without strings attached?
I don't know. I would like to make this place just that little bit better and if if that's a license that makes folks think about what is evil, heck why not!
[1] https://gist.github.com/kemitchell/fdc179d60dc88f0c9b76e5d38...
> Or they'll write to me and say, "how do I know if it's evil or not? I don't think it's evil, but someone else might think it's evil, so I'm not gonna use it."
> Great. It's working. My license works. I'm stopping the evildoers.
Or cautiously logical people who are probably doing good but don't have an absolute certainty that they are, which is probably the best way to live.
[dead]
[dead]
[flagged]
It does if you have blocking calls or you cannot yield to the event loop in time consuming code.
Are you asserting that single threaded execution of concurrent code is just as performant as multi-process execution?
Concurrency isn’t possible if there is only a single thread is my interpretation.
That is not true at all. Concurrency is a way of writing programs and is an orthogonal concern to how the program gets executed.
It certainly can be.