I am super excited about Temporal. I think one of the biggest underacknowledged problems in business software is the way executional details pervade business logic. As soon as your critical logic doesn't simply exist in a request-response logic, it becomes fragmented over queues, scheduled jobs, ETLs, microservices, functions, etc. The actual processes that matter become Rube Goldberg machines that are hard to understand, maintain, observe, debug, and analyze. Not to mention hard to depict to the actual end user.
When I read the article "Why the Serverless Revolution Has Stalled" [1] on here a couple weeks ago, my reaction was that the reason is that serverless doesn't solve the business logic issue. Serverless removes operational details of servers, but often exacerbates process fragmentation. And the need to have "serverful" operational expertise is replaced by the need for serverless expertise. At least right now, this new expertise is far from trivial.
I haven't yet used Temporal, but I've spent a lot of time evaluating it, and its predecessor, Cadence. The idea is to model long-running business logic essentially as procedures, in ordinary code. These are called Workflows, and must be free of external effects. External effects are carried out by Activities, and the scheduling and tracking of results are handled by the Temporal runtime. The upshot is that you get to write workflows as though they have no time constraints. It's like async programming, but liberated from the confines of a OS process or machine.
If it turns out to be a useful home for business logic (I understand that it has at Uber), I think the next frontier is integrating it within UI frameworks. I'm imagining the next Rails being something like Next.js + Temporal. I still have a bunch of questions in my mind, like how to decide which data lives in Temporal vs. a OLTP database. Someone with more experience using Temporal probably could better answer this.
One of the reasons I'm particularly interested in this topic is because my company, Better.com, uses our own homebuilt workflow engine to model the days-long, multi-user business project of mortgage origination. In our case, our workflow engine is actually built as part of a full-stack framework that goes well beyond Temporal in scope, but it's not built as a generic platform, and we look at Temporal for inspiration on where things might be going.
You might be interested in looking into Camunda, which does similar things. It's very quick and easy to set up detailed process flows which model and track multi-step transactions via microservices.
It also has a great visual interface using BPMN for understanding where your processes are in a flow. I don't see anything like that in Temporal.
The visual interface for workflow definition is more hassle than benefit. The majority of such applications' complexity is not in the sequencing of operations but in state management, expressions, data manipulation, parameter passing, and retry policies. None of this is visible in the diagram. So engineers have to use UI/DSL to design the flow, but then implement most of the complex logic in non strongly typed code as parameter passing is usually using a map.
Temporal represents all the business logic in one place in the programming language of your choice. All parameter passing is strongly typed.
I see people advocating visual representation for workflows. But if it is so great for programming in general, why they don't advocate the same for systems programming, for example? Linux kernel in JSON/XML anyone?
I've heard of Camunda from reading this excellent article [1], but I havent' looked too deeply into it yet.
BPMN vs code is something we have considered. Our workflow engine works more like state machines as code, which is kind of the worst of both worlds. The business processes get fragmented into logic for each state, and mixed with imperative effects, so its neither easy to see the process nor maintain the code.
The nice thing about BPMN is that it's probably a bit more learnable by the subject matter experts who ultimately determine what the correct business process should be. However, having seen some pretty monstrous process flowcharts, my guess is that it probably doesn't scale past a certain level of process complexity. I suspect that workflow-as-code is ultimately the more scalable option.
Look at how Temporal represents workflows as code. It doesn't use any intermediate representation like DAG or state machine. It executes your code directly as a synchronous program with blocking operations taking as long as necessary to execute. For example the following code would be absolutely valid as a Temporal workflow:
Camunda is pretty nice and I have proposed it for projects, but it was shot down almost every time because 'BPM visual modelling is old-world' (it kind of says that on the homepage as well; 'modernize your legacy BPM systems'). We did a prototype for a big insurer in the UK and that went well, but covid+brexit killed the project altogether. It is nice to work with and to just toss it aside because of BPM modelling (and the charts belonging to them that make people from the industry who encountered them early 2000s cringe) is shortsighted (considering you can do everything in code). It is just that when people read the homepage, I guess from comments partners/clients made, it looks like going back 15+ years when you had fat java applet clients doing BPM and it always turning into a mess in the end.
It's interesting - I clearly never encountered them the first time around, and to be quite honest when I discovered them alongside Camunda I found them pretty refreshing.
> Serverless removes operational details of servers, but often exacerbates process fragmentation. And the need to have "serverful" operational expertise is replaced by the need for serverless expertise.
I totally agree. As a consultant working with multiple companies, I see a slowly increasing number of Serverless applications. The code always looked bloated, with no chance of ever being migrated off the original cloud platform due to all the proprietary APIs in use. There was also no way to run any of them locally without extensive configuration. Serverless has a future, but this is not it.
It's interesting that you mention workflows and serverless together. In this regard, have you looked at things like AWS Step Functions and Azure Logic Apps? You get the best of both worlds - state machines with excellent workflow management tools along with the flexibility and cost effectiveness of AWS Lambda in the background.
Talking of "Serverless Revolution", I think we are going to see more of such abstractions as things evolve. Abstractions and tools built upon serverless functions that are going to cater more closely to problems being solved rather than worrying about managing new-found complexity of the functions themselves.
Before I'd ever heard of the concept of a "workflow engine", I looked into step functions, thinking that was exactly the solution I was looking for. But when I started studying them, it was apparent to me that they suffer from the exact same problem of fragmentation of logical processes. In my ideal world, the breaks between steps in a process look much more like `await`s in a process that is modeled by a single function, and not like a hard split between steps.
Our other co-founder Samar actually created Durable Task Framework which is the basis of Durable Functions. I actually do not think Max was involved in this project. :)
When I read the article "Why the Serverless Revolution Has Stalled" [1] on here a couple weeks ago, my reaction was that the reason is that serverless doesn't solve the business logic issue. Serverless removes operational details of servers, but often exacerbates process fragmentation. And the need to have "serverful" operational expertise is replaced by the need for serverless expertise. At least right now, this new expertise is far from trivial.
I haven't yet used Temporal, but I've spent a lot of time evaluating it, and its predecessor, Cadence. The idea is to model long-running business logic essentially as procedures, in ordinary code. These are called Workflows, and must be free of external effects. External effects are carried out by Activities, and the scheduling and tracking of results are handled by the Temporal runtime. The upshot is that you get to write workflows as though they have no time constraints. It's like async programming, but liberated from the confines of a OS process or machine.
If it turns out to be a useful home for business logic (I understand that it has at Uber), I think the next frontier is integrating it within UI frameworks. I'm imagining the next Rails being something like Next.js + Temporal. I still have a bunch of questions in my mind, like how to decide which data lives in Temporal vs. a OLTP database. Someone with more experience using Temporal probably could better answer this.
One of the reasons I'm particularly interested in this topic is because my company, Better.com, uses our own homebuilt workflow engine to model the days-long, multi-user business project of mortgage origination. In our case, our workflow engine is actually built as part of a full-stack framework that goes well beyond Temporal in scope, but it's not built as a generic platform, and we look at Temporal for inspiration on where things might be going.
[1] https://www.infoq.com/articles/serverless-stalled/
[2] I discuss this a bit here https://www.themuse.com/advice/engineering-manager-better-al...