Modern blockchain applications often involve complex workflows for creating and managing digital assets on-chain. These assets can represent anything from tokenized financial instruments to digital collectibles. Beyond the complexities of writing the smart contract code itself, creating a real world digital asset typically involves multiple sequential steps, such as creating on-chain accounts, deploying smart contracts, assigning permissions or ownership and calling contract functions to configure the assets once they’re deployed.
Because blockchains function as distributed ledgers, transactions must always be processed in order, and in real world blockchain applications the deployment of one contract will often depend on the existence or configuration of another. Considering that even a single digital asset could involve complex interactions between dozens of smart contracts, careful orchestration is not only challenging but absolutely imperative when it comes to scaling the creation of such assets beyond the level of a technical playground.
Consider something like a tokenized real world asset. At its core, the RWA might be a simple ERC-20 token. This is fine if all you want to do is create a memecoin – just write the smart contract and deploy it. But what happens when you’re tokenizing a more complex asset like an auto loan? What happens when there are counterparties involved who need on-chain visibility into the asset’s transactions? What happens when a single digital asset requires ten different connected smart contracts and you need to deploy 500,000 of these assets?
Something like an auto loan might have an originating bank, a borrower, a firm that services payments, an associated physical asset (the vehicle itself), the vehicle’s title, perhaps investors who are holding a portion of the loan in an investment portfolio … the possibilities are endless, and all of these parties and relationships need to be represented on-chain. On top of that, the system becomes increasingly complex as you add considerations for privacy, regulators and even government actors.
At a small scale, managing the workflows for these types of complex digital asset deployments is relatively straightforward. A few assets can be created and configured manually or through simple automation, and once deployed they never need to change. Infrastructure utilization is low, failures are rare, and orchestration is simple. But as the number of assets grows and contract dependencies multiply, as the need for upgradability comes into play, and as speed demands increase, coordination becomes much more intricate.
The Complexities of Scaling Blockchain Workflows
Once you move beyond a handful of digital assets, the challenges multiply quickly. What seems trivial at the playground level becomes a web of interdependent operations when hundreds, thousands, or even millions of assets are involved. The problem is not the blockchain itself. Modern layer 1 networks can process many thousands of transactions per second, but orchestrating multiple dependent workflows across many accounts, contracts, and systems is a different story.
Real world scaling issues include
- Sequential execution and nonce management: A single on-chain account cannot execute two transactions at once. Order of operations for accounts and transactions is handled by nonces and sequence numbers, with each transaction from each account requiring a unique attribute that identifies where it goes in the ledger.
- Multi-contract dependencies: Complex assets rely on multiple contracts that must be deployed and configured in a specific order. These dependencies can create natural bottlenecks as later steps cannot proceed until earlier ones are finished.
- Participant and permission management: Each participant in the workflow may require on-chain representation and specific permissions, which multiplies the number of transactions and introduces additional points of failure.
- Infrastructure limits: Worker systems, databases, and orchestration queues can become choke points long before the blockchain itself is saturated. Connection limits, memory consumption and CPU utilization all come into play.
- Partial failure scenarios: If one workflow fails mid-process, incomplete assets can be left on-chain, complicating reconciliation and retries and potentially interfering with the completion of other workflows.
- Operational costs: Continuously running hundreds of workers or large containerized orchestration environments can be expensive – very expensive – especially when scaling beyond a testing infrastructure.
The conclusion is that simple automation strategies that work for a few assets quickly crumble under real world scale. It’s not that blockchain technology cannot handle thousands of operations. It’s that the application layer must manage all the interactions, dependencies, and failures, which becomes exponentially more complex and expensive as scale increases.
Parallelization to the Rescue
The key to scaling these workflows is of course parallelization: structuring the system so that each unit of work, each asset or sub-workflow, can proceed independently wherever possible while only serializing operations that must be sequential on-chain. Easy, right? All we need is an async task queue like Celery, a broker like RabbitMQ to manage all the tasks, and a database like Redis to store the results.
Unfortunately it’s not that simple. In the blockchain space, parallelization is conceptually simple but practically challenging for a number of reasons. With decentralized systems we need to think about it a bit differently. For example, even if we design a parallel system that perfectly respects the strict transaction ordering rules imposed by blockchains, individual blockchain transactions still take time. Another reason is that the introduction of a distributed, sequence-sensitive ledger introduces potential race conditions at the network and worker layers, not just the database layer. As if that wasn’t confusing enough, race conditions in decentralized systems can cross layers.
So what’s a lowly blockchain developer to do? Surely there must be a way to efficiently orchestrate complex workflows in a parallelized, decentralized, distributed system, right? The good news: there is. The goal is to design a system that pushes the limiting factor away from your application infrastructure and onto the blockchain itself. If your application is operating efficiently, the only limiting factor should be the speed of the network itself.
Let’s learn how to actually do it.
Task Queues and Worker Pools
Before we get to a truly scalable workflow design, we need to understand an architecture that seems scalable, and where it breaks down. When designing asynchronous workflows, many teams will reach for the same tools initially: a task queue like Celery; a broker like RabbitMQ; a result store like Redis. We put it all together and create a pool of workers running on EC2 or ECS with a couple vCPUs and a few gigs of RAM. This pattern is familiar, well supported, reliable, and reasonably priced.
Now consider that in a system that creates complex real world assets – such as our tokenized auto loan – a single asset might require a multi-transaction sequence of on-chain operations like the following.
- Create and fund an account or multiple accounts that will deploy and own the various smart contracts.
- Deploy the contract that represents the asset itself (an ERC-20 token, for example, with a total supply of 50,000 tokens representing a $50,000 auto loan).
- Deploy a data or metadata contract that stores regulatory, compliance, or structured attribute information for the asset.
- Deploy contracts to represent key participants in the loan process (borrowers, banks, investors, due diligence firms, etc)
- Deploy an access control contract that defines which participants can view or modify parts of the asset.
- Link the contracts together as needed by calling contract functions that set variables and configure permissions and payment rules.
- Mint the auto loan tokens to appropriate addresses in appropriate amounts.
- Add the completed asset’s various contract and token holder addresses to a pre-deployed registry contract that tracks payments on all assets over time and provides functionality for end users.
While complex, none of this is exotic in systems that rely on multiple on-chain components working together. The sequence matters. Each step builds on the previous one, and the later parts of the workflow cannot proceed until the earlier components exist and are fully configured.
In a Celery based architecture, each of these steps becomes a task in a queue. We can group and chain the tasks together in the right order and create the needed dependencies between them with something like Celery’s Canvas. Let’s assume the whole process requires 25 transactions from various accounts and takes 60 seconds to complete on average, with each sequential transaction taking 2-3 seconds to be confirmed. Workers deploy contracts, call configuration functions, update workflow state, and enqueue the next operation.
When you need to deploy a single asset it takes 60 seconds. Fifty assets? No problem:
- Create and fund fifty on-chain deployment accounts so each asset has its own dedicated deployment account and that account can be locked for use by only one worker at a time.
- Scale vertically by bumping up the CPU and RAM on your ECS container.
- Scale horizontally by starting 50 celery workers.
Throughput improves. Parallelism increases. Deploying 50 assets still takes 60 seconds because you’ve got 50 workers deploying at once. Want to scale to a thousand assets? No problem. Bump up to 100 workers, 100 deployment accounts, a slightly bigger ECS container, and deploy 100 assets at once. The early results look promising. It still only takes 600 seconds – ten minutes – to deploy and configure 1000 assets.
But … what if you need to deploy and configure 3 million of these assets?
Welcome to the Edge
At this point, the system starts running into real limitations. The first and most obvious is cost. A worker-pool architecture scales linearly: more parallelism requires more workers, and each worker is a real process or thread consuming real CPU and RAM, even when it isn’t actively doing anything. Deploying a few hundred assets in parallel is manageable. Scaling to tens of thousands, hundreds of thousands or millions requires so many cores and so much memory that the infrastructure bill quickly becomes the primary funding mechanism for Jeff Bezos’ next Blue Origin mission. Additionally, deployments are probably not constant – they’ll tend to come in batches – which means you end up paying for idle capacity most of the time. Sure – we can orchestrate this better with autoscaling or by other means, but it’s still not ideal.
The next limitation is coordination overhead. Every deployment step runs through shared systems: your application and APIs, the message broker, the worker pool, a database that tracks workflow progress, caches and locking mechanisms used to manage deployment accounts…the list goes on. At small scale, these components behave predictably. At real scale, they become choke points at best and behave completely unpredictably at worst. Brokers and databases hit connection limits, result backends become saturated with writes, tasks are marked as successful but the results don’t exist in the result store yet, the OOM killer starts randomly killing long running workers due to memory leaks. Even stranger unexpected bottlenecks emerge too, like the fact that Flower – a framework for managing and monitoring celery tasks in real time – actually creates connections to the broker, requires RAM and CPU, and has default limits on how many tasks it can display before it starts wiping out earlier ones, creating the potential for tasks to fail without any visibility into what actually happened because flower’s UI maxes out at 100,000 tasks by default. The blockchain isn’t slowing you down; the coordination layer is.
There’s also the issue of state pressure. Each asset deployment isn’t just a transaction, it’s a sequence of dependent steps, and every in-flight sequence needs to track its own state: which step it’s on, which account it’s using, which contracts have already been deployed, and what’s ready to proceed. When you’re running a few dozen workflows, this is trivial. When you’re running hundreds of thousands, the volume of state updates and reads creates its own bottleneck. Your application spends more time managing metadata about the workflow than doing the work itself.
Finally, partial failures become a real operational burden. With millions of workflows executing concurrently, some fraction will inevitably fail mid-sequence: a network blip, a temporary RPC outage, a database stall, or a worker crash. Recovering these workflows cleanly – without duplicating on-chain operations, without leaking incomplete assets, and without creating inconsistent state – becomes increasingly difficult as concurrency rises. While certainly capable of scaling to extreme levels in specific environments, traditional task-queue systems were not designed for massive, highly interdependent sequences where the cost of retrying the wrong step is a permanent on-chain artifact.
The bottom line is that the traditional approach scales well, but only to a point. It works for dozens of assets, it’s tolerable for thousands, but it becomes economically and operationally infeasible long before you reach millions. The blockchain can handle the throughput. Your application infrastructure cannot. The bottleneck was never the blockchain.
Moving the Bottleneck Off the Application
The key insight for scaling to the deployment of millions of complex assets is that your application infrastructure must not be the limiting factor. The traditional task queue model centralizes orchestration in a fixed pool of workers, relying on memory, CPU, and broker and database connections. The scalable alternative pushes orchestration away from the application layer to systems that are designed to handle the level of scale we need, relying instead on ephemeral compute that is state-independent with respect to each workflow. Each deployment sequence is handled independently, without long-lived workers and task queues.
Serverless compute, such as AWS Lambda, allows each unit of work, each asset or sub-workflow, to execute independently. A lambda function or step function can spin up, perform the sequence of transactions required for that asset, and terminate. It can use your own existing REST APIs to communicate results and changes as needed with your application layer. Resources are allocated on-demand and decommissioned when they’re no longer needed, removing the need to pre-provision hundreds of CPU cores or gigabytes of RAM across always-on infrastructure. Horizontal scale is essentially infinite within service limits, and you only pay for the compute actually consumed during execution, not idle workers.
By decoupling workflow orchestration from persistent infrastructure, you eliminate many of the cost and coordination bottlenecks inherent in worker pools. You also reduce operational complexity. There are no ECS clusters to manage, no task monitors to scale manually, and no memory leaks to deal with. The workflow logic lives in a simple synchronous function which takes only the amount of time required to deploy one single asset, and persists its progress externally, using traditional REST API calls or scalable cloud-native services to manage updates and maintain data consistency with external systems as needed.
Respecting On-Chain Order at Scale
Of course, all parallelization must still respect the sequential nature of blockchain transactions. Even with the essentially infinite scalability of something like lambda, you cannot submit two transactions from the same account at the same time. A scalable system handles this by assigning independent accounts to independent workflows or carefully sequencing transactions with an external state tracker and submitting them perpetually over time. This ensures that nonces are managed correctly without requiring long-lived workers that could die at any moment to maintain in-memory state.
Each lambda invocation can of course maintain the minimal execution state required for its asset sequence, as well. This might include the current step, the account to use and the contracts already deployed, stored externally if needed. Management of nonces for the deployment account is built into the deployment function itself, with the only requirement being that the account is locked and unlocked at the beginning and the end respectively, ideally via an API call or by updating a lightweight DynamoDB table. If a Lambda invocation fails, the workflow can be retried safely without impacting other deployments, because it knows which step it was on and the deployment account can remain locked until retries have been exhausted and we’ve decided it’s time to give up and admit defeat – for that one single asset. This approach allows tens of thousands of deployments to proceed in parallel while respecting the strict ordering rules imposed by the blockchain.
By designing the system this way, costs drop dramatically by eliminating wasted resources and the only real limitation becomes the blockchain network itself. Chances are the amount of work you need to do – if your architecture is sound – is barely a blip on a modern blockchain network’s radar.
Event-Driven, Stateless Orchestration
A truly scalable architecture is event-driven, but not in the traditional task-queue sense. Instead of a central broker tracking every workflow and step, events are lightweight signals that trigger on-demand, short-lived and isolated function invocations. New asset deployments or sub-workflows are initiated when these events occur, allowing the system to scale horizontally without the bottlenecks of shared state, persistent queues, or tightly coupled worker pools.
External storage for workflow state and metadata ensures persistent visibility into progress – even for retried and failed invocations – without burdening the execution layer. Observability, logging, and error handling become simpler because each workflow is isolated. Scaling up is as simple as allowing more concurrent lambda executions, which the platform manages automatically. There is no manual provisioning, no load balancing across persistent workers, and no shared-state bottlenecks.
The result is a system that can handle creating millions of complex, multi-contract digital assets with minimal operational overhead. Orchestrating hundreds or thousands of dependent, sequential blockchain transactions no longer requires a massive pool of application servers or workers. Instead, the architecture is elastic, stateless, and naturally aligned with the parallelizable units of work that blockchain workflows demand.
The End Game
Moving orchestration off the application layer and into stateless, ephemeral compute dramatically changes the scalability equation. Instead of managing hundreds or thousands of persistent workers, queues, and connections, each workflow executes independently, allowing tens of thousands – or even millions – of assets to be deployed in parallel, and putting the burden of infrastructure scalability where it belongs: on your infrastructure provider. The bottleneck is no longer your infrastructure; it is now the blockchain network itself. This approach preserves strict ordering rules, enables safe retries, and minimizes operational overhead.
Of course, this architecture does not eliminate all scalability challenges. Rate limits on RPC endpoints, network latency, node CPU and memory utilization, transaction propagation times, and mempool eviction policies all remain factors that must be dealt with. Ledger space consumption and gas optimization are additional considerations.
The goal of this architecture is to make your application infrastructure irrelevant and let the blockchain itself determine the upper limit of throughput. By decoupling orchestration from persistent application state, using event-driven, minimal-state workflows, and respecting the sequential nature of on-chain operations, you can scale far beyond traditional worker-pool approaches while keeping operational complexity and cost under control.
“Overloading the node” is a good thing. It means your application is truly scalable and operating at max capacity, and ensures that your technology lives at the cutting edge of what’s possible.
What are your thoughts on building scalable blockchain applications? What challenges have you had, and how did you solve them? Comment below!