D.A.G.G.E.R. Testnet Phase 1 Learnings

Introduction

We are thrilled to share the progress and insights from the GenesysGo Directed Acyclic Gossiping Graph Enabling Replication (D.A.G.G.E.R.) Testnet. The Testnet Phase 1 has provided valuable insights into the system's performance, revealing areas for improvement and highlighting the system's capabilities. Our team has been working tirelessly to refine and optimize D.A.G.G.E.R., addressing performance bottlenecks, enhancing the RPCs, improving memory management, and more. This report will delve into the key learnings and improvements made thus far in the ongoing first phase of the D.A.G.G.E.R. Testnet.

Interactive Block Explorer

The first major milestone achieved during this phase was the release of the initial interactive block and transactions explorer. This tool allows users and developers to track and verify the path of each transaction, enhancing transparency and trust in the system. It also provides valuable insights for developers to optimize the system's performance.

Performance Bottlenecks

The Testnet Phase 1 led us to discover and resolve several performance issues within the D.A.G.G.E.R. core. By enhancing memory utilization and efficiency, D.A.G.G.E.R. can now handle file uploads and gossip network message processing more effectively, leading to faster transaction finalization and a more robust consensus mechanism.

RPC Improvements

We made significant improvements in multi-threading in the D.A.G.G.E.R. Client RPC, allowing for a higher number of transactions to be processed concurrently. This not only enhances the system's throughput but also shows its versatility in managing transactions from various sources.

Cluster Crash

We experienced a node crash under specific high throughput conditions, disrupting the consensus and execution process. However, we successfully diagnosed and resolved this scenario, which has strengthened the robustness and reliability of D.A.G.G.E.R..

Graph Improvements

We made significant strides in memory management by transitioning the finalized graph history to disk storage. This reduces the memory overhead for running nodes, making D.A.G.G.E.R. more lightweight and accessible.

250 Million Transactions to Study

We collected metrics on over 250 million transactions and events, demonstrating the system's capability to handle large volumes of transactions. The data collected from these transactions can be used for detailed system tuning, optimizing performance and efficiency.

Throughput

The system was able to handle up to 4 gigabits per second (gbps) of network traffic, processing an impressive 20,000 transactions per second. This accomplishment validates the technical design and scalability of D.A.G.G.E.R..

Node CLI

We improved the D.A.G.G.E.R. node Command Line Interface (CLI) and documentation, enhancing the ease of use and accessibility of the system for node operators. 

Graph History Snapshots:

The development of the graph history snapshot system allows for efficient storage of transaction history. By adjusting the quantities of transactions in bundles, we can optimize the system for peak performance and efficiency.

Conclusion

In conclusion, the D.A.G.G.E.R. Testnet Phase 1 has been a resounding success thus far, providing our team with invaluable insights and opportunities for refinement. We have made significant strides in improving the system's performance, scalability, and reliability, demonstrating D.A.G.G.E.R.'s readiness for large-scale applications. Our team remains committed to continuous improvement and is excited to build on these learnings in the next phase of testing. We are confident that D.A.G.G.E.R. will continue to push the boundaries of what is possible in decentralized transaction processing, and we look forward to sharing more updates as we progress on this exciting journey.