Performance Update: Testnet Learnings, Part 2

Building on the Foundations

Following the informative and successful first phase of the GenesysGo Directed Acyclic Gossiping Graph Enabling Replication (D.A.G.G.E.R.) Testnet, we are excited to present the latest developments and enhancements that have been implemented in the subsequent releases. The team has been diligently addressing feedback, refining features, and expanding the capabilities of D.A.G.G.E.R. to ensure a robust and scalable network. This post delves into the recent releases' key updates, illustrating our commitment to continuous improvement and transparency.

Version 0.2.0

• Testnet Phase 1 inception and when the D.A.G.G.E.R. Hammer front-end was stood up. 

Version 0.2.2

• The 0.2.2 update brought critical enhancements to both user experience and core functionality.
• Our sleeker installation process now works in tandem with the "Wield Installer" for smoother node setups.
• We solved recurring graph panics by excising two primary immutable borrows within the controller, helping network stability.
• Adjusting how we track epochs has streamlined transaction processing as we continue to improve our understanding of network time management.
• Our first round of performance metrics was gathered to focus on separating transaction processing and file ingestion.
• Consensus runtime metrics versus internal throughput are discussed in more detail here.
• Our longer-term focus is real-world throughput, and we map the needed telemetry to better understand internal throughput for file upload and propagation.
• We scoped and began work on a publicly accessible dashboard of the network planned for release before Testnet Phase 2 commences. 

Version 0.3.1

• Instructions were added for changing the user when running as root, and enhancements were made to logging, providing greater clarity on network activities.
• The transition to version 0.3.0 saw fixes to Wield node exit behaviors and the filtering of genesis events, which are crucial for maintaining the integrity of the network's history.

Version 0.4.1

• The epoch length was increased, which impacts the timing of certain network operations, potentially leading to a more stable network (we are continuing to refine epochs).
• A new menu and various checks were added to aid Wield node operators in navigating and maintaining their nodes.
• Fixes were applied to trusted nodes and the update configuration process, enhancing the reliability of node connections and configuration management.
• The synchronization tick rate was bumped to 400ms as an experiment, which may seem minor but can significantly impact the network's responsiveness and efficiency.
• Lastly, the introduction of an uninstall feature provides users with a clean method to remove the software if needed.

Version 0.4.2

• Version 0.4.2 brought a host of improvements aimed at optimizing the system's performance and user experience.
• Conditional compilation for types has been introduced, providing flexibility and reducing unnecessary compilation when certain features are not required.
• More installer updates were made with build script fixes to ensure a smoother setup for developers.
• The thread count has been adjusted to better utilize system resources, and CPU changes, limit adjustments, and enhanced messaging contribute to the overall stability and performance of the network.
• Another adjustment in sync size and sync number threshold promises to improve the synchronization process, ensuring that nodes stay up to date with the network state more effectively.
• Additionally, the update to thread defaults on startup ensures that the system is configured optimally from the get-go when operators join. 

Version 0.5.0

• The latest release, version 0.5.0, brings stability and performance improvements.
• After analyzing testnet data, we have adjusted the recommended thread allocations for the processor, graph, and comms modules.
• We made multithreaded performance improvements to the core graph data structures.
• We completed p2p networking tunes and optimizations.
• The epoch length has been adjusted to 64 bundles.
• We developed a new monitoring stack for wield nodes. To learn more, visit: https://docs.shdwdrive.com/wield/monitoring-stack

Conclusion

As we continue to iterate on the D.A.G.G.E.R. Testnet, each version release is a testament to our team's dedication to delivering a quality decentralized storage network. The changes we've implemented are not just fixes—they are stepping stones towards a future where D.A.G.G.E.R. stands as a paragon of efficiency and reliability in the realm of decentralized technology.

We are grateful to our contributors and community for their continued support and feedback. Through this collaborative effort, D.A.G.G.E.R. evolves as a technology and an ecosystem. Stay tuned for further updates as we forge ahead with Testnet Learning Part 3, where we will explore the next set of enhancements and milestones.‍

Additional Resources

• Read the D.A.G.G.E.R. Litepaper: https://github.com/GenesysGo/dagger-litepaper‍
• Play the D.A.G.G.E.R. Hammer Game: https://dagger-hammer.shdwdrive.com/
• Run a D.A.G.G.E.R. Wield Node: https://docs.shdwdrive.com/wield
• ‍Review SHDW Token Details: https://www.shdwdrive.com/blog/unlocking-the-shdw-token