Matific Transforms Data Processing with SingleStore:
Achieving Real-Time Insights and Cost Efficiency
About Matific
Matific, a leading innovator in e-learning, empowers educators globally to instill a passion for math in their students through its sophisticated platform. With over half a billion educational activities played annually and more than a million concurrent users during peak times, Matific handles a vast and complex data ecosystem. Established on a foundation of robust pedagogy and crafted by experts, Matific delivers over 2,000 curriculum-aligned activities in more than 40 languages, addressing diverse learning styles and national curricula across 50+ countries.
Dedicated to enhancing educational experiences, Matific seamlessly integrates into both classroom and homeschooling settings, offering a compelling mix of game-based activities and comprehensive lesson plans. This versatile platform supports educators by offering a self-sufficient teaching solution that promotes independent learning, enabling students to develop problem-solving skills and conceptual understanding at their own pace.
Challenges
Matific recognized a significant opportunity to enhance the world wide education system by leveraging its extensive data resources. Traditionally, the process of handling exam data—from writing and administering exams to grading and analyzing results—was lengthy, taking weeks or even months. This extended timeline often rendered the data outdated by the time it was reviewed. Additionally, students experienced delays in receiving graded exams, which further prolonged the extraction of actionable insights. Matific aimed to transform this process by capturing real-time data, enabling swift examination and aggregation, and delivering timely insights to relevant pedagogy personnel without disrupting students.
*Initial architecture.
At first, PostgreSQL served Matific’s needs effectively, but as user numbers grew, it became insufficient to handle the increasing load. Matific needed a major architectural shift and initially incorporated Amazon RedShift and DynamoDB. However, they soon encountered challenges, including a 10-minute delay between data updates and teacher visibility, along with rising costs. To find a better solution that could scale effectively and deliver sub-second analytics with near-instant data freshness, Matific turned to Twingo. Twingo recommended benchmarking SingleStore as a replacement for both Redshift and DynamoDB. This solution addressed their performance and scalability needs, completing the evolution of Matific’s data infrastructure.
Data Challenges
- Data Freshness: To be effective, Matific needed to transform data into actionable insights within seconds. This required millisecond-level ingestion times and rapid aggregation.
- Real-Time Analytics: The system required near real-time analytics with fast query processing to support millions of students and teachers generating and querying data simultaneously.
- Scalability and Cost-Effectiveness: The solution needed to be multi-tenant, cost-effective, and capable of managing significant data volume fluctuations, such as during annual competitions when data and usage could surge up to 15 times.
- Integration and Performance: Matific required efficient integration of data from various sources to provide a comprehensive view for teachers. The solution needed to manage this effectively and affordably for schools.
- Technical Constraints: Initial solutions using PostgreSQL and Amazon RedShift faced performance issues, particularly with query handling and data freshness. Data freshness was still around 15 minutes, limiting effectiveness.
Matific needed to overcome these challenges to deliver timely, actionable insights while managing costs and maintaining system performance at scale.
Solutions
In response to the challenges outlined, Matific embarked on a comprehensive search for an optimal solution. After evaluating various vendors and conducting proof-of-concept trials with different products, they chose to partner with Twingo. Twingo proposed a transformative approach: replacing both RedShift and DynamoDB with SingleStore as a unified, high-performance serving layer.
*New architecture with SingleStore
The new architecture includes:
- Real-Time Data Ingestion and Aggregation: SingleStore pipelines automatically ingest data directly from Kafka into the fact table. The data is then aggregated and analyzed with other student data at the complex pre aggregated table. This entire process happens with a latency of less than a second, ensuring exceptionally fresh data.
- Efficient Querying: Application services query SingleStore directly for both state data and statistical analysis, benefiting from very low latency and high concurrency.
Twingo supported Matific in designing this architecture and secured SingleStore at a discounted rate, providing continuous assistance throughout the implementation process.
Results
- Parallel Queries: The issue of parallel query limitations has been resolved, with the system now capable of handling orders of magnitude more parallel queries.
- Rapid Aggregation and Transformation: Data aggregation and transformation are now completed in less than a second, a significant improvement from the previous 10-minute delay. This enhancement provides teachers with accurate, real-time analytical data during classes, allowing them to offer timely assistance to students.
- Scalability and Cost Efficiency: Since the project began, Matific’s user base has increased significantly, yet SingleStore’s resource consumption has remained relatively stable. This demonstrates the efficiency of SingleStore and results in a significantly lower cost per student and teacher, reflecting effective cost optimization.
- Reduced Costs: By replacing Redshift and DynamoDB and using SingleStore Matific managed to save 50% on costs.
"Twingo guided us through the proof-of-concept (PoC) process exceptionally well. Their support included continuous communication with my team via Slack, where they responded promptly, thoroughly understood our needs, and kept us informed about relevant updates and features in SingleStore. They ensured that my team remained well-supported and autonomous throughout the process. Both Twingo and SingleStore continue to impress us. Even during peak competition days, with 500,000 to 1 million users online, the S2 cluster manages this level of concurrency and data ingestion seamlessly, without requiring scaling. The performance has far exceeded our expectations."