DealFinder application development

Customer: Under NDA

Solution: upon detailed analysis of the requirements, our team have outlined the following technical challenges for the development:

  • eBay data processing;
  • advanced search capabilities;
  • further supplies comparison;
  • system high load resistance;
  • high productive, easy scalable system, ready for evolution, flexible tuning and self-monitoring.

Among the other prime criteria was the requirement of one second search over the base.

Concept Soft specialists have developed the back-end layer using modularized easy scalable architecture.

Based on client's requirements, we have used NoSQL DBMS Apache Cassandra for operating with large amount of data for the following benefits:

  • fast processing,
  • easy scalability with redundancy,
  • fault-tolerance,
  • advantages of MapReduce pattern use.

The developed system is powered with Solr search (including geo search) and Hadoop for analysis and data transformation jobs. System job management and scheduling is Quartz based. We have implemented business logic scalability by the advanced use of Terracotta Cluster and its Master-Worker additional module. Cache performance optimization was leveraged by utilization of Terracotta clustered Ehcache. To bond key technologies and business logic Concept Soft engineers have used Spring framework.

Self-monitoring and benchmarking for bottle-neck detection in production environment makes system more predictable and stable. Cluster configuration results in 2 types of nodes easy for on-demand adding. For load balancing and high availability HAProxy was used.

Selecting Amazon EC2 proved to be a cost-efficient decision, as computing capacities are paid based on consumed instance hours; moreover the platform provides possibilities for easy application scaling. In case the use of Amazon is no longer required, the system may be seamlessly ported on customer's physical cluster.

Technologies: Java, Amazon AWS (EC2, EBS, S3), eBay API, Spring (IoC, AOP, RESTful WS, RMI), Apache Cassandra 0.7, Solr, Apache Hadoop, Terracotta 3.5 (including Distributed Ehcache, Quartz 2.0 scheduling), HAProxy, Tomcat 7.0.

Teaser: 
The development of shopping comparison web application focusing on eBay, Amazon, etc., which had to comprise Querty application, Addltem application and Findltems Advanced application.