Thursday, August 18, 2011

Asynchrony, Large Data Sets, and Crawling.

Okay, so I come from the bioinformatics world, and I never really cared that much for the concept of of asynchrony.



But I've spent the last week playing with Ning, as part of an architecture designed by Ivan & Pere at Datasalt.com, for peerindex.net; Ning is cool - it turns the nebulous concept of asynchrony into a NATURAL & EXTENSIBLE data acquisition strategy for novice programmers, that know asynchrony is valuable, but don't know why.



Anyways, I'm not going to explain Ning --- because its Java specific , and its very amenable to a google search. Rather, I'm going to explain the 3 important aspects of asynchrony.



- Asynchrony gives you scalability ---> you can start to decouple the ACQUISITION of data from the PROCESSING .



- Asynchrony gives you the FREEDOM to FAIL ----> One or two risky requests, or queries (run at the wrong time of day, for example) won't BREAK everything .



- Asynchrony gives you PERFORMANCE ! ----> By allowing you to process and integrate data in a seperate thread from acquisition , you're able to build applications

0 comments:

Post a Comment