Big Data and Apache Hadoop – Part 1

  • Big data is data that exceeds the processing capacity of conventional database systems. The data is too big, moves too fast, or doesn't fit the strictures of your database architectures.
  • Big data is a collection of data from traditional and digital sources inside and outside your company that represents a source for ongoing discovery and analysis.
  • Unstructured data comes from information that is not organized or easily interpreted by traditional databases or data models, and typically, it's text-heavy. Metadata, Twitter tweets, and other social media posts are good examples of unstructured data.
  • Multi-structured data refers to a variety of data formats and types and can be derived from interactions between people and machines, such as web applications or social networks. A great example is web log data, which includes a combination of text and visual images along with structured data like form or transactional information. As digital disruption transforms communication and interaction channels—and as marketers enhance the customer experience across devices, web properties, face-to-face interactions and social platforms—multi-structured data will continue to evolve.
  • The majority of big data solutions are now provided in three forms: software-only, as an appliance or cloud-based.
  • The three Vs. of volume, velocity and variety are commonly used to characterize different aspects of big data
  • Volume:
    • to process large amounts of information
    • It calls for scalable storage, and a distributed approach to querying.
    • Hadoop is a platform for distributing computing problems across a number of servers.
    • Hadoop is the core platform for structuring Big Data, and solves the problem of making it useful for analytics purposes.
      • "MapReduce" approach pioneered by Google.
      • Typical Hadoop usage pattern involves three stages
        • loading data into HDFS
        • MapReduce operations
        • retrieving results from HDFS
  • Velocity:
    • the increasing rate at which data flows into an organization
    • fast-moving data tends to be either "streaming data," or "complex event processing."
    • NoSQL:
    • It's this need for speed, particularly on the web, that has driven the development of key-value stores and columnar databases, optimized for the fast retrieval of pre-computed information.
  • Variety:
    • The source data is diverse, and doesn't fall into neat relational structures
    • Semi-structured NoSQL databases meet this need for flexibility: they provide enough structure to organize data, but do not require the exact schema of the data before storing it.
Apache Hadoop:
  • A software framework for sorting, processing, and analyzing "big data".
    • Distributed
    • Scalable
    • Fault-tolerant
    • Open Source
  • Apache Hadoop is a fast-growing big data framework.

Advantages:
    • Problems with Traditional Large-Scale Systems
      • Processor-bound and lots of complex processing with bigger computers ( changed with distributed computers)
    • Programming complexity
    • Keeping data and processes in sync
    • Finite Bandwidth
    • Partial Failures
    • Distributed Systems: The Data Bottleneck
      • Traditionally, data is stored in a central location
      • Data is copied to processors at runtime
      • Fine for limited amount of data

Comments

Post a Comment

Popular posts from this blog

BDD - Acceptance Test Driven Development

Image
Acceptance Test Driven Development What is BDD: BDD is an agile software development technique that encourages collaboration between developers, QA, and non-technical or business participants in a software project. It's more about business specifications than about tests. You write a specification for a story and verify whether the specs work as expected. The main features of BDD development are outlined below:   A testable story (it should be the smallest unit that fits in an iteration) The title should describe an activity The narrative should include a role, a feature, and a benefit The scenario title should say what's different The scenario should be described in terms of Givens, Events, and Outcomes The givens should define all of, and no more than, the required context The event should describe the feature   Advantage of BDD:   The first advantage is that this domain language is usable/understandable by domain experts as well. Given that you e...
Read more

Angular JS – Part 2

Angular Routing Services and Directives: Routing Services: $route - $routeProvider is used to map the URL with template + assigned controller. (specially for SPAs). Methods: when(), otherwise(redirectTo:), current, params, pathParams, reload(). Also we can use resolve() to decide before load template. $routeParams – Used to pass the parameters via URL $location –Use to enable html5 routing. Can get the URL informations Eg. absUrl, protocol, port, host, path, search, hash, url. Replace() method will not keep URL history. Directives: Uses of Directives: Create custom element, custom event, observe and react to changes Use eventApp.dirctive("directiveName",function(){ return {restrict: 'E', template: "<input>"}}); Can restrict the directive to how should use. E(element), C(class), A(attribute), M(comment). Instead of template HTML text, the URL can be used. Use" replace: true" to replace the element name to html i...
Read more

.Net Collections

The  System.Collections namespace contains interfaces and classes that define various collections of objects, such as lists, queues, bit arrays, hash tables and dictionaries. Array list: Implements the IList interface using an array whose size is dynamically increased as required. Dictionary Base: Provides the abstract (MustInherit in Visual Basic) base class for a strongly typed collection of key-and-value pairs. Dictionary is a Generic type. So we can get type safety with Dictionary. Hash table: Represents a collection of key-and-value pairs that are organized based on the hash code of the key. A key cannot be a null reference (Nothing in Visual Basic), but a value can be. Hashtable is not a Generic type. So we can’t get type safety with Hashtable. Hashtable is thread safe. Queue: Represents a first-in, first-out collection of objects. Queues are useful for storing messages in the order they were received for sequential processing. Stack: ...
Read more