Before Starting this make sure you have gone through the basic of BigData tutorial.

What is Hadoop?

Hadoop is an open-source software framework for storing data and running applications on clusters of commodity hardware. The Hadoop framework application works in an environment that provides distributed storage and computation across clusters of computers. It provides massive storage for any kind of data, enormous processing power and the ability to handle virtually limitless concurrent tasks or jobs. Hadoop is designed to scale up from single server to thousands of machines, each offering local computation and storage.

Hadoop has two major :-

  • Processing layer (MapReduce)
  • Storage layer (Hadoop Distributed File System).

What is MapReduce?

MapReduce is the processing layer in Hadoop. It is a software framework for writing an application that performs distributed processing. MapReduce is a programming model and an associated implementation for processing and generating big data sets with a parallel, distributed algorithm on a cluster.
We will learn more about MapReduce later on this tutorial.


Hadoop Distributed File System

HDFS is the world’s most reliable storage system. HDFS is a File system of Hadoop designed for storing very large files running on a cluster of commodity hardware. It is designed on the principle of storage of less number of large files rather than the huge number of small files. Hadoop HDFS provides a fault-tolerant storage layer for Hadoop and its other components. HDFS Replication of data helps us to attain this feature. It stores data reliably, even in the case of hardware failure. It provides high throughput access to application data by providing the data access in parallel.

Apart from the above-mentioned two core components, Hadoop framework also includes the following two modules −

  • Hadoop Common − These are Java libraries and utilities required by other Hadoop modules.
  • Hadoop YARN − This is a framework for job scheduling and cluster resource management.


How Does Hadoop Works ?



You can tie together many commodity computers with single-CPU, as a single functional distributed system and practically, the clustered machines can read the dataset in parallel and provide a much higher throughput. Moreover, it is cheaper than one high-end server. So this is the first motivational factor behind using Hadoop that it runs across clustered and low-cost machines.

Hadoop runs code across a cluster of computers. This process includes the following core tasks that Hadoop performs −

  • Data is initially divided into directories and files. Files are divided into uniform sized blocks of 128M and 64M (preferably 128M).
  • These files are then distributed across various cluster nodes for further processing.
  • HDFS, being on top of the local file system, supervises the processing.
  • Blocks are replicated for handling hardware failure.
  • Checking that the code was executed successfully.
  • Performing the sort that takes place between the map and reduce stages.
  • Sending the sorted data to a certain computer.
  • Writing the debugging logs for each job.


Advantages of Hadoop

  • Scalable: Hadoop is a highly scalable storage platform because it can store and distribute very large data sets across hundreds of inexpensive servers that operate in parallel.
  • Cost-effective: The problem with traditional relational database management systems is that it is extremely cost prohibitive to scale to such a degree in order to process such massive volumes of data. In an effort to reduce costs, many companies in the past would have had to down-sample data and classify it based on certain assumptions as to which data was the most valuable.
  • Flexible: Hadoop enables businesses to easily access new data sources and tap into different types of data (both structured and unstructured) to generate value from that data. This means businesses can use Hadoop to derive valuable business insights from data sources such as social media, email conversations.
  • Fast: Hadoop’s unique storage method is based on a distributed file system that basically ‘maps’ data wherever it is located on a cluster. The tools for data processing are often on the same servers where the data is located, resulting in much faster data processing. If you’re dealing with large volumes of unstructured data, Hadoop is able to efficiently process terabytes of data in just minutes, and petabytes in hours.
  • Resilient to failure:A key advantage of using Hadoop is its fault tolerance. When data is sent to an individual node, that data is also replicated to other nodes in the cluster, which means that in the event of failure, there is another copy available for use.



Lets know about the hdfs architecture which will give you a complete picture of the Hadoop Distributed File System. There is a single NameNode that stores metadata, and there are multiple DataNodes that do actual storage work.
Nodes are arranged in racks, and replicas of data blocks are stored on different racks in the cluster to provide fault tolerance. NameNode stores metadata, and DataNode stores actual data. The client interacts with NameNode for performing any tasks, as NameNode is the centerpiece in the cluster. There are several DataNodes in the cluster which store HDFS data in the local disk.
DataNode sends a heartbeat message to NameNode periodically to indicate that it is alive. Also, it replicates data to other DataNode as per the replication factor.

Hadoop Distributed File System follows the master-slave architecture. Each cluster comprises a single master node and multiple slave nodes.
Internally the files get divided into one or more blocks, and each block is stored on different slave machines depending on the replication factor (which you will see later in this article).The master node stores and manages the file system namespace, that is information about blocks of files like block locations, permissions, etc. The slave nodes store data blocks of files.




HDFS has a master/slave architecture. An HDFS cluster consists of a single NameNode,a master server that manages the file system namespace and regulates access to files by clients. In addition, there are a number of DataNodes, usually one per node in the cluster,which manage storage attached to the nodes that they run on. HDFS exposes a file system namespace and allows user data to be stored in files. Internally, a file is split into one or more blocks and these blocks are stored in a set of DataNodes. The NameNode executes file system namespace operations like opening, closing, and renaming files and directories. It
also determines the mapping of blocks to DataNodes. The DataNodes are responsible for serving read and write requests from the file system’s clients. The DataNodes also perform block creation, deletion, and replication upon instruction from the NameNode.



Generally the user data is stored in the files of HDFS. The file in a file system will be divided into one or more segments and/or stored in individual data nodes. These file segments are called as blocks. In other words, the minimum amount of data that HDFS can read or write is called a Block. The default block size is 64MB, but it can be increased as per the need to change in HDFS configuration.



4,105 thought on “HADOOP HDFS”

  1. 720p says:

    Waooo Jades Hotel I will be there. Please book two tables of N6,000 category at the Pool side. Willetta Bentlee Autry

  2. 720p says:

    Hello friends, good paragraph and nice arguments commented at this place, I am actually enjoying by these. Peg Buddie Henley

  3. turkce says:

    Hello, I want to subscribe for this web site to take most up-to-date updates, so where can i do it please assist. Celka Nikki Strait

  4. 720p says:

    Way cool! Some extremely valid points! I appreciate you writing this article plus the rest of the site is extremely good. Kylen Conroy Glarum

  5. 720p says:

    Hi, after reading this amazing paragraph i am also delighted to share my know-how here with colleagues. Suzanna Orlando Zel

  6. yetiskin says:

    Although web-sites we backlink to beneath are considerably not associated to ours, we feel they are actually worth a go through, so have a look. Esma Jeth Grier

  7. yify says:

    I really like and appreciate your article. Thanks Again. Kessia Cos Wernsman

  8. bedava says:

    What a information of un-ambiguity and preserveness of valuable know-how about unexpected emotions. Cinda Gibbie Shaylynn

  9. yify says:

    This is my first time pay a visit at here and i am really impressed to read all at one place. Alameda Lefty Woodsum

  10. bluray says:

    Way cool! Some very valid points! I appreciate you writing this article and the rest of the site is also very good. Linet Karlik Schweiker

  11. web-dl says:

    Way cool! Some very valid points! I appreciate you penning this write-up and the rest of the website is really good. Norrie Algernon Krongold

  12. Fgrsboale says:

    cheap viagra generic 100mg generic viagra sildenafil citrate viagra hong kong

  13. online loan says:

    A lot of thanks for all your valuable efforts on this site. Kate loves engaging in investigation and it’s really easy to understand why. I know all concerning the dynamic medium you give important secrets on this web blog and as well as foster participation from others on that content plus our daughter is always discovering a lot. Have fun with the remaining portion of the new year. You are conducting a great job.

  14. Thank you for each of your hard work on this web page. My aunt really likes going through investigations and it’s really obvious why. Most of us learn all of the lively method you offer practical strategies through this web site and even increase participation from some other people on the issue so our favorite girl is in fact learning a lot of things. Take advantage of the rest of the year. You are always doing a really good job.

  15. I precisely needed to appreciate you once again. I am not sure the things that I might have handled in the absence of the entire points provided by you on such a problem. It has been a very frightful dilemma in my circumstances, however , noticing the skilled mode you processed that made me to jump over fulfillment. I am happier for this help and in addition hope that you know what a great job you are always getting into instructing others through the use of your web blog. I know that you have never encountered all of us.

  16. Thanks for your whole work on this website. My mum takes pleasure in setting aside time for research and it is simple to grasp why. We notice all relating to the powerful ways you render very useful tips and tricks via your website and therefore strongly encourage response from website visitors on this theme while our own simple princess is without a doubt understanding so much. Take pleasure in the rest of the year. Your performing a useful job.

  17. My spouse and i have been really lucky that Louis managed to finish off his research through the ideas he grabbed out of your web page. It’s not at all simplistic to just continually be giving out methods that many others have been selling. And we also fully understand we have got the website owner to be grateful to for that. The main explanations you have made, the simple site menu, the friendships you can help foster – it’s got many impressive, and it’s really helping our son and the family imagine that the matter is excellent, and that is pretty indispensable. Thank you for everything!

  18. NllpFeetS says:

    usa pharmacy http://pharmacy-onlineasxs.com/ best online canadian pharmacy

  19. ucretsiz says:

    A big thank you for creating such an useful article on this subject. I hope to find here more of these done by you, in the coming weeks. Dayle Joey Hen

  20. KhthSeks says:

    best drugstore mascara nearest drugstore drugstore makeup

  21. Lokuboale says:

    24 hours pharmacy pharmacy near me mail order pharmacy

  22. Jllophirl says:

    canadian pharmacy reviews cheapest pharmacy canadian pharmacies online

  23. ucretsiz says:

    There is noticeably a bundle to know about this. I assume you made certain nice points in features also. Linea Warren Fiedler

  24. 554628 477438Excellent read, I just passed this onto a friend who was doing some research on that. And he just bought me lunch since I found it for him smile Thus let me rephrase that: Thank you for lunch! 173700

  25. Even you need to use facetime for official video chat. Tracee Caesar Caitlin

  26. yabanci says:

    I really enjoy the article post. Really thank you! Keep writing. Sheeree Theobald Theodoric

  27. 627538 645662Das beste Webdesign Berlin erhalten Sie bei uns, genauso wie professionelles Webdesign. Denn wir sind die Webdesign Agentur mit pfiff. 175857

  28. donmadan says:

    Hi there. I discovered your web site by way of Google whilst searching for a comparable subject, your site came up. It seems to be good. I have bookmarked it in my google bookmarks to come back then. Frieda Burton Henka

  29. Jbbvboale says:

    pharmacy online navarro pharmacy pharmacy on line

  30. KbcxSeks says:

    canada pharmacies online prescriptions cialis super active ed meds online

  31. movie online says:

    Aute mi ut suspendisse velit leo, vel risus ac. Amet dui dignissim fermentum malesuada auctor volutpat, vestibulum ipsum nulla. Berrie Natty Gobert

  32. 720p says:

    I think this is a real great article post. Really thank you! Much obliged. Nicky Chad Malloch

  33. Ahkdboale says:

    canadian prescriptions online best online pharmacy stores kamagra oral jelly

  34. hacker shell says:

    If you are going for best contents like myself, only go
    to see this web page all the time as it provides feature contents, thanks

  35. bluray says:

    Inspiring story there. What occurred after? Thanks! Vida Gannie Randell

  36. leke kremi says:

    Thanks for the auspicious writeup. It in reality was once
    a enjoyment account it. Look complex to more delivered agreeable from you!
    However, how can we be in contact?

  37. turkce says:

    Very good article. I definitely appreciate this site. Keep writing! Terese Cort Dygall

  38. takip2018 says:

    Normally I don’t learn post on blogs, but I wish to say that this write-up very compelled me to check out and do
    it! Your writing taste has been amazed me. Thanks, quite nice

  39. turkce says:

    As I web site possessor I believe the content matter here is rattling wonderful , appreciate it for your efforts. You should keep it up forever! Best of luck. Lolly Andy Lorou

  40. torrent says:

    Hello Dear, are you truly visiting this website on a regular basis, if so afterward you will definitely get good know-how.| Viviyan Donall Hulda

  41. anime says:

    Nice!! Get high-performance spray coating as per your choice. Protect your vehicle by coating. Spray coating is the best option for protection of vehicles. Viviene Kerby Stander

  42. NncsFeetS says:

    can i buy viagra over the counter in canada viagra for sell buy viagra buy in canada

  43. Fqhhboale says:

    online discount pharmacy erectile dysfunction canada medication pharmacy

  44. dizi says:

    Hello, I enjoy reading through your article. I like to write a little comment to support you.| Debbi Millard Kluge

  45. hindi movie says:

    Flashy cinematography to hide disappointing product specs? Jen Cal Taryn

  46. Very good article! We are linking to this particularly great article on our website. Keep up the great writing. Aveline Justen Phiona

  47. online says:

    Some really nice and utilitarian information on this web site, as well I believe the style contains excellent features. Barbaraanne Freeman Ressler

  48. indirmeden says:

    Happy Holidays to you too! This matcha holiday bark is so fun to give as holiday gifts! Something out of the ordinary. Quentin Rudolph Hullda

  49. turkce says:

    You write so honestly about this. Thanks for sharing! Tanhya Morse Haddad

  50. indirmeden says:

    Do you mind if I quote a few of your articles as long as I provide credit and sources back to your weblog? My website is in the very same niche as yours and my users would certainly benefit from a lot of the information you present here. Please let me know if this ok with you. Thank you! Loralee Kip Lustig

  51. torrent says:

    Very good article. I will be going through a few of these issues as well.. Andriana Gustave Shaughn

  52. Jbnvphirl says:

    cash loan in charlottesville va payday loans in springfield oh the guild 2 loan money

  53. Jbbnboale says:

    por cuanto tiempo puedo tomar cialis site vente cialis fiable how much cialis is too much

  54. KvaxSeks says:

    cash advance in minutes payday advance clinton tn payday advance jane missouri

  55. 126977 165285Hi there! Someone in my Myspace group shared this site with us so I came to give it a look. Im definitely loving the information. Im bookmarking and will be tweeting this to my followers! Outstanding weblog and fantastic style and style. 435492

  56. Ahbzboale says:

    cash advance lincoln blvd payday loans dixon ca do payday loans ruin your credit

  57. Labxboale says:

    cialis polen achat de vrai cialis drugs that interact with cialis

  58. Donaldtib says:

    Very nice quality article

    free robux

  59. Fqbbboale says:

    payday loans cwmbran manual cash advance payday loans in dunn nc

  60. Today, considering the fast life-style that everyone leads, credit cards have a big demand throughout the economy. Persons throughout every area are using the credit card and people who aren’t using the credit cards have arranged to apply for 1. Thanks for revealing your ideas about credit cards. https://psoriasismedi.com best cream for psoriasis over the counter

  61. excellent put up, very informative. I ponder why the opposite specialists of this sector don’t notice this. You must proceed your writing. I’m confident, you’ve a great readers’ base already! https://epilepsymedi.com new epilepsy drugs

  62. Kxcydp says:

    tadalafil tablets – tadstrong.com tadalafil 20

  63. WesleySig says:

    tadalafil gel tadalafil tablets
    tadalafil 40

  64. MarcosVot says:

    40 mg tadalafil: http://tadalafilonline20.com/ tadalafil 40 mg from india

  65. Nxqzoe says:

    how much does propecia cost without insurance – http://finasteridepls.com/ finasteride alternative

  66. Randyheips says:

    meds online without doctor prescription: https://genericwdp.com/ generic pills for sale

  67. MichaelBup says:

    prescription drugs online without doctor: https://genericwdp.com/ india pharmacy mail order

  68. Randyheips says:

    generic pills without a doctor prescription buy prescription drugs

  69. Crpptg says:

    does propecia thicken thinning hair – cost of propecia generic propecia before and after reddit

  70. MichaelBup says:

    usa pharmacy india: https://genericwdp.com/ cheap generic drugs from india

  71. is amoxicillin amoxicillin over the counter amoxicillin trihydrate and clavulanate potassium how long does it take for amoxicillin to start working

  72. WayneSok says:

    where to buy viagra online viagra over the counter walmart
    over the counter viagra

  73. dog lasix lasix 100mg online what is a renal scan with lasix where can i buy furosemide

  74. Jeffreysuife says:

    over the counter viagra viagra cost per pill
    viagra cost

  75. ivermectin and alcohol ivermectin 24 mg ivermectin for lice in humans how much 1.87% ivermectin per pound for mange

  76. azithromycin near me cost of zithromax 500 mg azithromycin for gonorrhea and chlamydia what is azithromycin tablets for

  77. WayneSok says:

    100mg viagra viagra cost per pill
    viagra over the counter

  78. Sqlxdi says:

    tadalafil professional – sildenafil vardenafil and tadalafil tadalafil cheap tadalafil

  79. ivermectin dog wormer stromectol 3 mg dosage does ivermectin kill barber pole worms how much ivermectin topical to give a dog

  80. Jeffreysuife says:

    best place to buy viagra online viagra without a doctor prescription usa
    order viagra online

  81. albuterol inhaler pregnancy ventolin 8g albuterol is a generic name for: when will albuterol inhalers go generic

  82. WayneSok says:

    how much is viagra viagra price
    viagra from india

  83. Jeffreysuife says:

    buy generic 100mg viagra online viagra amazon
    viagra price

  84. ventolin salbutamol ventolin cost usa what is albuterol sulfate used for how often can you take albuterol breathing treatments

  85. WayneSok says:

    buy real viagra online best over the counter viagra
    viagra from india

  86. Jeffreysuife says:

    over the counter viagra viagra online usa
    viagra cost

  87. ivermectin for tapeworm stromectol tab 3mg side effects of ivermectin in dogs how long does ivermectin stay in the system

  88. WayneSok says:

    viagra cost viagra without a doctor prescription
    buy viagra online canada

  89. Jeffreysuife says:

    online doctor prescription for viagra where to buy viagra
    order viagra online

  90. amoxicillin gonorrhea dosage price of amoxil can i take amoxicillin with nyquil what is amoxicillin 500

  91. xopenex vs ventolin albuterol 0.063 does albuterol keep you awake what is difference in ventolin and proair

  92. WayneSok says:

    cheap viagra online best place to buy generic viagra online
    viagra over the counter

  93. RicardoFaugs says:

    paxil 10mg paxil for menopause
    online canadian pharmacy

  94. amoxicillin syrup buy augmentin 625mg amoxicillin 400mg/5ml dosage calculator what are the side effects of amoxicillin?

  95. drug lasix says:

    furosemide 40mg tablet can you buy furosemide otc taking furosemide to lose weight what is the drug lasix used for

  96. RicardoFaugs says:

    paxil for ocd paxil for children
    top rated ed pills

  97. StanleyJoync says:

    chloroquine for sale chloroquine

  98. furosemide blood pressure lasix cheap side effects of lasix in cats a nurse is caring for a patient who is taking furosemide

  99. SKincell Pro is well known by the folks who utilized it as a perfect
    skin fixor. The product has the exact same components as much of the various other moles removing creams on the market today, yet it includes more than a
    number of them. The product likewise boosts your skin appearance as well as supplies even-toned skin. The
    components, the item has are all natural active ingredients which are safe for your skin. http://gatherbookmarks.com/story8530432/how-much-is-a-bottle-of-skincell-pro

  100. zithromax suspension dosing buying zithromax in mexico is azithromycin the same as zithromax what is azithromycin 250 mg dose pack used for

  101. RicardoFaugs says:

    п»їviagra pills https://viagrapills100.com/ best place to buy generic viagra online

  102. Thomasnut says:

    generic viagra walmart https://viagrapills100.com/ viagra from india

  103. zithromax 250mg zithromax 500mg australia azithromycin dose for strep throat how to get azithromycin dose for chlamydia

  104. Bardzo interesujący temat, dzięki za wysłanie wiadomości plastry na opryszczkę.

  105. RicardoFaugs says:

    cost of viagra https://viagrapills100.com/ viagra over the counter walmart

  106. Thomasnut says:

    buy generic 100mg viagra online https://viagrapills100.com/ viagra without a doctor prescription usa

  107. doxycycline esophagitis buy doxycycline south africa doxycycline dosage for cats uti how long can you take doxycycline for acne

  108. fertility drugs clomid buy clomid from mexico how much clomid should a man take how long after clomid should i use opk

  109. RicardoFaugs says:

    viagra over the counter https://viagrapills100.com/ viagra over the counter

  110. Thomasnut says:

    viagra over the counter https://viagrapills100.com/ order viagra online

  111. prednisolone and fioricet buy prednisolone 5mg singapore ofloxacin and prednisolone eye drops what does prednisolone do after cataract surgery

  112. Richardfroke says:

    when will viagra be generic best over the counter viagra
    mexican viagra

  113. doxycycline toxicity doxycycline pills price in south africa what does doxycycline hyclate do how long does doxycycline take to clear acne

  114. clomid cheap online clomid online purchase when do you ovulate after taking clomid how to take clomid pct

  115. Marvinslani says:

    buy ed drugs cheap ed pills from canada
    order ed pills

  116. Elmerspurn says:

    buy ed pills ed pills without a doctor prescription
    how to treat ed

  117. cat prednisolone dosage prednisolone tablets can i take benadryl with prednisolone prednisolone 5 mg how many times take

  118. tabletki priligy buy generic dapoxetine online can you buy priligy in usa what is priligy 60 mg (dapoxetine)

  119. TerrySeets says:

    buy ed pills from canada medications for ed
    order ed pills

  120. Marvinslani says:

    cheap ed pills from canada cheap ed pills from canada
    cheap ed pills usa

  121. TerrySeets says:

    medicine for ed cheap ed pills in mexico
    erectyle dysfunction

  122. oxvow says:

    good post, i love it

  123. TerrySeets says:

    cheap ed pills from india best ed pills non prescription
    ed pills online

  124. Elmerspurn says:

    buy ed pills drugs online
    ed pills without a doctor prescription

  125. fish diflucan diflucan 150 capsule price of diflucan without insurance what kind of medication is diflucan

  126. priligy buy dapoxetine online no prescription priligy farmacias del ahorro when will priligy be available in the us

  127. Tmptfy says:

    Teqlpx – mens ed pills Gvdrmx nonjyo

  128. synthroid directions synthroid 115 mcg white round pill 50 synthroid what to avoid when taking synthroid

  129. Vdynoi says:

    Ghajmc – lasix 100 mg Glcnqg rhtstw

  130. diflucan thrush breastfeeding how to buy diflucan online can i take diflucan with food how long does it take for diflucan to work for thrush

  131. Marvinslani says:

    plaquenil tablets 200mg: cheap plaquenil – plaquenil 300 mg

  132. Jasonshumb says:

    http://zithromaxst.com/# generic zithromax online paypal

  133. Marvinslani says:

    neurontin capsules 600mg: neurontin 600 mg coupon – neurontin generic south africa

  134. Ddrvsu says:

    Uuihrf – azithromycin Ivvnbm zmmpdd

  135. Jasonshumb says:

    https://zithromaxst.com/# zithromax 500 price

  136. Marvinslani says:

    prednisone 50 mg for sale: 50mg prednisone tablet – prednisone 20mg

  137. Marvinslani says:

    prednisone buy canada: buy prednisone online – buy prednisone online fast shipping

  138. Hey! Someone in my Facebook group shared this website with us so I came to take a look. I’m definitely loving the information. I’m book-marking and will be tweeting this to my followers! Great blog and terrific design.

  139. Jasonshumb says:

    https://hydroxychloroquinest.com/# hydroxychloroquine-o-sulfate

  140. Victorastef says:

    https://zithromaxproff.com/# where can i get zithromax
    zithromax order online uk

  141. Scottfoeda says:

    https://zithromaxproff.com/# zithromax antibiotic without prescription
    average cost of generic zithromax

  142. Victorastef says:

    http://zithromaxproff.com/# zithromax z-pak price without insurance
    zithromax over the counter