BIGDATA SIMULATION LIBRARY OF DREAM BY SANDMAN BOOKS
=================
Solution Architecture
Description
In the realm of Dreaming, its ruler SANDMAN, DREAM has a certain hobby; books. In his castle there is a Library in which they are kept, among other things, stories conceived by their authors but never written in our reality; Lucien, the person responsible for his organization, needs some help. Many people dream of published books, sales markets, stories that, in their reality, they would never imagine conceiving. And this voluminous data needs to be worked on. In order not to get lost in the information, Lucien receives all his dreams in a Non-Relational bank, MONGO. And he needs this to be organized in a relational way, that is, each author in his proper place. For that he pulled our dream and saw this Architecture where data arrives in MONGO undergo a transformation process in the STAGIN area and are populated in MYSQL. In its population, we split two final tables. One in its raw state, for complete queries, and another with metrics that informs the number of dreamers, their books and the total number of files. In this way, data is more organized, undergoing deduplication and consolidation processes.
Glossary of Data
| Fields | Type | Description |
|---|---|---|
| _id | long | undescore ID |
| kind | string | type of book or text file |
| title | string | title of book |
| subtitle | string | subtitle of book |
| author | array | one or more authors who can dream of stories |
| publisher | string | publisher or not dreamed of by the author |
| publishedDate | string | year of published |
| edition | string | which edition does the book belong to |
| sample | string | sample of books |
| type | string | ISBN code |
| identifier | string | isbn identification number |
| pageCount | integer | number of pages |
| capCount | integer | number of chapters |
| wordCount | integer | number of words |
| categories | string | literary genre |
| original_price | double | original price |
| current_prefix | string | country currency prefix |
| current_sufix | string | country currency name |
| barcode | string | barcode |
| dreaming_date | string | the day you had the dream |
image
Start the Project
To run the project, you need to install the dependencies located in the "dependencies" folder and in the root of the project, run the shell_script "run_script.sh".
Sample of Payload MONGO
mongo
{
"_id" : ObjectId("61b1fe6944dd42158674af31"),
"kind" : "books#volume",
"volumeInfo" : {
"title" : "STORE VISIT",
"Subtitle" : "GLASS IT HAIR MEMBER KEY ALMOST QUALITY. MARKET ALREADY AIR STILL ARTICLE. DECADE DECADE MEASURE PRESENT HUMAN MORNING. BIG BLOOD ECONOMIC FRONT SUCCESS AGO THEM. EVERY SON TROUBLE SIMPLE.",
"author" : [
"PETER RODRIGUEZ",
"KELLY TORRES"
]
},
"publisher" : "FALL AWAY ABOUT INDEPENDENT",
"publishedDate" : "1994",
"edition" : "7º EDITION",
"sample" : "...onto sport room audience. page dinner hundred. week statement should watch she even ball.\nour able tv break defense seek baby. employee last around music produce reach tv..",
"industryIdentifiers" : [
{
"type" : "ISBN_10",
"identifier" : "1-55027-208-X"
},
{
"type" : "ISBN_10",
"identifier" : "0-405-30324-6"
}
],
"pageCount" : 796,
"wordCount" : 83331,
"capCount" : 14,
"categories" : [
"NOVEL"
],
"saleInfo" : {
"original_price" : 78,
"current_prefix" : "LAK",
"current_sufix" : "Lao kip",
"barcode" : "6747254889534"
}
}
Sample of Payload in MYSQL
library
_id |kind |title |subtitle |author |publisher |publishedDate|edition |sample |type |identifier |pageCount|wordCount|capCount|categories |original_price|current_prefix|current_sufix |barcode |dreaming_date|
-----+------------+----------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------+----------------------------------------+-------------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+-----------------+---------+---------+--------+-------------------------+--------------+--------------+---------------------------+-------------+-------------+
670|csv#volume |GOOD DETERMINE OF |DON'T HAVE |KAREN ODOM |DARK WAR INDEPENDENT |1982 |9º EDITION|...involve star apply later including truth. next while nor worry staff economic.¶condition region write college. return half offer. popular could direction above fish.. |ISBN_10|978-1-4340-7508-6| 479| 64333| 11|EPISTOLARY NOVEL | 97.0|BHD |Bahraini dinar |3894426059691| 20211209|
resume
metric |value|
----------------+-----+
Total of Dreamns|71395|
Total of Books |59154|
Total of Data |78000|
5 Jun 11, 2022
1.6k Jan 03, 2023
1 Oct 22, 2021
4 Jun 05, 2022
1 Oct 02, 2022
5 Nov 13, 2022
7 Sep 04, 2022
5 Dec 22, 2021
6 Sep 07, 2022
1 Nov 08, 2021
67 Dec 27, 2022
17 Aug 21, 2022
1 Nov 22, 2021
2k Dec 29, 2022
6 Oct 11, 2022
32 Dec 31, 2022
214 Jan 02, 2023
0 Nov 10, 2021
1.2k Jan 03, 2023