Tài liệu Computer hardware analytic for electronic commerce

VIETNAM NATIONAL UNIVERSITY - HO CHI MINH CITY HO CHI MINH UNIVERSITY OF TECHNOLOGY COMPUTER SCIENCE AND ENGINEERING FACULTY GRADUATION THESIS Computer Hardware Analytic for Electronic Commerce Department: Computer science Committee: Advisor: Reviewer: Students: Computer Science 2 Dr. Phan Trong Nhan Assoc. Prof. Nguyen Thanh Binh ---o0o--Ly Gia Bao 1752089 Tran Nguyen Anh Huy 1752244 Tran Anh Thai 1752494 Le Huynh Long Vy 1752637 HO CHI MINH CITY, 12/2021 HO CHI MINH CITY, 12/2021 ĈҤ,+Ӑ&48Ӕ&*,$73+&0 ---------75ѬӠ1*ĈҤ,+Ӑ&%È&+.+2$ KHOA:KH & KT Máy tính ____ %Ӝ0Ð1HTTT ____________ &Ӝ1*+Ñ$;­+Ӝ,&+Ӫ1*+Ƭ$9,ӊ71$0 ĈӝFOұS- 7ӵGR- +ҥQKSK~F 1+,ӊ09Ө/8Ұ1È17Ӕ71*+,ӊ3 &K~ê6LQKYLrQSK̫LGiQWͥQj\YjRWUDQJQK̭WFͯDE̫QWKX\͇WWUình +Ӑ9¬7Ç1 Lý Gia BҧR____________________________ MSSV: 1752089 ______ +Ӑ9¬7Ç1 TrҫQ1JX\ӉQ$QK+X\ __________________ MSSV: 1752244 ______ +Ӑ9¬7Ç1 TrҫQ$QK7Kái _________________________ MSSV: 1752494 ______ +Ӑ9¬7Ç1 Lê HuǤQK/RQg Vӻ _____________________ MSSV: 1752637 ______ NGÀNH: Khoa hӑF0áy tính ___________________ /Ӟ3______________________ ĈҫXÿӅOXұQiQ Computer Hardware Analytics for Electronic Commerce ______________________________ 1KLӋPYө\rXFҫXYӅQӝLGXQJYjVӕOLӋXEDQÿҫX ✔ Surveying and analyzing related systems. ✔ Finding out hardware data sources for analytics. ✔ Proposing core features for the whole system towards recommendation, association rules, personalization, segmentation, and trends of hardware combination. ✔ Identifying main academic problems to be solved. ✔ Researching relevant theory (mainly association rules, prediction, recommendation techniques) and practical approaches for the core features implementation as well as for those issues. ✔Researching in-need technologies. ✔Designing the system with the proposed features and solutions. ✔Developing a system prototype. ✔Implementing and perfect the system. ✔Conducting empirical experiments. 3. 1Jj\JLDRQKLӋPYөOXұQiQ 23/08/2021 1Jj\KRjQWKjQKQKLӋPYө 13/12/2021 +ӑWrQJLҧQJYLrQKѭӟQJGүQ 3KҫQKѭӟQJGүQ 1) TS. Phan TrӑQJ1Kân ________________________________________________________ 1ӝLGXQJYj\rXFҫX/971ÿmÿѭӧFWK{QJTXD%ӝP{Q Ngày 23. tháng 08QăP2021 &+Ӫ1+,ӊ0%Ӝ0Ð1 *,Ҧ1*9,Ç1+ѬӞ1*'Ү1&+Ë1+ .êYjJKLU}K͕WrQ .êYjJKLU}K͕WrQ PGS. TS. TrҫQ0LQK4XDQJ 3+̮1'¬1+&+2.+2$%͠0Ð1 1JѭӡLGX\ӋWFKҩPVѫEӝ________________________ ĈѫQYӏ _______________________________________ 1Jj\EҧRYӋ __________________________________ ĈLӇPWәQJNӃW _________________________________ 1ѫLOѭXWUӳOXұQiQ _____________________________ TS. Phan TrӑQJ1Kân 75ѬӠ1*ĈҤ,+Ӑ&%È&+.+2$ KHOA KH & KT MÁY TÍNH &Ӝ1*+Ñ$;­+Ӝ,&+Ӫ1*+Ƭ$9,ӊ71$0 ĈӝFOұS- 7ӵGR- +ҥQKSK~F ---------------------------Ngày 20 tháng 12 QăP 2021 3+,ӂ8&+Ҩ0%Ҧ29ӊ/971 'jQKFKRQJ˱ͥLK˱ͣQJG̳Q) +ӑYjWrQ69 Lý Gia BҧR MSSV: 1752089 Ngành (chuyên ngành): Khoa hӑF0áy tính +ӑYjWrQ69 TrҫQ1JX\ӉQ$QK+X\ MSSV: 1752244 Ngành (chuyên ngành): Khoa hӑF0áy tính +ӑYjWrQ69 TrҫQ$QK7Kái MSSV: 1752494 Ngành (chuyên ngành): Khoa hӑF0áy tính +ӑYjWrQ69 Lê HuǤQK/RQg Vӻ MSSV: 1752637 Ngành (chuyên ngành): Khoa hӑF0áy tính ĈӅWjL Computer Hardware Analytics for Electronic Commerce +ӑWrQQJѭӡLKѭӟQJGүQ: TS. Phan TrӑQJ1Kân 7әQJTXiWYӅEҧQWKX\ӃWPLQK 6ӕWUDQJ 6ӕFKѭѫQJ 6ӕEҧQJVӕOLӋX 6ӕKuQKYӁ 6ӕWjLOLӋXWKDPNKҧR 3KҫQPӅPWtQKWRiQ +LӋQYұWVҧQSKҭP 7әQJTXiWYӅFiFEҧQYӁ - 6ӕEҧQYӁ %ҧQ$ %ҧQ$ .KәNKiF - 6ӕEҧQYӁYӁWD\ 6ӕEҧQYӁWUrQPi\WtQK 1KӳQJѭXÿLӇPFKtQKFӫD/971 - Students developed a system that analyzes computer components¶ score bands and recommends the combination of them, especially with their compatibility while there are many choices, to meet the end-users¶ need. Both auto and manual modes with advanced options are supported. Moreover, the users can customize their options as they wish. Even when the result is not found, the recommendation module will be automatically triggered. - Students analyzed problem statement as well as its related work and challenges. - Students applied modern technologies such as Django/python compatible with MVC pattern, redis, selenium, scrapy to implement the system and deployed the database on Google Cloud Platform. - Students were very proactive and open to novel knowledge and technology. 1KӳQJWKLӃXVyWFKtQKFӫDLVTN: - The evaluation of recommendation and system testing is still limited. - Some features need to improve such as dashboard with more metrics and component choices with further personalization, flexibility, and reliable benchmark scores. 8. ĈӅQJKӏĈѭӧFEҧRYӋ; %әVXQJWKrPÿӇEҧRYӋ† FkXKӓL69SKҧLWUҧOӡLWUѭӟF+ӝLÿӗQJ ĈiQKJLiFKXQJEҵQJFKӳJLӓLNKi7% Very good .K{QJÿѭӧFEҧRYӋ† ĈLӇP9.5/10 .êWrQJKLU}KӑWrQ Phan TrӑQJ1Kân Ho Chi Minh City University of Technology Faculty of Computer Science and Engineering SOCIALIST REPUBLIC OF VIETNAM Independence ± Freedom ± Happiness REVIEW OF GRADUATION THESIS HCMC, 28th December 2021 1. Students name: /ê*LD%ҧR1752089) 7UҫQ1JX\ӉQ$QK+X\1752244) 7UҫQ$QK7KiL /r+XǤQK/RQJ9ӻ 2. Major: Computer Science 3. Thesis title: Computer Hardware Analytic for Electronic Commerce 4. Reviewer: Assoc.Prof. Nguyen Thanh Binh, PhD 5. Contents - In this thesis, the authors built a system helping the users to build a computer even if they have no experience, while simultaneously providing an alternative for those who want to pick and choose the components for their own computer. The tasks in this thesis: + Build a web-server to interact with users. + Build a collection of data from multiple sources. + Build a data storage. + Develop an algorithm that will automatically generate a PC based on the user's requirements. + DevHORSDVLPSOHUHFRPPHQGDWLRQV\VWHPEDVHGRQXVHU¶VLQWHUDFWLRQV - The authors should explain more system which the authors built. The system has disadvantages for users. - Some spelling errors in the essay. - The author should clarify the advantages of the system which was built compared to existing systems. 6. General assessment: Marks 8.5/10 Reviewer Nguyen Thanh Binh Ho Chi Minh City, University of Technology Department of Computer Science and Engineer Declaration We guarantee that this research is our own, conducted under the supervision and guidance of our Instructor Phan Trong Nhan. The result of our research is legitimate and has not been published in any forms prior to this. All materials used within this research are collected by ourselves, by various sources which are appropriately listed in the references section. In any case of mistake, we will take full responsibility for it. Graduation Thesis Page 1 /83 Ho Chi Minh City, University of Technology Department of Computer Science and Engineer Acknowledgements We would like to express our deepest gratitude to our advisor Dr. Phan Trong Nhan for the continuous support and guidance during the making of our thesis. Besides Dr. Nhan, we would also like to thank all of our faculty lecturers, who have given us valuable and necessary knowledge to accomplish this project. Finally, we wish you health, wealth, and success in your endeavors. Graduation Thesis Page 2 /83 Ho Chi Minh City, University of Technology Department of Computer Science and Engineer Abstract Nowadays, in the age of technologies, a personal computer (PC) is becoming the most crucial part of our everyday life. However, for people who do not have a solid understanding or familiarity with the area, purchasing or building a proper computer for a person’s needs can be difficult. Therefore, it is an urgent need to research and develop a system that can take care of collecting the components’ data and help the user to build a computer that meets their requirements. In this topic, we are trying to create a system that focuses on helping the users to build a computer even if they have no experience, while simultaneously providing an alternative for those who want to pick and choose the components for their own computer. The system’s major goals will be to create a computer based on the user’s requirements, collect necessary data, and store it. Furthermore, using the information gathered from user interactions, we will create a simple recommendation system that will identify some of the most popular PCs and have them suggested to users. Graduation Thesis Page 3 /83 Ho Chi Minh City, University of Technology Department of Computer Science and Engineer Content 1 2 Introduction 10 1.1 Problem statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.2 Goal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.3 Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.4 Thesis structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Methodologies and Theoretical Background 12 2.1 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.1.1 Manual PC builders . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.1.2 Automatic PC generator . . . . . . . . . . . . . . . . . . . . . . . 12 2.1.3 PC performance calculator . . . . . . . . . . . . . . . . . . . . . . 12 Crawler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.2.1 Selenium . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.2.2 Scrapy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Web server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.3.1 Django . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.3.2 Redis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Google Cloud platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.2 2.3 2.4 3 System Requirement Analysis 18 3.1 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3.1.1 Data gathering . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3.1.2 Building PC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.1.3 Recommendation system . . . . . . . . . . . . . . . . . . . . . . . 19 Functional requirement . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.2.1 Automatic computer builder . . . . . . . . . . . . . . . . . . . . . 20 3.2.2 Manual computer buider . . . . . . . . . . . . . . . . . . . . . . . 20 3.3 Non-Functional requirement . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.4 Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 3.4.1 Use-case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 3.4.2 Use-case Specification . . . . . . . . . . . . . . . . . . . . . . . . 22 3.2 4 System Design 34 4.1 General Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 4.1.1 How the system work . . . . . . . . . . . . . . . . . . . . . . . . . 34 4.1.2 System architecture . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Crawler component . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 4.2 Graduation Thesis Page 4 /83 Ho Chi Minh City, University of Technology Department of Computer Science and Engineer 4.3 4.4 4.5 5 6 7 Database Component . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 4.3.1 Database schema . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 4.3.2 Detailed information of each entity in the ERD . . . . . . . . . . . 37 Web server Component . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 4.4.1 General architecture . . . . . . . . . . . . . . . . . . . . . . . . . 48 4.4.2 Build PC module . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 4.4.2.a Checking the compatibility between the components 50 4.4.2.b Build a PC from the user’s requirements . . . . . 53 4.4.3 Database module . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 4.4.4 Redis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 Recommendation system . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 System Implementation 59 5.1 Technologies used . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 5.2 Building a Prototype . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 5.2.1 Home Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 5.2.2 PC Auto Generator Page . . . . . . . . . . . . . . . . . . . . . . . 60 5.2.3 PC Auto Generator Result Page . . . . . . . . . . . . . . . . . . . 61 5.2.4 PC Manual Generator Page . . . . . . . . . . . . . . . . . . . . . . 65 5.2.5 Customize PC Set Page . . . . . . . . . . . . . . . . . . . . . . . . 66 5.2.6 View PC Set Details Page . . . . . . . . . . . . . . . . . . . . . . 68 5.2.7 Search Product Page . . . . . . . . . . . . . . . . . . . . . . . . . 69 5.2.8 Data Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . 69 System Testing 70 6.1 Non-functional Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 6.2 Functional Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 Conclusion 76 7.1 Achieved result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 7.2 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 7.3 Future development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 Graduation Thesis Page 5 /83 List of Figures 1.1 Budget distribution for gaming PC [3] . . . . . . . . . . . . . . . . . . . . . 10 2.1 Scrapy architecture [13] . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.2 Django Architecture [7] . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 pattern [7] 2.3 MTV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.1 Use-Case diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 3.2 Use-Case diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 4.1 General architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 4.2 General idea of index page and product page . . . . . . . . . . . . . . . . . 35 4.3 The flowchart of crawling . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 4.4 Database schema . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 4.5 Web server general architecture . . . . . . . . . . . . . . . . . . . . . . . . 49 4.6 Checking compatibility between the CPU and motherboard . . . . . . . . . 51 4.7 Intel CPU generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 4.8 Workflow for auto building PC . . . . . . . . . . . . . . . . . . . . . . . . 54 4.9 How PC scores are calculated [16] . . . . . . . . . . . . . . . . . . . . . . . 56 4.10 Workflow with data from Redis . . . . . . . . . . . . . . . . . . . . . . . . 58 5.1 Home Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 5.2 PC Auto Generator Page . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 5.3 Expand Advanced options . . . . . . . . . . . . . . . . . . . . . . . . . . 61 5.4 Generate successfully . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 5.5 Expand PC 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 5.6 Generate fails . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 5.7 Recommendation when generate fails . . . . . . . . . . . . . . . . . . . . 64 5.8 PC Manual Generator Page . . . . . . . . . . . . . . . . . . . . . . . . . . 65 5.9 Modify RAMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 5.10 Customize PC 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 5.11 PC 1 Details Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 5.12 Search Product Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 5.13 Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 6.1 Non-functional testing for the auto builder. . . . . . . . . . . . . . . . . . . 71 6.2 Non-functional testing for the auto builder result page. . . . . . . . . . . . 71 6.3 Non-functional testing for the manual builder. . . . . . . . . . . . . . . . . 72 6.4 Functional testing for the auto builder. . . . . . . . . . . . . . . . . . . . . 73 6.5 Functional testing for the auto builder (cont.) . . . . . . . . . . . . . . . . 74 6.6 Functional testing for the manual builder. . . . . . . . . . . . . . . . . . . 75 6.7 Functional testing for the recommendation system. . . . . . . . . . . . . . 76 Ho Chi Minh City, University of Technology Department of Computer Science and Engineer 7.1 Non-functional testing for navigation and login page. . . . . . . . . . . . . 80 7.2 Non-functional testing for register page. . . . . . . . . . . . . . . . . . . . 81 7.3 Non-functional testing for detail page and product page. . . . . . . . . . . . 81 7.4 Functional testing for login page and register page. . . . . . . . . . . . . . 82 7.5 Functional testing for the auto builder result page. . . . . . . . . . . . . . . 82 7.6 Functional testing for detail page and product page. . . . . . . . . . . . . . 83 Graduation Thesis Page 7 /83 List of Tables 1 User Login . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2 User Login (cont.) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3 User Register . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 4 Build PC Automatically . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 5 Build PC Manually . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 6 Re-examine Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . 27 7 View PC Set’s Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 8 View PC Set’s Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 9 Access Seller Pages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 10 Save a Complete PC Set . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 11 Search Product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 12 Sort Product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 13 View Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 14 CPU table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 15 Motherboard table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 16 GPU table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 17 RAM table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 18 SSD table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 19 HDD table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 20 PSU table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 21 CASE table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 22 RAM_Usage table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 23 SSD_Usage table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 24 HDD_Usage table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 25 PSU_Usage table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 26 Main_comp table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 27 Base budget percentage for each component compare to maximum budget. . 55 28 Technologies used . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 Abbreviation and Acronym Abbreviation Meaning PC Personal computer CPU Central processing unit GPU Graphics processing unit RAM Random-access memory HDD Hard disk drive SSD Solid-state drive PSU Power supply unit Ho Chi Minh City, University of Technology Department of Computer Science and Engineer 1 1.1 Introduction Problem statement In recent years, with the advancement of technology, a PC now is able to help us with many aspects of life including studying, working, entertaining,etc. But a PC is not a simple product and not all PCs are the same. In fact, a PC is made up of various components, and the purpose and grade of the computer might vary depending on the components used. For example, a PC used for entertaining purposes, specifically gaming, first prioritizes the budget on the graphics processing unit (GPU), second CPU and memory, other components such as storage and power supply unit (PSU) have lower priority because GPU, CPU and memory will have higher impact on the gaming experience compared to others. Figure. 1.1: Budget distribution for gaming PC [3] A PC utilized for working purposes, notably office work, on the other hand, may not even require a GPU, but rather a strong CPU and RAM to provide greater performance when performing office tasks. When compared to the requirement to handle the heavy image processing load when operating a game, office tasks frequently include only dealing with documents and calculations. Each component of a PC may have many manufacturers and each of them may also have their own specifications that need to be taken into consideration when purchasing. As a result, a system that allows a user to select from a wide range of components and build a PC that meets their needs while staying within budget has shown to be beneficial. Graduation Thesis Page 10 /83 Ho Chi Minh City, University of Technology Department of Computer Science and Engineer 1.2 Goal The main objective of the thesis is to develop a web-based system that can assist the users in the process of building a PC by checking the compatibility and the performance of the components, making sure that it matches the user’s requirements while also providing a tool for those who want to pick and choose their own components. The system will need a way to collect necessary data from multiple websites, specifically from computer hardware websites as well as a storage for those data. In addition, the thesis will also propose a simple recommendation system that bases on the user’s interactions with the system. 1.3 Scope In the scope of this thesis, we will focus on: • Build a web-server to interact with users. • Build a collection of data from multiple sources. • Build a data storage. • Develop an algorithm that will automatically generate a PC based on user’s requirements. • Develop a simple recommendation system based on user’s interactions. 1.4 Thesis structure The content of the thesis will be presented as follow: • Chapter 1. Introduction: Introduction overview, scope and goal of the thesis. • Chapter 2. Methodologies and Theoretical Background: Presents the theoretical background of the related topics as well as the tools and technologies that will be used. • Chapter 3. System Requirement Analysis: Analyzes the requirements of the system as well the functions it will provide. • Chapter 4. System Design: Presents the design and architecture of the entire system as well as its components. • Chapter 5. System Implementation: Describes how each component of the system is implemented. • Chapter 6. System Testing: Presents the test results of the system. Graduation Thesis Page 11 /83 Ho Chi Minh City, University of Technology Department of Computer Science and Engineer • Chapter 7. Conclusion: Summarizes the achieved results as well as the direction of development of the topic. • Reference 2 Methodologies and Theoretical Background 2.1 Related work 2.1.1 Manual PC builders The feature which allows users to manually build and customize their PC is now very popular all across computer hardware theme websites, ranging from online shops such as https://www.hanoicomputer.vn/, https://thenewxgear.com/, https://tinhocng oisao.com/, etc. to hardware statistic websites like https://www.userbenchmark.com/. There are even a few websites dedicated to building PC only like https://pcpartpicker .com/. It operates by determining which component is compatible with the one the user selected earlier. However, some of the restrictions are not checked or incorrect, like when the Random-access memory (RAM) slots you can choose exceeds the amount supported by the motherboard or when you choose an Intel CPU to go with a motherboard which only supports AMD CPUs. Only the dedicated website like https://pcpartpicker.com/ is the most reliable as it thoughtfully checks all the compatibility issues when adding a new component to the build. 2.1.2 Automatic PC generator For the automatically generating PC feature, our group could only locate one site which is https://wccftech.com/pc-builder/. It has some options for the user to specify their requirements before generating the PC. The result is generated fast and matches the user’s requirements. However, the system has some downsides. Firstly, the system may not yield any results at some budget point. Secondly, there are only a few options for the user to describe their PC requirements. Finally, for a specific budget, the system only yields one result. 2.1.3 PC performance calculator In order to calculate a PC’s performance, a specific testing application must be executed on the PC. Therefore, you need to have actual hardware in order to run the testing program which will provide the performance score for the PC components. Graduation Thesis Page 12 /83 Ho Chi Minh City, University of Technology Department of Computer Science and Engineer This will require a big pool of sample hardware in order to have enough data. To do this the https://www.userbenchmark.com/ website created a software to calculate the performance of PC running it and published that software for the user to download. The software then gathers those data from users and display them on the website. In this thesis, we will use the performance data from this website. 2.2 Crawler 2.2.1 Selenium Selenium [15] is a free (open-source) automated testing framework used to validate web applications across different browsers. It is not developed mainly for scraping but still can be used for it. It can be used in multiple programming languages like Java, C#, Python. Selenium has four major components – Selenium IDE, Selenium RC, Selenium Web driver, Selenium GRID. In our project, we will be using Selenium on Python with the Google Chrome WebDriver. Through Selenium Python API we can access all functionalities of Selenium WebDriver in an intuitive way. 2.2.2 Scrapy Scrapy [14] is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful applications, like data mining, information processing or historical archival. Even though Scrapy is originally designed for web scraping, it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general-purpose web crawler. Scrapy architecture: [13] The data flow in Scrapy is controlled by the execution engine, and goes like this: 1. The Engine gets the initial Requests to crawl from the Spider. 2. The Engine schedules the Requests in the Scheduler and asks for the next Requests to crawl. 3. The Engine sends the Requests to the Downloader, passing through the Downloader Middlewares. 4. Once the page finishes downloading, the Downloader generates a Response (with that page) and sends it to the Engine, passing through the Downloader Middlewares. 5. The Engine receives the Response from the Downloader and sends it to the Spider for processing, passing through the Spider Middleware. Graduation Thesis Page 13 /83 Ho Chi Minh City, University of Technology Department of Computer Science and Engineer Figure. 2.1: Scrapy architecture [13] 6. The Spider processes the Response and returns scraped items and new Requests (to follow) to the Engine, passing through the Spider Middleware 7. The Engine sends processed items to Item Pipelines, then sends processed Requests to the Scheduler and asks for possible next Requests to crawl. 8. The process repeats (from step 1) until there are no more requests from the Scheduler. Scrapy Component: [13] • Scrapy Engine: The engine is responsible for controlling the data flow between all components of the system, and triggering events when certain actions occur. • Scheduler: The Scheduler receives requests from the engine and enqueues them for feeding them later (also to the engine) when the engine requests them. • Downloader: The Downloader is responsible for fetching web pages and feeding them to the engine which, in turn, feeds them to the spiders. • Spiders: Spiders are custom classes written by Scrapy users to parse responses and extract items from them or additional requests to follow. • Item Pipeline: The Item Pipeline is responsible for processing the items once they have been extracted (or scraped) by the spiders. Typical tasks include cleansing, validation and persistence (like storing the item in a database). Graduation Thesis Page 14 /83 Ho Chi Minh City, University of Technology Department of Computer Science and Engineer • Downloader middlewares: Downloader middlewares are specific hooks that sit between the Engine and the Downloader and process requests when they pass from the Engine to the Downloader, and responses that pass from Downloader to the Engine. • Spider middlewares: Spider middlewares are specific hooks that sit between the Engine and the Spiders and are able to process spider input (responses) and output (items and requests). 2.3 Web server 2.3.1 Django Django is a high-level Python web framework that encourages rapid development and clean, pragmatic design. It offers a standard method for fast and effective website development by providing many tools and built- in features to handle common web development tasks like user authentication, content administration, site maps, etc. Figure. 2.2: Django Architecture [7] Django is designed based on the Model-Template-View (MTV) pattern which is a little different from the commonly used Model-View-Controller (MVC) pattern. The main different is that Django itself manages the user’s interaction which is the Controller part in MVC. Figure. 2.3: MTV pattern [7] Graduation Thesis Page 15 /83 Ho Chi Minh City, University of Technology Department of Computer Science and Engineer After handling user’s interaction, view will be called which interacts with the model and templates to produce a response that will be returned to the user. Advantages of Django [8] • Django is easy to set up and run. • It provides an easy-to-use interface for various administrative activities. • Helps you to define patterns for the URLs in your application. • Offers built-in authentication system. • Cache framework comes with multiple cache mechanisms. • High-level framework for rapid web development. • A complete stack of tools. • Data modelled with Python classes. Disadvanges of Django [8] • It is a monolithic platform. • High dependence on Django ORM. Broad Knowledge required. • Only allows you to handle a single request per time. • Routing requires some knowledge of regular expressions A few general steps to build an application on Django: [4] • Design the model • Install the model • Use the provided API to access the data • Use the administrative interface to set up the data • Design URLs • Design template • Run the application Graduation Thesis Page 16 /83